Author SHA1 Message Date
jschoubben 3a9deb8cdb Merge pull request 'R4g (mesh-lab): the checker judges the lab; its scripts become tests (design 47)' (#108) from feat/r4g into main
Reviewed-on: #108
2026-10-23 17:54:01 +00:00
jschoubben 4653b17e18 Delete merge-check.sh; the type check runs before npm test, where the Node level runs it (issue 1295, S5 root; ADR 0529)
mesh/repo-check head 4653b17e18a7 merged into trunk 72ea2988be50 — fail: mesh/repo-check is required here and this head defines no merge-check.sh — a perso…
mesh/delivery delivered
mesh/delivery-group group feat/r4g merged: every member is merged
mesh/check pass 4653b17e18a7 on 3a9deb8cdbd9 | unit 39 run, 5 kept, 1 skipped | integration 39 run, 12 kept, 1 skipped | composition passed
2026-10-23 15:37:13 +02:00
jschoubben bdb927f83d Delete asks/merge-check.sh; the proof against the clones is an integration test, the shell syntax a unit test (issue 1295, S5 asks; ADR 0529) 2026-10-23 15:37:08 +02:00
jschoubben 49ee611847 Delete replays/merge-check.sh; tests that raise containers or a bus move to the integration level (issue 1295, S5 replays; ADR 0529)
gofmt and vet are the Go build module's; the unit level keeps the register and the bed's reading of a change.
2026-10-23 15:36:51 +02:00
mesh-tools 72ea2988be Merge pull request 'Register issue 867's replays (issue 867, ADR 0382)' (#107) from fix/867-register-the-replay into main 2026-10-16 01:50:15 +00:00
mesh-toolsandClaude 9294defef8 Register issue 867's replays: a delivery group's members checked beside each other (issue 867, ADR 0382)
mesh/merge-gate head 9294defef824 merged into trunk f024be6e0457 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 9294defef824 merged into trunk f024be6e0457 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
R867, R-issue867-recheck, R-issue867-recheck-verb and R-issue867-release-group name mesh-controller's replay867_test.go at the merges of #348, #350, #351 and #352; prove shows each failing before its fix and passing on it.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-16 01:46:06 +00:00
mesh-admin f024be6e04 Merge pull request 'Give the base build several mirrors and retry a failed pacman step (issue 700)' (#106) from fix/700-the-base-build-has-mirrors into main 2026-10-14 05:45:50 +00:00
jochen b053dc66bf Give the base build several mirrors and retry a failed pacman step (issue 700)
mesh/merge-gate head b053dc66bf9d merged into trunk c0b1225f6d7d — pass: the change touches no module of the mesh's graph
mesh/repo-check head b053dc66bf9d merged into trunk c0b1225f6d7d — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
The stock Arch image names one mirror, and the base build failed twice in
a morning when mirrors.kernel.org timed out and then answered 500. Write a
fixed list of six mirrors before anything is installed, so pacman falls
back per file, and retry a failed step three times with a growing pause,
saying which mirror failed.
2026-10-14 07:39:30 +02:00
mesh-admin c0b1225f6d Merge pull request 'A suite run in which no test passed exits non-zero and names its skips (issue 693)' (#105) from fix/693-a-run-that-proved-nothing-is-not-passed into main 2026-10-14 05:05:36 +00:00
jochen 5ba88c69fa Read the runner's last summary and NOT REACHED in any case (issue 693, review 253)
mesh/merge-gate head 5ba88c69fa80 merged into trunk f9be71b9ad44 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 5ba88c69fa80 merged into trunk f9be71b9ad44 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
mesh/delivery-group group fix/693-a-run-that-proved-nothing-is-not-passed merged: every member is merged
A test that printed a summary-shaped line before it skipped was read as the
runner's totals: "ℹ pass 5" made a run that proved nothing pass.
2026-10-14 03:28:14 +02:00
jochen 8635560993 A suite run in which no test passed exits non-zero and names its skips (issue 693)
mesh/merge-gate head 8635560993aa merged into trunk f9be71b9ad44 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 8635560993aa merged into trunk f9be71b9ad44 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery-group group fix/693-a-run-that-proved-nothing-is-not-passed ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
The runner exits 0 when every test skipped, so a bed that stopped before its
subject and skipped its only test as NOT REACHED was recorded as passed, "0
passed, 0 failed". The suite now collects each skip and its reason through a
reporter of its own, prints NOT PASSED and a skipped line per skip after its
recorded line, keeps them in the receipt, and exits 1 when no test passed or a
test was not reached. A skip for a declared reason beside passes stays passed.
2026-10-14 03:14:05 +02:00
mesh-admin f9be71b9ad Merge pull request 'A bed's forge takes the lab's shallow checkouts (issue 689)' (#103) from fix/689-a-shallow-checkout-reaches-the-forge into main 2026-10-14 00:52:06 +00:00
mesh-admin 307dd23d6d Merge pull request 'The lab's phone shows an HTML message as Telegram does, so a tap carries back the words shown (issue 692)' (#104) from fix/692-the-fake-phone-reads-html into main 2026-10-14 00:45:34 +00:00
jochen dcaf843b8a The lab's phone shows an HTML message as Telegram does, so a tap carries back the words shown (issue 692)
mesh/merge-gate head dcaf843b8a6e merged into trunk 1316a405a630 — pass: the change touches no module of the mesh's graph
mesh/repo-check head dcaf843b8a6e merged into trunk 1316a405a630 — pass: THE CHANGE ALTERS ITS OWN CHECK (asks/merge-check.sh): main's version judged it; t…
mesh/delivery delivered
Since mesh-catalog #278 the channel sends every message in HTML (issues 383, 639). The fake Bot API kept
the text as sent, tags and entities included, and gave that back on a tap; the channel compares a tap by
the plain words it showed, so the proof's tap was refused as "this message does not show what the mesh put
there". Telegram reads the markup and gives the plain words back; the fake now does too.
2026-10-14 02:37:06 +02:00
jochen 7ed7da1398 Let a bed's forge take the lab's shallow checkouts (issue 689)
mesh/delivery delivered
mesh/merge-gate head 7ed7da1398df merged into trunk 307dd23d6d2a — pass: the change touches no module of the mesh's graph
mesh/repo-check head 7ed7da1398df merged into trunk 307dd23d6d2a — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
The lab module clones every repository with --depth 50, and a fresh bare
repository refuses a push whose history ends in a cut, so the genesis bed
raised its anchor and forge and then skipped. The three beds that carry a
checkout into a forge now share one helper that sets receive.shallowUpdate
on the bare repository. Unshallowing instead cannot work in a lab run: the
forge credential is removed once the run's clones are done.
2026-10-14 02:21:21 +02:00
mesh-admin 1316a405a6 Merge pull request 'Register R164: a credential only a person can give was made anyway (hq issue 164)' (#102) from fix/164-replay into main 2026-10-13 11:21:00 +00:00
jochen a0754ab06d Point R164 at the merge of its fix (mesh-controller #298) now it is merged
mesh/merge-gate head a0754ab06d59 merged into trunk a8d86bb10b9d — pass: the change touches no module of the mesh's graph
mesh/repo-check head a0754ab06d59 merged into trunk a8d86bb10b9d — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
2026-10-13 13:11:01 +02:00
jochen adaf525a88 Register R164: a credential only a person can give was made anyway (hq issue 164)
The replay lives in mesh-controller (TestReplay164) and is proved against the main its fix branched
from until the fix merges: go run ./cmd/prove R164 says it fails before and passes on the fix.
2026-10-13 13:10:03 +02:00
mesh-admin a8d86bb10b Merge pull request 'Register R614: a new SDK planned alone, rebuilding neither its toolchain nor its bundles (hq issue 614)' (#100) from fix/614-replay into main 2026-10-13 09:40:22 +00:00
jochen 9deefc6370 R614's fix is the merge of mesh-controller #296 (hq issue 614)
mesh/merge-gate head 9deefc637065 merged into trunk d7098c4b2781 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 9deefc637065 merged into trunk d7098c4b2781 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
Proved against the merge: fails on its first parent, passes on it.
2026-10-13 11:36:37 +02:00
mesh-admin d7098c4b27 Merge pull request 'Serve the Go toolchain's repository to the genesis bed's installer (issue 616)' (#101) from fix/616-the-forge-serves-the-go-toolchain into main 2026-10-13 09:23:08 +00:00
jochen d21949c041 Serve the Go toolchain's repository to the genesis bed's installer (hq issue 616)
mesh/merge-gate head d21949c04150 merged into trunk f71250c3ddd5 — pass: the change touches no module of the mesh's graph
mesh/repo-check head d21949c04150 merged into trunk f71250c3ddd5 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
Genesis's base step now builds mesh-tools-go beside mesh-tools, from the repository beside --tools-source;
the bed's forge served only the four repositories genesis cloned before, so the step would find nothing
there.
2026-10-13 11:13:58 +02:00
jochen 7806a42c45 Register R614: a new SDK planned alone, rebuilding neither its toolchain nor the bundles in it (hq issue 614)
mesh/merge-gate out of date: judged against trunk f71250c3ddd5, the trunk is now d7098c4b2781; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f71250c3ddd5, the trunk is now d7098c4b2781; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Its replay (mesh-controller #296) holds the plan of 2026-10-13, "builds mesh-sdk, sent
nowhere", against the fix that reads a built module's build edges from what is kept of it.
Proved: fails on main, passes on the fix.
2026-10-13 11:10:16 +02:00
mesh-admin f71250c3dd Merge pull request 'Measure that the postgres module's image initialises where genesis raises the store (issue 609, hq ADR 0338 §5)' (#99) from fix/609-the-store-genesis-raises-initialises into main 2026-10-13 08:55:43 +00:00
jochen ba2806b985 Measure that the postgres module's image initialises where genesis now raises the store (hq issue 609, ADR 0338 §5)
mesh/merge-gate head ba2806b985ce merged into trunk 9497dda9291c — pass: the change touches no module of the mesh's graph
mesh/repo-check head ba2806b985ce merged into trunk 9497dda9291c — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
The genesis bundle raises the store as the postgres module's server: its pgvector image, its data in a
directory made before the store starts and owned as the module declares it, its password from a file.
The decision names one measurement before it is relied on: that image, on such a directory, empty,
initialises a store that answers with that password. The image and the owner are read from the
catalogue's manifest, so the measurement moves with the module's pin.
2026-10-13 10:47:34 +02:00
mesh-admin 9497dda929 Merge pull request 'Register the replay of issue 239: a module's name is not taken over by another repository (hq issue 239)' (#98) from fix/239-replay into main 2026-10-13 05:37:14 +00:00
jochen 3b4ba29bec Register the replay of issue 239: a module's name is not taken over by another repository
mesh/merge-gate head 3b4ba29becc8 merged into trunk 3eb48c949fc2 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 3b4ba29becc8 merged into trunk 3eb48c949fc2 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
mesh/delivery-group group fix/239-replay merged: every member is merged
Its replay (mesh-controller #294) holds the photo app's outage of 2026-10-04 against the fix that
refuses a build from a repository other than the module's own (mesh-controller #174).
2026-10-13 07:25:26 +02:00
mesh-admin 3eb48c949f Merge pull request 'Register R363: a README-only merge made a new digest of an unchanged Go program (hq issue 363)' (#97) from fix/363-replay into main 2026-10-13 05:08:19 +00:00
jochen 5f13bf0aff Register R363: a README-only merge made a new digest of an unchanged Go program (hq issue 363)
mesh/merge-gate head 5f13bf0aff9c merged into trunk 73800225c3d0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 5f13bf0aff9c merged into trunk 73800225c3d0 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
mesh/delivery-group group fix/363-replay closed: a member was closed before the group merged
Proved with replays/cmd/prove: it fails on mesh-controller #292's first parent
and passes on its merge.
2026-10-13 07:01:47 +02:00
mesh-admin 73800225c3 Merge pull request 'Register R348: a fault from before a send failed the builds sent after it (hq issue 348)' (#96) from fix/348-replay into main 2026-10-13 04:39:44 +00:00
jochen becff66e0a Register R348: a fault from before a send failed the builds sent after it (hq issue 348)
mesh/merge-gate head becff66e0a8a merged into trunk 2aebf8f40e25 — pass: the change touches no module of the mesh's graph
mesh/repo-check head becff66e0a8a merged into trunk 2aebf8f40e25 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
mesh/delivery-group group fix/348-replay merged: every member is merged
Proved with replays/cmd/prove: it fails on mesh-controller #179's first parent
and passes on the fix. The resolver's half (mesh-catalog #161) has no replay:
the module had no Go before it, and the binding's loss never reproduced past
its burst; the entry says so.
2026-10-13 06:09:50 +02:00
mesh-admin 2aebf8f40e Merge pull request 'Register R349: a merge acted on late superseded the merge after it (hq issue 349)' (#95) from fix/349-replay into main 2026-10-13 03:54:15 +00:00
mesh-admin 0ca7a83e0b Merge pull request 'Register R92 as a bed: genesis on a runtime that trusts no registry (hq issue 92) — NOT PROVED, blocked by issue 146' (#85) from fix/92-replay into main 2026-10-13 03:49:20 +00:00
jochen d16c3db9d2 Point R349 at the replay that reaches main's batch cut (review of mesh-controller #290)
mesh/merge-gate head d16c3db9d2d0 merged into trunk 0ca7a83e0b62 — pass: the change touches no module of the mesh's graph
mesh/repo-check head d16c3db9d2d0 merged into trunk 0ca7a83e0b62 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
mesh/delivery-group group fix/349-replay merged: every member is merged
2026-10-13 05:49:05 +02:00
jochen 39ec294853 Register R349: a merge acted on late superseded the merge after it (hq issue 349)
mesh/merge-gate head 39ec2948539f merged into trunk 6ba65a12f218 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 39ec2948539f merged into trunk 6ba65a12f218 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery-group group fix/349-replay checking: 1 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
Issue 349 is a core issue and resolves with a replay. The replay lives in
mesh-controller (#290) and is proved to fail on #179's parent and pass on it.
2026-10-13 05:22:56 +02:00
mesh-admin 6ba65a12f2 Merge pull request 'Register R340: a verb set what every agent session obeys (hq issue 340)' (#94) from fix/340-replay into main 2026-10-13 01:07:06 +00:00
jochen ea24e7890f Register R340: a verb set what every agent session obeys (hq issue 340)
mesh/merge-gate head ea24e7890f57 merged into trunk 51fa3a73ab3d — pass: the change touches no module of the mesh's graph
mesh/repo-check head ea24e7890f57 merged into trunk 51fa3a73ab3d — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
mesh/delivery-group group fix/340-replay merged: every member is merged
Issue 340 is a core issue and resolves with a replay; the test lives in
mesh-controller and is proved against the commit before #172.
2026-10-13 03:00:43 +02:00
jochen 0714f7a509 Pin R92 to today's genesis: main with every genesis fix, and the same without #91 (hq issue 92)
mesh/merge-gate head 0714f7a50933 merged into trunk 6ba65a12f218 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 0714f7a50933 merged into trunk 6ba65a12f218 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
At the fix's merge no genesis got past step 6 (issue 146), so neither side of the replay reached the
push it replays. Fix is mesh-host main 0d11b4fc; Before is that main with #91 reverted, on the
unmerged branch replay/r92-before; the controller and catalogue at their mains on both sides.
2026-10-13 02:59:45 +02:00
jochen d03f87b00d Register R92 as a bed: genesis on a runtime that trusts no registry (hq issue 92)
The lab's base image trusts the documentation ranges as plain-HTTP registries, so no genesis bed could
see the push to an untrusted registry that issue 92 is about (reviews 73 and 76 on mesh-host #91). The
bed takes that trust off the anchor before the installer runs and asks the runtime it trusts loopback
only, then runs genesis with --registry 192.0.2.10:5000 and holds that the control plane reaches the
registry, trusted by the mesh's name too.

The prover gains a bed kind: it builds the node-engine and its installer at each commit, the carried
builder from the pinned controller, and raises the bed; a bed that never reached the step it replays
skips, and a skip is an error, never the fault reproduced.

Not proved yet: genesis stops at step 6 on both sides, because the temporary controller has no
MESH_BUS_NATS and the bus template is refused by the installer (issue 146).
2026-10-13 02:57:21 +02:00
mesh-admin 51fa3a73ab Merge pull request 'Name the build agent where genesis needs the module that builds (issue 596)' (#93) from fix/596-genesis-names-the-build-agent into main 2026-10-13 00:57:03 +00:00
mesh-admin 383176a7ea Merge pull request 'Register R279 and R412: a folder could not follow its module's account; a seat named no paths (hq issues 279, 412)' (#90) from fix/279-412-replays into main 2026-10-13 00:45:30 +00:00
jochen 671d56a08a Register R279 and R412: a folder could not follow its module's account; a seat named no paths (hq issues 279, 412)
mesh/merge-gate head 671d56a08a47 merged into trunk b7f46559f053 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 671d56a08a47 merged into trunk b7f46559f053 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
mesh/delivery-group group fix/279-412-replays merged: every member is merged
Both proved with cmd/prove: each fails on the commit before its fix (mesh-controller #260, #267) and passes on it.
The replays' files are in mesh-controller fix/279-412-replays (3bdab29).
2026-10-13 02:40:08 +02:00
mesh-admin b7f46559f0 Merge pull request 'Register R166: a requirement could not be optional (hq issue 166)' (#92) from fix/166-replay into main 2026-10-13 00:38:10 +00:00
jochen abc022eece Name the build agent where genesis needs the module that builds (hq issue 596)
mesh/merge-gate head abc022eece61 merged into trunk 383176a7eab3 — pass: the change touches no module of the mesh's graph
mesh/repo-check head abc022eece61 merged into trunk 383176a7eab3 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
The beds checked the catalogue for builder/module.json before running genesis and the mesh beds held
'builder' among a raised mesh's modules; the catalogue retired builder for build-agent on 2026-10-03
(ADR 0190), so every genesis bed stopped before the installer ran.
2026-10-13 02:29:19 +02:00
jochen 87eca0c9a9 Register R166: a requirement could not be optional (hq issue 166)
mesh/merge-gate head 87eca0c9a9e8 merged into trunk e5ccf484684d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 87eca0c9a9e8 merged into trunk e5ccf484684d — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
Issue 166 resolves with its replay. TestReplay166 in the controller (mesh-controller #287) uses only
what the catalogue had before #281, so the prover lays it over the commit before: it fails there and
passes on the fix.
2026-10-13 02:28:18 +02:00
mesh-admin e5ccf48468 Merge pull request 'Raise genesis past its broker step and hold that the bus is kept (issue 590, hq ADR 0333 §5)' (#91) from fix/590-genesis-keeps-its-bus into main 2026-10-13 00:22:07 +00:00
jochen c7c7268db0 Raise genesis past its broker step and hold that the bus is kept (hq issue 590, ADR 0333 §5)
mesh/merge-gate head c7c7268db08c merged into trunk d3ef6fa44caf — pass: the change touches no module of the mesh's graph (run 2)
mesh/repo-check head c7c7268db08c merged into trunk d3ef6fa44caf — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
The bed runs the installer from bare twice: first with a catalogue missing the nats module, so it stops
at the broker step by name and the bed reads the bus there; then with the whole catalogue, through the
step. It passes only when exactly one bus container runs, it is mesh-broker-nats on the module's image,
every stream that held something before the step is still the same stream, and the mesh still hears
from the first node. A first run that never reaches the step skips with the reason; a second run that
stops at the step fails.

The judgement is a pure function, test/integration/bus-kept.ts, held by unit tests without a lab.
2026-10-13 02:13:35 +02:00
mesh-admin d3ef6fa44c Merge pull request 'Name the one genesis bundle in every bed and the README (issue 146, ADR 0331)' (#89) from fix/146-one-genesis-bundle into main 2026-10-12 21:12:13 +00:00
jochen 1af6efb356 Name the one genesis bundle in every bed and the README (hq issue 146, ADR 0331)
mesh/merge-gate head 1af6efb35677 merged into trunk 782aa9f73791 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 1af6efb35677 merged into trunk 782aa9f73791 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
mesh-host now carries one genesis bundle, foundation-first-node.lock, on the bus the
mesh runs on; the NATS one it replaces is deleted. Beds that still named it would point
at a file that is gone, and the harness's rewrite for the previous broker's image
rewrote nothing.
2026-10-12 22:50:50 +02:00
mesh-admin 782aa9f737 Merge pull request 'Register R201: a plan sent the control node a controller older than the one running there (hq issue 201)' (#88) from fix/201-replay into main 2026-10-12 20:22:43 +00:00
jochen a96226f228 Register R201: a plan sent the control node a controller older than the one running there (hq issue 201)
mesh/merge-gate head a96226f22819 merged into trunk 5c1c6f56c9d9 — pass: the change touches no module of the mesh's graph
mesh/repo-check head a96226f22819 merged into trunk 5c1c6f56c9d9 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
Proved: fails on mesh-controller 84deb53^1, passes on 84deb53 (the fix, #273).
The replay is mesh-controller #275's TestReplay201ThePlansSend, since the
fix's own TestReplay201 reads what the fix added.
2026-10-12 21:18:59 +02:00
mesh-admin 5c1c6f56c9 Merge pull request 'Register R401 and R561: an outside secret nobody gave was made; a plan's id was pasted into SQL (hq issues 401, 561)' (#87) from fix/401-replay into main 2026-10-12 18:22:12 +00:00
jochen 3c655d83e1 Register R561: a plan's id was pasted into the SQL that reads it (hq issue 561)
mesh/merge-gate head 3c655d83e11e merged into trunk 24b833d19dad — pass: the change touches no module of the mesh's graph
mesh/repo-check head 3c655d83e11e merged into trunk 24b833d19dad — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery delivered
mesh/delivery-group group fix/401-replay merged: every member is merged
Proved: fails on mesh-controller f8daec0^1, passes on f8daec0 (the fix, #268).
2026-10-12 19:01:18 +02:00
jochen e63459dd70 Register R401: the send made a value for an own secret issued outside the mesh that nobody gave (hq issue 401)
mesh/delivery-group group fix/401-replay checking: 0 of 2 member(s) ready
mesh/merge-gate head e63459dd702a merged into trunk 24b833d19dad — pass: the change touches no module of the mesh's graph
mesh/repo-check head e63459dd702a merged into trunk 24b833d19dad — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript)…
mesh/delivery superseded: a newer head of the same pull request
Proved: fails on mesh-controller 1e28342^1, passes on 1e28342 (the fix, #262). The replay is mesh-controller #274's TestReplay401, since the fix's own test reads what the fix added.
2026-10-12 18:37:30 +02:00
mesh-admin 24b833d19d Merge pull request 'Register R341: a check judged the head alone and let a merge that does not compile through (hq issue 341)' (#86) from feat/341-a-merge-check-judges-the-merge into main 2026-10-12 16:11:30 +00:00
jschoubben 79108a72a0 Register R341: a check judged the head alone and let a merge that does not compile through
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh …
mesh/delivery delivered
mesh/delivery-group group feat/341-a-merge-check-judges-the-merge merged: every member is merged
Proved with replays/cmd/prove: TestReplay341 fails on the controller's main
and passes on its fix (novox/hq ADR 0323, issue 341).
2026-10-12 17:29:59 +02:00
mesh-admin 930b13ff47 Merge pull request 'Register R233: a declaration that restores the package manager applies while it does not answer (hq issue 233)' (#84) from fix/233-replay-2 into main 2026-10-12 14:53:24 +00:00
mesh-admin 39531a9972 Register R233: a declaration that restores the package manager applies while it does not answer (hq issue 233)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh …
mesh/delivery delivered
Proved with go run ./cmd/prove R233: fails on mesh-host#90's commit before
(36704b5), passes on the fix (a1f9856, mesh-host#90). Placed after R237,
which merged first (#83); replaces #82.
2026-10-12 14:41:40 +00:00
mesh-admin a97ba661d9 Merge pull request 'Register R237: assign's answer agrees with plan right after the act (hq issue 237)' (#83) from replay/r237-rebased into main 2026-10-12 14:37:45 +00:00
mesh-admin bd33abf3dd Register R237: assign's answer agrees with plan right after the act (hq issue 237)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh …
mesh/delivery delivered
Rebase of #81 (a020f60) onto main after #80 (R358) merged into the same
spot in register.go. The R237 block is byte-identical to the one approved
at a020f60; only its place in the file differs, now after R358.

Proved with go run ./cmd/prove R237: fails on mesh-controller #253's commit
before (72edb45), passes on the fix (5facd8b, mesh-controller #253).
TestReplay237 needs MESH_TEST_POSTGRES (inventory.ForTest raises a throwaway
database), same as the controller's other store-backed replays.
2026-10-12 14:24:32 +00:00
mesh-admin 1f008aeb18 Merge pull request 'Register R358: a parent swapped for a link leaves its target untouched' (#80) from replay/r358 into main 2026-10-12 13:46:20 +00:00
jochen 26a0f45269 Register R358: a parent swapped for a link leaves its target untouched (hq issue 358)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh …
mesh/delivery delivered
Proved with go run ./cmd/prove R358: fails on mesh-host#87's commit before
(dce0eef), passes on the fix (36704b5, mesh-host#87).
2026-10-12 15:23:17 +02:00
mesh-admin 9bd51f3c6c Merge pull request 'Register R294: a quote in a command line's value is said, and an open quote refused naming where (hq issue 294)' (#79) from replay/294-quote-in-a-quoted-value into main 2026-10-10 17:40:57 +00:00
jochen 8c78bcf9ce Register R294: a quote in a command line's value is said, and an open quote refused naming where (hq issue 294)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh …
mesh/delivery delivered
2026-10-10 19:31:32 +02:00
mesh-admin 858b15a1a3 Merge pull request 'Register R368: a moved path is no shrink, and a shrink at the new path still is (hq issue 368)' (#77) from replay/368-a-moved-path-is-no-shrink into main 2026-10-10 16:32:48 +00:00
jochen b3d6d65db1 Merge main into replay/368: the asks check fixed for issue 393 (#78)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh …
mesh/delivery delivered
2026-10-10 18:27:49 +02:00
mesh-admin c48d68ac92 Merge pull request 'asks: a replayed tap is said as the answer already recorded (issue 393)' (#78) from fix/393-asks-replayed-tap into main 2026-10-10 16:27:21 +00:00
jochen af0a32a780 asks: a replayed tap is said as the answer already recorded, on its acknowledgement (issue 393)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh …
mesh/delivery delivered
Since issue 372 the router answers the same answer again from the same
account with 'You already chose Approve at HH:MM: recorded.', on the
tap's acknowledgement rather than in the chat. The proof from #74 still
waited for a chat refusal, so it failed on every mesh-lab pull request.
The lab's phone now keeps a tap's acknowledgement words, and the replay
subtest waits for them; its checks that nothing more is done stay.
2026-10-10 18:22:22 +02:00
jochen 95736b0eab Register R368: a moved path is no shrink, and a shrink at the new path still is (hq issue 368)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check fail: merge-check.sh (typescript), replays/merge-check.sh (go) passed; its asks/merge-check.sh failed: --- FAIL: TestTheOperatorsAnswerEndT…
mesh/delivery superseded: a newer head of the same pull request
2026-10-10 18:09:19 +02:00
mesh-admin 037bc32a6f Merge pull request 'Prove the operator's answers end to end before anybody uses them (hq ADR 0259)' (#74) from proofs/asks-answered-on-the-phone into main 2026-10-10 12:15:05 +00:00
mesh-admin 22043a3c86 Merge pull request 'Register R362: two merges seconds apart are delivered by one walk without a person (hq issue 362)' (#75) from replays/362-two-merges-seconds-apart into main 2026-10-10 11:14:08 +00:00
jochen 719ca7ae0b R362: the fix's commit after its branch was rebuilt without a binary (13ffae7)
mesh/delivery delivered
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go)
2026-10-10 13:06:01 +02:00
jochen bdb6efa8c6 Register R362: two merges seconds apart are delivered by one walk without a person
mesh/repo-check pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go)
mesh/merge-gate pass: the change touches no module of the mesh's graph
mesh/delivery superseded: a newer head of the same pull request
Proved with cmd/prove: fails on 52f34d3, passes on f8c4bc5 (mesh-catalog #171).
2026-10-10 12:55:51 +02:00
68 changed files with 2312 additions and 262 deletions
+4 -4
View File
@@ -148,10 +148,10 @@ is written down here rather than reconstructed a third time.
# built for \"\", which is not a system it knows" — which reads like a broken bundle and is not
# (novox/hq 04-ISSUES/146).
export MESH_LAB_HOST_BINARY=<mesh-host>/mesh-host
# The bundle that raises the bus the mesh runs on. The other one in that directory raises the
# predecessor's broker and a control plane that crash-loops on a missing MESH_BUS_NATS — which
# looks like a broken lab and is a bundle nobody moved (novox/hq 04-ISSUES/146).
export MESH_LAB_BUNDLE=<mesh-host>/examples/foundation-first-node-nats.lock
# The one genesis bundle, on the bus the mesh runs on (novox/hq ADR 0331). The bundle that raised the
# predecessor's broker, and a control plane that crash-looped on a missing MESH_BUS_NATS, is gone
# (novox/hq issue 146).
export MESH_LAB_BUNDLE=<mesh-host>/examples/foundation-first-node.lock
export MESH_LAB_MODULES=<mesh-controller>/examples/modules
export MESH_LAB_BUILDER=<mesh-controller>/build/mesh-builder # build/, which is git-ignored
+70 -1
View File
@@ -3,8 +3,10 @@ package asks
import (
"encoding/json"
"fmt"
"html"
"net/http"
"net/http/httptest"
"regexp"
"strings"
"sync"
"testing"
@@ -27,6 +29,11 @@ type fakeTelegram struct {
msgID int64
taps int // answerCallbackQuery calls
deleted int
// toasts are the words a tap's acknowledgement showed over the chat (answerCallbackQuery), by the chat the
// tap came from: what the phone shows the tapper and never puts in the chat. Kept apart from sent, so a
// chat's messages stay what the chat holds.
toasts []tgMessage
tappedIn map[string]int64 // a tap's callback id to the chat it came from
}
// tgMessage is one message the bot sent or edited: its chat, its id, its words and its buttons (label →
@@ -95,6 +102,11 @@ func (f *fakeTelegram) serve(w http.ResponseWriter, r *http.Request) {
case "sendMessage", "editMessageText":
chat := int64(num(body["chat_id"]))
text, _ := body["text"].(string)
if mode, _ := body["parse_mode"].(string); mode == "HTML" {
// What a message shows, and what a tap on it carries back, is its text with the markup read: tags gone,
// entities unescaped — as the service gives it (novox/hq issue 692).
text = shownOfHTML(text)
}
m := tgMessage{Chat: chat, Text: text, Buttons: buttonsOf(body["reply_markup"]), At: time.Now()}
f.mu.Lock()
if method == "sendMessage" {
@@ -108,9 +120,18 @@ func (f *fakeTelegram) serve(w http.ResponseWriter, r *http.Request) {
f.t.Logf("phone: %s to %d (#%d): %q %v", method, chat, m.ID, firstLine(text), m.Buttons)
ok(map[string]any{"message_id": m.ID, "chat": map[string]any{"id": chat, "type": "private"}, "text": text})
case "answerCallbackQuery":
id, _ := body["callback_query_id"].(string)
text, _ := body["text"].(string)
f.mu.Lock()
f.taps++
chat, known := f.tappedIn[id]
if known && text != "" {
f.toasts = append(f.toasts, tgMessage{Chat: chat, Text: text, At: time.Now()})
}
f.mu.Unlock()
if text != "" {
f.t.Logf("phone: answerCallbackQuery %s in %d: %q", id, chat, firstLine(text))
}
ok(true)
case "deleteMessage":
f.mu.Lock()
@@ -152,7 +173,14 @@ func (f *fakeTelegram) tapped(from int64, chat int64, chatType string, onMessage
// tappedOn is a tap on a message showing the words given: what a message changed under the operator's thumb,
// or forwarded and edited elsewhere, carries.
func (f *fakeTelegram) tappedOn(from int64, chat int64, chatType string, onMessage int64, data, text string) {
f.push(map[string]any{"callback_query": map[string]any{"id": fmt.Sprintf("cb-%d", f.nextID()), "data": data,
id := fmt.Sprintf("cb-%d", f.nextID())
f.mu.Lock()
if f.tappedIn == nil {
f.tappedIn = map[string]int64{}
}
f.tappedIn[id] = chat
f.mu.Unlock()
f.push(map[string]any{"callback_query": map[string]any{"id": id, "data": data,
"from": map[string]any{"id": from, "is_bot": false, "first_name": fmt.Sprintf("Account %d", from)},
"message": map[string]any{"message_id": onMessage, "text": text,
"chat": map[string]any{"id": chat, "type": chatType}}}})
@@ -197,6 +225,26 @@ func (f *fakeTelegram) waitFor(what string, chat int64, since time.Time, within
return tgMessage{}
}
// toldTapper waits for words the phone showed in a chat at or after a time, as a message in it or as a tap's
// acknowledgement over it: where the channel says the router's word on a tap (issue 372).
func (f *fakeTelegram) toldTapper(what string, chat int64, since time.Time, within time.Duration, match func(string) bool) string {
f.t.Helper()
deadline := time.Now().Add(within)
for time.Now().Before(deadline) {
f.mu.Lock()
for _, m := range append(append([]tgMessage{}, f.sent...), f.toasts...) {
if m.Chat == chat && !m.At.Before(since) && match(m.Text) {
f.mu.Unlock()
return m.Text
}
}
f.mu.Unlock()
time.Sleep(100 * time.Millisecond)
}
f.t.Fatalf("the phone never showed %s (chat %d) within %s; it showed:\n%s", what, chat, within, f.transcript())
return ""
}
// since is every message to a chat sent at or after a time.
func (f *fakeTelegram) since(chat int64, at time.Time) []tgMessage {
f.mu.Lock()
@@ -218,6 +266,9 @@ func (f *fakeTelegram) transcript() string {
fmt.Fprintf(&b, " %s to %d #%d edited=%v %q %v\n", m.At.Format("15:04:05.000"), m.Chat, m.ID, m.Edited,
firstLine(m.Text), m.Buttons)
}
for _, m := range f.toasts {
fmt.Fprintf(&b, " %s to %d tap acknowledged %q\n", m.At.Format("15:04:05.000"), m.Chat, firstLine(m.Text))
}
return b.String()
}
@@ -280,3 +331,21 @@ func firstLine(s string) string {
}
return s
}
var htmlTag = regexp.MustCompile(`</?[a-zA-Z][^>]*>`)
// shownOfHTML is a text sent in Telegram's HTML as the service shows it and gives it back: its tags read and taken
// out, its entities written as the characters they stand for.
func shownOfHTML(text string) string {
return html.UnescapeString(htmlTag.ReplaceAllString(text, ""))
}
// The fake phone shows a message sent in HTML as Telegram does (novox/hq issue 692): the words without their
// markup, so a tap carries back what the channel digests, as on the real service.
func TestTheFakePhoneShowsHTMLAsTelegramDoes(t *testing.T) {
in := "<b>Flip the lab&#39;s switch?</b>\nA &lt;b&gt; &amp; c\n<code>+ k: v</code>\n\n<blockquote expandable>Asked by lab-asker.</blockquote>"
want := "Flip the lab's switch?\nA <b> & c\n+ k: v\n\nAsked by lab-asker."
if got := shownOfHTML(in); got != want {
t.Errorf("%q, want %q", got, want)
}
}
+17
View File
@@ -0,0 +1,17 @@
package asks
import (
"os/exec"
"testing"
)
// live-acceptance.sh is run by a person at the terminal and by no test: its syntax is judged here, as the lab's script
// judged it with `sh -n` (novox/hq ADR 0529).
func TestTheLiveAcceptanceScriptIsValidShell(t *testing.T) {
if _, err := exec.LookPath("sh"); err != nil {
t.Skip("NOT TESTED HERE: no sh on the path")
}
if out, err := exec.Command("sh", "-n", "live-acceptance.sh").CombinedOutput(); err != nil {
t.Fatalf("live-acceptance.sh is not valid shell: %v\n%s", err, out)
}
}
-37
View File
@@ -1,37 +0,0 @@
#!/bin/sh
# mesh-check-toolchain: go
#
# The proof of the operator's answers (novox/hq ADR 0259), the Go part of the lab's own check: run by the
# build seat in the mesh's Go toolchain, its own container, because the lab's merge-check.sh declares it
# (`# mesh-check-also: go asks/merge-check.sh`) — and by hand: `sh asks/merge-check.sh`.
#
# Formatted, vetted, and run against the clones beside the lab: on the build seat those are the controller,
# the runtime and the catalogue **the mesh runs**, so every pull request of the lab proves the answers the
# operator would get today. By hand, MESH_LAB_REPOS, or MESH_LAB_CONTROLLER, MESH_LAB_TOOLS and
# MESH_LAB_CATALOGUE, point it at branches under review. It takes about four minutes when it runs.
#
# **Said, never passed silently**: where a clone is missing, or the controller predates the proof's fixture,
# it says NOT RUN and why, in capitals, and passes; a proof that runs and fails, fails the check.
set -eu
cd "$(dirname "$0")"
export GOPRIVATE=git.novox.be
unformatted=$(gofmt -l .)
if [ -n "$unformatted" ]; then
echo "not gofmt'd:"
echo "$unformatted"
exit 1
fi
sh -n live-acceptance.sh
go vet ./...
out=$(mktemp)
status=0
go test -count=1 -timeout 25m -v -run '^TestTheOperatorsAnswerEndToEnd$' . >"$out" 2>&1 || status=$?
grep -E '^(--- | --- |ok|FAIL|PASS)' "$out" || true
if grep -q 'NOT RUN' "$out"; then
echo "NOT RUN: THE PROOF OF THE OPERATOR'S ANSWERS DID NOT RUN HERE:"
grep -o 'NOT RUN: .*' "$out" | head -3
fi
[ "$status" -eq 0 ] || tail -60 "$out"
rm -f "$out"
exit "$status"
+8 -2
View File
@@ -1,3 +1,5 @@
//go:build integration
package asks
import (
@@ -178,11 +180,15 @@ func TestTheOperatorsAnswerEndToEnd(t *testing.T) {
})
// --- 4. A replayed tap, and a redelivered warrant, do nothing more.
// The replay is the operator's own answer again: the router says it is the one already recorded (on the tap's
// acknowledgement, issue 372), not refused as if the approval failed, and it is no second warrant or act.
t.Run("a replayed tap and a redelivered warrant do nothing more", func(t *testing.T) {
tapped := time.Now()
l.tg.tapped(operatorAccount, operatorAccount, "private", firstMessage.ID, button(firstMessage, "Approve"))
l.tg.waitFor("the refusal of a replayed tap", operatorAccount, tapped, 90*time.Second,
func(m tgMessage) bool { return strings.Contains(m.Text, "That answer was not taken") })
l.tg.toldTapper("that a replayed tap is the answer already recorded", operatorAccount, tapped, 90*time.Second,
func(text string) bool {
return strings.Contains(text, "You already chose Approve") && strings.Contains(text, "recorded")
})
time.Sleep(2 * time.Second)
if n := a.wordsOn("a1"); n != 1 {
t.Errorf("the router said %d words on one ask", n)
-37
View File
@@ -1,37 +0,0 @@
#!/bin/sh
# mesh-check-toolchain: typescript
# mesh-check-also: go replays/merge-check.sh
# mesh-check-also: go asks/merge-check.sh
#
# The lab's own check (novox/hq ADR 0237 as amended): the second layer of a pull request's merge check,
# `mesh/repo-check`, run by the build seat — and by hand.
#
# The mesh builds no module from the lab, so no gate runs for its pull requests (`mesh/merge-gate` says the
# change touches no module of the graph); this is all that is checked before a change to it merges. The lab
# is in two languages, and each part runs in its own toolchain's container (novox/hq issue 302):
#
# - this script, in the TypeScript toolchain: installed from the lock file, type-checked, sources and
# tests, and the unit suite;
# - replays/merge-check.sh, in the Go toolchain, declared above: the replays register and its prover
# formatted, vetted and compiled, and the register's own tests;
# - asks/merge-check.sh, in the Go toolchain, declared above: the proof of the operator's answers, run
# against the controller, runtime and catalogue cloned beside the lab (novox/hq ADR 0259).
#
# **Said, never passed silently**: the integration suite raises a lab on a workstation, and the replays
# raise containers through the container runtime — neither is given to code nobody has approved. The
# replays run from the lab's main, reviewed, in every gate of a core change; a change to them is proven by
# `go run ./replays/cmd/prove` by hand before it merges.
set -eu
npm ci --no-audit --no-fund --loglevel=error
npm run typecheck
npm test
# By hand, with Go at hand, the Go part runs here too; on the build seat it runs in the Go toolchain.
if command -v go >/dev/null 2>&1; then
sh replays/merge-check.sh
sh asks/merge-check.sh
else
echo "replays/ and asks/ (Go) are checked by their merge-check.sh, which the build seat runs in the Go toolchain"
fi
echo "NOT RUN HERE: the integration suite and the replays — they need a lab and the container runtime"
+1
View File
@@ -8,6 +8,7 @@
},
"scripts": {
"typecheck": "tsc --noEmit && tsc --noEmit -p tsconfig.test.json",
"pretest": "npm run typecheck",
"test": "node --test --experimental-strip-types 'test/*.test.ts'",
"test:integration": "node --experimental-strip-types src/cli.ts suite",
"check": "npm run typecheck && npm test && npm run test:integration",
+67
View File
@@ -0,0 +1,67 @@
//go:build integration
package replays
import (
"context"
"fmt"
"strings"
"testing"
"time"
)
// **R-studio — a check that asks an address the program does not bind fails the bed, not a machine**
// (novox/hq ADR 0240 Phase D, done when). The hosted database suite's studio binds the address the runtime
// puts in HOSTNAME, so it answered only on its network address while its image's own check asked localhost:
// unhealthy for ever while working, until the module set HOSTNAME=0.0.0.0. Adopted by name without proof, it
// would have put back a good build.
//
// The replay is that image in miniature: a web server that binds $HOSTNAME's address, and an image check that
// asks localhost. Adopted as the module declared it before the fix, the bed fails it; with the fix, it proves
// it. The image is built here and removed after, with everything the bed made.
func TestBedFailsTheStudiosFalseUnhealthy(t *testing.T) {
docker, ok := DockerFromEnv()
if !ok {
t.Skip("no container runtime: the studio is a container")
}
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Minute)
defer cancel()
if err := docker.Pull(ctx, "busybox:1.36"); err != nil {
t.Fatal(err)
}
tag := fmt.Sprintf("mesh-replay-studio:%d", time.Now().UnixNano())
if err := docker.Build(ctx, tag, StudioDockerfile); err != nil {
t.Fatal(err)
}
defer docker.RemoveImage(context.Background(), tag)
was := BedFloor
BedFloor = 15 * time.Second
defer func() { BedFloor = was }()
studio := func(env map[string]any) BedResource {
r := map[string]any{"id": "studio", "type": "container", "name": "supabase-studio", "image": tag,
"health": map[string]any{"kind": "runtime", "grace": "10s"}}
if env != nil {
r["env"] = env
}
return BedResource{Module: "supabase", ID: "studio", Resource: r, Listens: map[string]int{}}
}
before := docker.Prove(ctx, studio(nil), nil)
if !before.Failed || !strings.Contains(before.Said, "does not see it") {
t.Fatalf("the studio's false unhealthy was not failed on the bed: %+v", before)
}
t.Logf("before the fix: %s", before.Said)
after := docker.Prove(ctx, studio(map[string]any{"HOSTNAME": "0.0.0.0"}), nil)
if !after.Proved {
t.Fatalf("the studio with HOSTNAME=0.0.0.0 was not proved: %+v", after)
}
t.Logf("with the fix: %s", after.Said)
}
// StudioDockerfile is the studio in miniature: it serves on the address HOSTNAME names, and its own check
// asks localhost, as the studio's image does.
const StudioDockerfile = `FROM busybox:1.36
RUN mkdir -p /www && echo studio > /www/index.html
HEALTHCHECK --interval=5s --timeout=2s --retries=2 CMD wget -q -O /dev/null http://localhost:3000/ || exit 1
CMD addr=$(grep -w "$HOSTNAME" /etc/hosts | head -n 1 | cut -f 1); exec httpd -f -p "${addr:-$HOSTNAME}:3000" -h /www
`
+7 -57
View File
@@ -2,7 +2,6 @@ package replays
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
@@ -58,62 +57,6 @@ func TestBedProvesEveryChangedHealthCheck(t *testing.T) {
}
}
// **R-studio — a check that asks an address the program does not bind fails the bed, not a machine**
// (novox/hq ADR 0240 Phase D, done when). The hosted database suite's studio binds the address the runtime
// puts in HOSTNAME, so it answered only on its network address while its image's own check asked localhost:
// unhealthy for ever while working, until the module set HOSTNAME=0.0.0.0. Adopted by name without proof, it
// would have put back a good build.
//
// The replay is that image in miniature: a web server that binds $HOSTNAME's address, and an image check that
// asks localhost. Adopted as the module declared it before the fix, the bed fails it; with the fix, it proves
// it. The image is built here and removed after, with everything the bed made.
func TestBedFailsTheStudiosFalseUnhealthy(t *testing.T) {
docker, ok := DockerFromEnv()
if !ok {
t.Skip("no container runtime: the studio is a container")
}
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Minute)
defer cancel()
if err := docker.Pull(ctx, "busybox:1.36"); err != nil {
t.Fatal(err)
}
tag := fmt.Sprintf("mesh-replay-studio:%d", time.Now().UnixNano())
if err := docker.Build(ctx, tag, StudioDockerfile); err != nil {
t.Fatal(err)
}
defer docker.RemoveImage(context.Background(), tag)
was := BedFloor
BedFloor = 15 * time.Second
defer func() { BedFloor = was }()
studio := func(env map[string]any) BedResource {
r := map[string]any{"id": "studio", "type": "container", "name": "supabase-studio", "image": tag,
"health": map[string]any{"kind": "runtime", "grace": "10s"}}
if env != nil {
r["env"] = env
}
return BedResource{Module: "supabase", ID: "studio", Resource: r, Listens: map[string]int{}}
}
before := docker.Prove(ctx, studio(nil), nil)
if !before.Failed || !strings.Contains(before.Said, "does not see it") {
t.Fatalf("the studio's false unhealthy was not failed on the bed: %+v", before)
}
t.Logf("before the fix: %s", before.Said)
after := docker.Prove(ctx, studio(map[string]any{"HOSTNAME": "0.0.0.0"}), nil)
if !after.Proved {
t.Fatalf("the studio with HOSTNAME=0.0.0.0 was not proved: %+v", after)
}
t.Logf("with the fix: %s", after.Said)
}
// StudioDockerfile is the studio in miniature: it serves on the address HOSTNAME names, and its own check
// asks localhost, as the studio's image does.
const StudioDockerfile = `FROM busybox:1.36
RUN mkdir -p /www && echo studio > /www/index.html
HEALTHCHECK --interval=5s --timeout=2s --retries=2 CMD wget -q -O /dev/null http://localhost:3000/ || exit 1
CMD addr=$(grep -w "$HOSTNAME" /etc/hosts | head -n 1 | cut -f 1); exec httpd -f -p "${addr:-$HOSTNAME}:3000" -h /www
`
// What the bed proves is what the change touches: a long-running container whose declared check or image
// changed, or that is new; never a step, never one left as it was.
func TestTheBedProvesWhatTheChangeTouches(t *testing.T) {
@@ -159,3 +102,10 @@ func TestTheBedProvesWhatTheChangeTouches(t *testing.T) {
t.Fatalf("the bed would prove %v; want the new module's checked container and the one whose image moved", got)
}
}
func orDefault(s, fallback string) string {
if s == "" {
return fallback
}
return s
}
+2
View File
@@ -1,3 +1,5 @@
//go:build integration
package replays
import (
+143 -2
View File
@@ -7,7 +7,9 @@
// The clones are the core repositories, side by side as the lab expects them. A replay in its repository
// is laid over a worktree of each commit and run with go test; the bus replay runs against a server of
// the release the catalogue pinned at each commit, raised for it; the resolver replay with the catalogue's
// resolver at each commit. MESH_TEST_POSTGRES is passed through to the controller's.
// resolver at each commit; a bed of the lab is raised on its hypervisor with the node-engine and its
// installer built at each commit (a workstation where `mesh-lab check` passes). MESH_TEST_POSTGRES is passed
// through to the controller's.
package main
import (
@@ -20,6 +22,8 @@ import (
"os"
"os/exec"
"path/filepath"
"regexp"
"strconv"
"strings"
"time"
@@ -63,7 +67,12 @@ func main() {
// run is the replay at one commit: passes, fails, or absent (the check it replays did not exist).
func run(r replays.Replay, repos, at string, fixed bool) (string, string) {
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
limit := 15 * time.Minute
if r.Kind == replays.Bed {
// Two machines raised, a foundation pulled and a control plane built inside one of them.
limit = 90 * time.Minute
}
ctx, cancel := context.WithTimeout(context.Background(), limit)
defer cancel()
repo := filepath.Join(repos, r.Repository)
tree, err := os.MkdirTemp("", "prove-"+r.ID+"-")
@@ -77,6 +86,8 @@ func run(r replays.Replay, repos, at string, fixed bool) (string, string) {
defer exec.Command("git", "-C", repo, "worktree", "remove", "--force", tree).Run()
switch r.Kind {
case replays.Bed:
return aBed(ctx, r, repos, tree, fixed)
case replays.InRepository, replays.Gate, replays.Liveness:
home := filepath.Join(repos, r.Repository)
if r.Home != "" {
@@ -155,6 +166,136 @@ func run(r replays.Replay, repos, at string, fixed bool) (string, string) {
return "error", "no such kind"
}
// aBed raises the lab's bed at one commit of the fixed repository (novox/hq issue 92): the node-engine and its
// installer built from it, the installer carrying a builder built from the controller the replay pins, and
// the bed run with them, the pinned controller to build from and the pinned catalogue to read.
func aBed(ctx context.Context, r replays.Replay, repos, tree string, fixed bool) (string, string) {
beside := map[string]string{}
for _, also := range r.With {
ref := also.Fix
if !fixed {
ref = also.Fix + "^1"
if also.Before != "" {
ref = also.Before
}
}
other, err := os.MkdirTemp("", "prove-"+r.ID+"-"+also.Repository+"-")
if err != nil {
return "error", err.Error()
}
defer os.RemoveAll(other)
clone := filepath.Join(repos, also.Repository)
if out, err := exec.CommandContext(ctx, "git", "-C", clone, "worktree", "add", "--quiet", "--detach", other,
ref).CombinedOutput(); err != nil {
return "error", string(out)
}
defer exec.Command("git", "-C", clone, "worktree", "remove", "--force", other).Run()
beside[also.Repository] = other
}
controller, catalogue := beside["mesh-controller"], beside["mesh-catalog"]
if controller == "" || catalogue == "" {
return "error", "a bed is raised with the controller and the catalogue it pins, and this one pins " +
"not both"
}
builder, err := builderImage(ctx, controller)
if err != nil {
return "error", err.Error()
}
host, installer := filepath.Join(tree, "mesh-host"), filepath.Join(tree, "mesh-bootstrap")
for _, step := range [][]string{
{"make", "build", "SYSTEM=arch"},
{"make", "bootstrap", "IMAGE=" + builder, "BOOTSTRAP_OUT=" + installer},
} {
cmd := exec.CommandContext(ctx, step[0], step[1:]...)
cmd.Dir = tree
if out, err := cmd.CombinedOutput(); err != nil {
return "error", fmt.Sprintf("%s: %v %s", strings.Join(step, " "), err, lastLines(string(out), 6))
}
}
lab, err := filepath.Abs("..")
if err != nil {
return "error", err.Error()
}
cmd := exec.CommandContext(ctx, "node", "--test", "--test-reporter=tap", "--experimental-strip-types", r.Test)
cmd.Dir = lab
cmd.Env = append(os.Environ(),
"MESH_LAB_HOST_BINARY="+host,
"MESH_LAB_BOOTSTRAP_BINARY="+installer,
// The template the installer knows how to make safe: it rotates the store's and the broker's
// well-known credentials, and refuses a template naming neither (the bus one names no broker).
"MESH_LAB_BUNDLE="+filepath.Join(tree, "examples", "foundation-first-node.lock"),
"MESH_LAB_CATALOG="+filepath.Join(catalogue, "modules"),
"MESH_REPLAY_CONTROLLER="+controller)
out, err := cmd.CombinedOutput()
return bedVerdict(string(out), err)
}
// builderImage is the builder the installer carries, built from the controller at the commit the replay
// pins, on the Go release its go.mod asks for; built once and kept under a tag naming the commit.
func builderImage(ctx context.Context, controller string) (string, error) {
head, err := exec.CommandContext(ctx, "git", "-C", controller, "rev-parse", "--short=12", "HEAD").Output()
if err != nil {
return "", err
}
tag := "mesh-builder:prove-" + strings.TrimSpace(string(head))
if exec.CommandContext(ctx, "docker", "image", "inspect", tag).Run() == nil {
return tag, nil
}
mod, err := os.ReadFile(filepath.Join(controller, "go.mod"))
if err != nil {
return "", err
}
release := ""
for _, line := range strings.Split(string(mod), "\n") {
if v, ok := strings.CutPrefix(line, "go "); ok {
parts := strings.Split(strings.TrimSpace(v), ".")
if len(parts) >= 2 {
release = parts[0] + "." + parts[1]
}
}
}
if release == "" {
return "", errors.New("the controller's go.mod names no Go release to build its builder on")
}
cmd := exec.CommandContext(ctx, "docker", "build", "-q", "--build-arg", "GO_BASE=golang:"+release+"-alpine",
"-f", "cmd/mesh-builder/Dockerfile", "-t", tag, ".")
cmd.Dir = controller
if out, err := cmd.CombinedOutput(); err != nil {
return "", fmt.Errorf("the builder could not be built from %s: %v %s", controller, err, lastLines(string(out), 6))
}
return tag, nil
}
// bedVerdict reads node's test runner: a skipped bed is an error — it never reached what it replays, or was
// not given what it raises — and never the fault reproduced.
func bedVerdict(out string, err error) (string, string) {
count := func(what string) int {
m := regexp.MustCompile(`(?m)^# ` + what + ` (\d+)$`).FindStringSubmatch(out)
if m == nil {
return -1
}
n, _ := strconv.Atoi(m[1])
return n
}
pass, fail, skipped := count("pass"), count("fail"), count("skipped")
switch {
case skipped > 0:
said := regexp.MustCompile(`(?m)# SKIP (.*)$`).FindStringSubmatch(out)
if said != nil {
return "error", "the bed skipped: " + said[1]
}
return "error", "the bed skipped: " + lastLines(out, 6)
case fail > 0:
if m := regexp.MustCompile(`(?m)^\s+error: [|>]?-?\s*\n?\s*(.+)$`).FindStringSubmatch(out); m != nil {
return "fails", strings.TrimSpace(m[1])
}
return "fails", lastLines(out, 6)
case pass > 0 && err == nil:
return "passes", lastLines(out, 3)
}
return "error", lastLines(out, 6)
}
func replayHere(ctx context.Context, r replays.Replay, env string) (string, string) {
cmd := exec.CommandContext(ctx, "go", "test", "-count=1", "-run", "^"+r.Test, r.Package)
cmd.Env = append(os.Environ(), env)
+2 -7
View File
@@ -1,3 +1,5 @@
//go:build integration
package replays
import (
@@ -111,13 +113,6 @@ func statementOf(path string) string {
return string(raw)
}
func orDefault(s, fallback string) string {
if s == "" {
return fallback
}
return s
}
func firstLineOf(s string) string {
line, _, _ := strings.Cut(strings.TrimSpace(s), "\n")
if len(line) > 400 {
+2 -1
View File
@@ -109,6 +109,7 @@ func (d *Docker) RemoveNetwork(ctx context.Context, id string) {
type Run struct {
Image string
Cmd []string
Env []string
Binds []string
Network string
DNS []string
@@ -138,7 +139,7 @@ func (d *Docker) Start(ctx context.Context, r Run) (string, string, error) {
path += "?name=" + url.QueryEscape(r.Name)
}
if err := d.call(ctx, http.MethodPost, path, map[string]any{"Image": r.Image, "Cmd": r.Cmd,
"Labels": labels, "HostConfig": host}, &made); err != nil {
"Env": r.Env, "Labels": labels, "HostConfig": host}, &made); err != nil {
return "", "", err
}
if err := d.call(ctx, http.MethodPost, "/containers/"+made.ID+"/start", nil, nil); err != nil {
-28
View File
@@ -1,28 +0,0 @@
#!/bin/sh
# mesh-check-toolchain: go
#
# The Go part of the lab's own check (novox/hq issue 302): the replays register and its prover, run by the
# build seat in the mesh's Go toolchain, its own container, because the lab's merge-check.sh declares it
# (`# mesh-check-also: go replays/merge-check.sh`) — and by hand, from anywhere:
# `sh replays/merge-check.sh`.
#
# 1. formatted and vetted — the register, the replays and the prover compile, tests included;
# 2. the tests that need nothing but Go and git: every replay registered whole (TestEveryReplayIsWhole),
# and the bed's reading of what a change touches.
#
# **Said, never passed silently**: the replays themselves raise containers and a bus of the release the mesh
# runs; this container holds no container runtime, so they are not run here. They run from the lab's main,
# reviewed, in the gate of every core change, and a change to one is proven by `go run ./cmd/prove` before
# it merges.
set -eu
cd "$(dirname "$0")"
unformatted=$(gofmt -l .)
if [ -n "$unformatted" ]; then
echo "not gofmt'd:"
echo "$unformatted"
exit 1
fi
go vet ./...
go test -count=1 -run '^(TestEveryReplayIsWhole|TestTheBedProvesWhatTheChangeTouches)$' .
echo "NOT RUN HERE: the replays themselves — they raise containers, and this container holds no container runtime"
+353
View File
@@ -32,6 +32,11 @@ const (
// judged by the node-engine at the commit, and the controller at the commit judging its gate from what
// the engine said (novox/hq ADR 0240). Two repositories move together: Repository and With.
Liveness Kind = "liveness"
// Bed is a bed of the lab, raised on its hypervisor: Test is its file under test/integration, run with
// the node-engine and its installer built from Repository at the commit, beside the With repositories at
// theirs (novox/hq issue 92). A bed that could not reach the step it replays skips, and the prover reads
// a skip as an error, never as the fault reproduced.
Bed Kind = "bed"
)
// Also is another repository a replay runs at its own commits beside Repository's: before its fix and on
@@ -275,6 +280,113 @@ var Register = []Replay{
"the operator naming what waits and the call that ends it, and it clears once the machine is sent the " +
"new bus",
Package: "./cmd/mesh-controller", Test: "TestReplay336", Files: []string{"cmd/mesh-controller/replay336_test.go"}},
// Two merges seconds apart left their delivery waiting for a person (2026-10-10): two catalogue merges 18 s
// apart were walked by one walk at the later commit, which superseded the first; mesh-delivery tied it to the
// delivery the supersession had ended, so nothing gave it its word for 58 minutes. A test in mesh-delivery,
// written with only what it had before the fix, as the controller of that night said its walks. The fix is
// still on its branch (mesh-catalog #171): proved against the main it branched from.
{ID: "R362", Issue: 362, Kind: InRepository, Repository: "mesh-catalog", Fix: "13ffae7",
Before: "52f34d3",
What: "claude-code and dunst merged 18 s apart; the one walk at the dunst commit was tied to the dunst " +
"delivery its supersession had ended, never let go, and claude-code was held for a person after ten minutes",
Asserts: "the walk at the later commit is let go once and delivers both merges without a person — dunst " +
"delivered, claude-code covered by it — whichever of the controller's two records arrives first",
Module: "modules/mesh-delivery", Package: "./cmd/mesh-delivery", Test: "TestReplay362",
Files: []string{"modules/mesh-delivery/cmd/mesh-delivery/replay362_test.go",
"modules/mesh-delivery/cmd/mesh-delivery/fakes_test.go"}},
// A path change read as data loss (2026-10-10): the agent's home on the control-node moved from the
// operator's own home to the agent account's (ADR 0266), and data-shrank was raised because the fresh
// directory was read against the old one's size. A test in the controller, written with only what it had
// before the fix, the incident replayed through the store into D13's findings. Fix is the merge of its
// pull request (mesh-controller #204), proved against its first parent.
{ID: "R368", Issue: 368, Kind: InRepository, Repository: "mesh-controller", Fix: "798738c",
What: "agent-home of claude-code on the control-node was raised as shrunk to 490 B from 94.7 MB when its " +
"path moved to the agent account's fresh home; nothing was lost",
Asserts: "an item whose path moved raises no data-shrank, and a genuine shrink at its new path is still " +
"raised, urgent for what is irreplaceable",
Package: "./cmd/mesh-controller", Test: "TestAMovedPathIsNoShrinkAndAShrinkThereStillIs",
Files: []string{"cmd/mesh-controller/data_test.go"}},
// A quote that could not be said (2026-10-07): a machine's role, a JSON value in single quotes with an
// apostrophe inside, was cut at the apostrophe by the controller's `command` verb, and the line was refused
// as a bare unclosed quote, naming neither where nor how a quote is written. A test in the controller,
// written with only what it had before the fix. Fix is the merge of its pull request (mesh-controller
// #205), proved against its first parent.
{ID: "R294", Issue: 294, Kind: InRepository, Repository: "mesh-controller", Fix: "690b75f",
What: "settings set claude-code '{\"role\":\"the operator's laptop\"}' through `command` was cut at the " +
"apostrophe and refused as an unclosed quote, with no position and no way to write a quote",
Asserts: "a quote in a value reads back through both shell forms, and a line left with an open quote is " +
"refused naming the character and how a quote is written, quoting none of the line",
Package: "./cmd/mesh-controller", Test: "TestAQuotedValueCanHoldAQuote",
Files: []string{"cmd/mesh-controller/seatverbs_test.go"}},
// A mesh directory outside every home was changed by path (2026-10-12): the directory applier set a
// directory's mode and owner by calling chmod and chown on its path, which follows a link anywhere in
// the path, so an account able to swap a parent for a link between two applies could direct root's
// chmod and chown at the link's target. Found by code reading, no known exploit. A test in the
// node-engine, written only with what applyDirectory had before its fix, laid over the commit before
// (mesh-host #87). Fix is the merge of its pull request.
{ID: "R358", Issue: 358, Kind: InRepository, Repository: "mesh-host", Fix: "36704b5471cb7b4e585847a08e4fd7fd64aea357",
What: "a mesh directory outside every home was given its mode and owner by path, so a parent swapped " +
"for a link between two applies carried root's chmod and chown to the link's target",
Asserts: "no apply changes the mode or owner of anything but the directory its declaration names, " +
"however a parent of its path is swapped for a link between applies, and the refusal names the link " +
"and the account that could have made it",
Package: "./internal/apply", Test: "TestAParentSwappedForALinkLeavesItsTargetUntouched",
Files: []string{"internal/apply/replay358_test.go"}},
// assign said two things that could not both be true (2026-10-04): a workstation already ran a module
// contributing node-hotkeys, and the module holding that seat was assigned to it; the answer said in
// order that the seat was now held, then that the node was left out of the rest of the mesh for lack
// of it, naming the assignment just made as the remedy. `plan` for the node right after resolved. A
// test in the controller, written with only what it had before its fix — register, assign's command,
// planFor — laid over the commit before. Needs a store: raised by inventory.ForTest against
// MESH_TEST_POSTGRES, same as the controller's other store-backed replays. Fix is the merge of its
// pull request (mesh-controller #253).
{ID: "R237", Issue: 237, Kind: InRepository, Repository: "mesh-controller", Fix: "5facd8b08e245545fb7656dd4af11280d9306a08",
What: "assign on a workstation already running a module that contributed node-hotkeys, after " +
"assigning the module holding that seat, said the seat was now held and then that the node was " +
"left out of the mesh for lack of it, naming the assignment just made as the remedy",
Asserts: "every line of assign's answer is derived from one resolution of the node taken after the " +
"write committed, and agrees with plan for the node run right after the act",
Package: "./cmd/mesh-controller", Test: "TestReplay237", Files: []string{"cmd/mesh-controller/replay237_test.go"}},
// A host without its package manager's configuration refused the declaration that would restore it
// (2026-10-04): a declaration that left the package manager's module out removed pacman's configuration
// file; from then on pacman did not answer, and the node-engine refused every declaration whole, the
// one that named the module again and would have written the file back included. A person restored the
// file by hand an hour and a half later. A test in the node-engine, written only with what it had before
// its fix, laid over the commit before (mesh-host #90). Fix is the merge of its pull request. The
// controller's half of the fix (mesh-controller #263, db7e4869: a machine refused or applied only partly
// past one apply interval raises a condition) is held by its own tests there, not by this replay.
{ID: "R233", Issue: 233, Kind: InRepository, Repository: "mesh-host", Fix: "a1f9856b3957fa0b11c04294f6367bc6febf5eb0",
What: "the package manager's configuration file was removed, pacman stopped answering, and the node-engine " +
"refused every declaration whole, the one that would have written the file back included",
Asserts: "a declaration that restores the package manager's configuration is applied while pacman does not " +
"answer: only what needs pacman fails, with the probe's words, and the file is written and recorded",
Package: "./cmd/mesh-host", Test: "TestReplay233",
Files: []string{"cmd/mesh-host/replay233_test.go"}},
// Genesis pushed to a registry its runtime did not yet trust (2026-09-22): the first real genesis raised the
// mesh's registry and pushed the control plane's image to it by the address it was given, and the runtime
// refused plain HTTP, because the trust was written by the docker module six steps after the push. The
// lab never saw it: its base image trusts the documentation ranges before any bed starts. A bed here,
// whose anchor has that trust taken off before the installer runs, raised with the node-engine and its
// installer built at the commit; the control plane genesis builds and the catalogue it reads are pinned,
// the same on both sides.
//
// **Pinned to today's genesis, not to the fix's merge** (hq issues 146, 590, 596, 597): at the merge of
// the fix (mesh-host #91, dd9ab792) no genesis got past step 6, so neither side reached the push. Fix is
// mesh-host main with every genesis fix since; Before is the same main with #91 reverted, kept on the
// branch replay/r92-before and never merged.
{ID: "R92", Issue: 92, Kind: Bed, Repository: "mesh-host", Fix: "0d11b4fc930d6723f3dc54dc173175f66fb6b3b3",
Before: "9e233e1d87569273b4ed353c450b0bd27d21039b",
What: "the first real genesis pushed the control plane's image to the registry it had just raised and was " +
"refused, server gave HTTP response to HTTPS client: the runtime's trust of the registry was written " +
"by the docker module, six steps after the push",
Asserts: "on a runtime that trusts no registry but loopback, genesis given --registry <its own address>:5000 " +
"pushes the control plane to the registry it raised, and the runtime trusts that registry by the " +
"mesh's name too",
Test: "test/integration/replay92.test.ts",
With: []Also{
{Repository: "mesh-controller", Fix: "49102e1645c920e42e453caf0aa0746d2b93ce90", Before: "49102e1645c920e42e453caf0aa0746d2b93ce90"},
{Repository: "mesh-catalog", Fix: "f88e3452f8b8b285ed86e1fe93e1a153f71faada", Before: "f88e3452f8b8b285ed86e1fe93e1a153f71faada"},
}},
// Not a core incident, and the first of its kind: a module's. The crash loop was found by a person
// reading the agent server's log for another reason (issue 268); research 032 measured it, and ADR
// 0240 makes the node-engine judge it and the gate fail it.
@@ -296,6 +408,247 @@ var Register = []Replay{
Asserts: "a web server whose application never answers is said unhealthy by its declared HTTP check within two " +
"looks, and the controller raises the module's condition on the second",
Package: ".", Test: "TestReplaySilentWebApp", Files: []string{"cmd/mesh-controller/replays_test.go"}},
// A merge check judged the head alone (2026-10-09): node-engine #66 was checked green at its head after #64/#65
// had moved the trunk, both adding a field under the JSON key `declared`; git merged the two edits cleanly and the
// trunk no longer compiled. A test in the controller's builder, written with only what it had before the fix, run
// with a container runtime that runs each container's command on the machine. The fix is still on its branch:
// Fix is its pull request's head, proved against the main it branched from (novox/hq ADR 0323).
{ID: "R341", Issue: 341, Kind: InRepository, Repository: "mesh-controller", Fix: "83a9cf6", Before: "3cfc121",
What: "node-engine #66 passed both statuses at its head and merged; merged into the trunk #65 had moved, it did " +
"not compile: Declared redeclared",
Asserts: "a check judges the head merged into the trunk's head, and fails a head whose merge does not compile, " +
"saying so",
Package: "./internal/builder", Test: "TestReplay341", Files: []string{"internal/builder/replay341_test.go"}},
// The mesh made a value for a secret only an outside party issues (2026-10-10): nobody had given the mounts
// module's smb-credentials, the send made a random value for it and placed it, and the module read it as a
// credential given and wrong (ADR 0228: such a secret is never made). The fix's own test reads what the fix added,
// so the replay is a test of its own in the controller, written with only what it had before the fix — register,
// assign, planFor, declarationFor, the inventory's secret reads — laid over the commit before. Needs a store:
// MESH_TEST_POSTGRES, as R237. Fix is the merge of its pull request (mesh-controller #262); Home is the commit that
// added the replay (mesh-controller #274), reachable once that pull request merges.
{ID: "R401", Issue: 401, Kind: InRepository, Repository: "mesh-controller", Fix: "1e28342edec170d59570e8127ddead7b19a70c20",
What: "the send made a random value for an own secret issued outside the mesh that nobody gave, and placed it: " +
"the module read it as a credential given and wrong",
Asserts: "the send makes no value for an own secret issued outside the mesh that nobody gave and places none, " +
"places no value the mesh made for it before, and places a value once a person gives one",
Package: "./cmd/mesh-controller", Test: "TestReplay401", Files: []string{"cmd/mesh-controller/replay401_test.go"},
Home: "mesh-controller", HomeRef: "f3c15e00af5004d053bc424fe01c7f9a24656d59"},
// A plan's id was pasted into its query (2026-10-12): an id given to the plans verb holding a quote answered
// another plan, or failed as a syntax error. The fix's own test is a file of its own using only what the
// inventory had before (ForTest, SavePlan, PlanByID, RecentPlans), so it is laid over the commit before. Needs a
// store: MESH_TEST_POSTGRES. Fix is the merge of its pull request (mesh-controller #268).
{ID: "R561", Issue: 561, Kind: InRepository, Repository: "mesh-controller", Fix: "f8daec01d3f085ad30aa7fb584055d8d451c79dc",
What: "a plan's id was pasted into the SQL that reads it: an id holding a quote answered another plan, or " +
"failed as a syntax error",
Asserts: "an id holding a quote is read as a plain id: no plan answers it but one of that very id, and " +
"RecentPlans takes its limit as a parameter",
Package: "./internal/inventory", Test: "TestAPlanIdHoldingAQuoteIsReadAsAPlainId",
Files: []string{"internal/inventory/plan_id_test.go"}},
// A plan sent the control node a controller older than the one running there (2026-10-02): two merges a minute
// apart made two plans; the controller's plan rolled the newer controller, which widened its seat's row, and the
// node-engine's plan then sent the build before it, whose binary refused the row. The fix's own TestReplay201
// reads what the fix added, so the replay is a test of its own in the controller, written with only what it had
// before — the inventory's build, gate, assignment and send records, keepingRecorded, planFor, declarationFor —
// laid over the commit before. Needs a store: MESH_TEST_POSTGRES. Fix is the merge of its pull request
// (mesh-controller #273); Home is the commit that added the replay (mesh-controller #275), reachable once that
// pull request merges.
{ID: "R201", Issue: 201, Kind: InRepository, Repository: "mesh-controller", Fix: "84deb530ee5afe2ffdf4d2cc65cdc355b12f8ccb",
What: "a plan sent the control node a controller older than the one running there, whose seat row the newer " +
"one had widened: the older binary refused the row and the mesh went quiet",
Asserts: "a send that is no person's push carries the controller at the build its machine was last sent, not " +
"an older record; a person's push carries the record",
Package: "./cmd/mesh-controller", Test: "TestReplay201ThePlansSend",
Files: []string{"cmd/mesh-controller/replay201_test.go"},
Home: "mesh-controller", HomeRef: "45a1783ba9a43fd466a518ee85673c701223b517"},
// A requirement could not be optional (issue 166): every soft dependency had become a hard requirement, so a
// manifest could not say grafana runs without a data source — a mesh without influxdb could not register it,
// and with influxdb only in the catalogue, assigning grafana installed influxdb for it. A test of its own in
// the controller, written with only what it had before — ParseManifest, Resolve, the resolve tests' helpers —
// laid over the commit before. Fix is the merge of its pull request (mesh-controller #281); Home is the commit
// that added the replay (mesh-controller #287), reachable once that pull request merges.
{ID: "R166", Issue: 166, Kind: InRepository, Repository: "mesh-controller", Fix: "4473bc7d7315bb462dcce7d6a8927bff37ecdc59",
What: "a manifest could not say a requirement is optional: grafana, which runs without a data source, could " +
"not be assigned on a mesh without influxdb, and influxdb only in the catalogue was installed for it",
Asserts: "a requirement marked optional parses; with no provider running, assigning its consumer is not refused " +
"and installs nothing for it; with a provider running elsewhere it binds",
Package: "./internal/catalogue", Test: "TestReplay166", Files: []string{"internal/catalogue/replay166_test.go"},
Home: "mesh-controller", HomeRef: "b1baa8d8baf0289b76e8082b3b95d146af6a973e"},
// A folder could not be owned by the account its module runs as (2026-10-06): the media managers' recycle-bin
// folders were declared owned by a literal user and group, and on a machine whose puid and pgid named another
// account the apps refused them as not writable. The fix's own tests read what the fix added, so the replay is
// a test of its own in the controller, written with only what it had before — Compose, a setting layer, the
// adoption tests' anchor — laid over the commit before. Fix is the merge of its pull request (mesh-controller
// #260); Home is the commit that added the replay, reachable once its pull request merges.
{ID: "R279", Issue: 279, Kind: InRepository, Repository: "mesh-controller", Fix: "9eddf36503f2a9af7ac22f6dffd4ab4462592b27",
What: "the TV and film managers refused their recycle-bin folders as not writable: a declared directory's owner " +
"was literal, and could not follow the puid and pgid the module runs as on that machine",
Asserts: "a directory owned by ${setting:puid}:${setting:pgid} reaches the node-engine owned by the numbers the " +
"machine's settings give",
Package: "./internal/catalogue", Test: "TestReplay279", Files: []string{"internal/catalogue/replay279_test.go"},
Home: "mesh-controller", HomeRef: "3bdab29a5a0ed0f15cbfdb4738a321f968ac9467"},
// A seat did not name the paths its holder owns (2026-10-10): who owns /etc/hosts was enforced only when two
// modules on one machine declared it, so a module not claiming node-hostname could declare it, and a holder of
// node-hostname could declare none of its paths. A test of its own in the controller, written with only what it
// had before — CatalogueProblems, SeatNamed, CanHold — laid over the commit before. Fix is the merge of its pull
// request (mesh-controller #267); Home as R279.
{ID: "R412", Issue: 412, Kind: InRepository, Repository: "mesh-controller", Fix: "4dbd4066ca54681bd92169502379ec42ff7599b6",
What: "a module not claiming node-hostname could declare /etc/hosts, and a module could claim node-hostname " +
"declaring neither /etc/hosts nor /etc/hostname: nothing refused either",
Asserts: "a module declaring a seat's path without claiming the seat is refused naming the seat, and a claim " +
"whose module does not declare the seat's paths is refused naming each",
Package: "./internal/catalogue", Test: "TestReplay412", Files: []string{"internal/catalogue/replay412_test.go"},
Home: "mesh-controller", HomeRef: "3bdab29a5a0ed0f15cbfdb4738a321f968ac9467"},
// A verb set what every agent session obeys (2026-10-09): the agent's module keeps the managed settings and tool
// servers every Claude Code session on a node obeys in a mergeable file, and the settings verb took any key for it
// from any caller, so an agent could plant a SessionStart hook in the operator's sessions on every node; an
// identity provider's empty runtime configuration took a url of the caller's the same way. The fix's own tests
// read what the fix added, so the replay is a test of its own in the controller, written with only what it had
// before — a mesh, a registered manifest, an assignment, the settings verb run as a verb and as the terminal runs
// it, the inventory's layer — laid over the commit before. Needs a store: MESH_TEST_POSTGRES. Fix is the merge of
// its pull request (mesh-controller #172); Home is the commit that added the replay, reachable once its pull
// request merges.
{ID: "R340", Issue: 340, Kind: InRepository, Repository: "mesh-controller", Fix: "4d60628f37530c23c072011a697e6014de14b3f6",
What: "the settings verb took a SessionStart hook in the agent module's managed settings, a tool server of the " +
"caller's, and a url for an identity provider's empty runtime configuration, from any caller of the mesh's verbs",
Asserts: "through a verb, any change to the layer of a module with a mergeable file not marked trusted: false is " +
"refused as the controller's terminal's and leaves the layer as it was; at the terminal it is taken",
Package: "./cmd/mesh-controller", Test: "TestReplay340", Files: []string{"cmd/mesh-controller/replay340_test.go"},
Home: "mesh-controller", HomeRef: "e44c726a38125ade30a6d70cee9873a4e2ea6ac2"},
// A merge acted on late superseded the merge after it (2026-10-09): two catalogue merges were acted on in reverse
// order. The later one, a security fix to the forge's module, was planned first; the earlier one, handed over late,
// was planned after it, superseded its plan at tier 0 and planned the fixed module again at the commit without the
// fix, because a plan's age was read from when it was made, never from the branch's order. The fix (mesh-controller
// #179) ordered a branch's plans by their merge times and planned a late merge at the newer commit; ADR 0276
// (mesh-controller #197) later replaced that with batches, and the replay holds the outcome under both. A test of its
// own in the controller, written with only what it had before — a mesh, registered modules, following.SourceMoved,
// the plans the inventory keeps — laid over the commit before. Needs a store: MESH_TEST_POSTGRES. Fix is the merge of
// its pull request; Home is the commit that added the replay, reachable once its pull request merges.
{ID: "R349", Issue: 349, Kind: InRepository, Repository: "mesh-controller", Fix: "9ca7952d5e0a5ed1e23bdcf5aea36fe967e62d14",
What: "a merge handed over late superseded the plan of the merge made after it, and planned the module that " +
"merge fixed again at the earlier commit, without the fix",
Asserts: "with the later merge acted on first and the earlier one after it, every plan left standing that plans " +
"the fixed module plans it at the later merge's commit, and one does",
Package: "./cmd/mesh-controller", Test: "TestReplay349", Files: []string{"cmd/mesh-controller/replay349_test.go"},
Home: "mesh-controller", HomeRef: "5740874ce3b99b47c27403f33e6c6d2a334ad3d7"},
// A resolver that stopped answering ran on unnoticed, and builds sent after it were blamed for it (2026-10-09):
// the control node's resolver refused every question on its private address from 10:57:57 UTC and the machine's
// network condition was raised at 10:58:45. The controller and the node-engine sent at 10:59:34 failed their gate
// with "raised since it was sent", because the restarted engine's first, undecided statement cleared the condition
// and its next look raised it anew after the send. A test of its own in the controller, written with only what it
// had before — a mesh, stateHealth, the keeper's open conditions, aboutTheMachine — laid over the commit before.
// Needs a store: MESH_TEST_POSTGRES. Fix is the merge of its pull request (mesh-controller #179); Home is the
// commit that added the replay, reachable once its pull request merges. The resolver's own half — it binds each
// address once and its watch asks it a real question and restarts it (mesh-catalog #161, 105cc7c0) — has no
// replay here: the module had no Go before that fix, and the binding's loss was never reproduced past the burst
// that caused it (hq issue 348's diagnosis); dnsmasq's own tests and the controller's manifest test hold it.
{ID: "R348", Issue: 348, Kind: InRepository, Repository: "mesh-controller", Fix: "9ca7952d5e0a5ed1e23bdcf5aea36fe967e62d14",
What: "a network fault raised before a send was cleared by a restarted node-engine's undecided statement and " +
"raised again after it, and the gate put back the builds sent after the fault began",
Asserts: "a build sent while the machine's network condition is open, which flaps after the send, is not held " +
"for it, and an undecided statement does not clear the condition",
Package: "./cmd/mesh-controller", Test: "TestReplay348AFaultFromBeforeTheSendFailsNoGate",
Files: []string{"cmd/mesh-controller/replay348_test.go"},
Home: "mesh-controller", HomeRef: "6eea92c323f4862d4162f9d977c8783d8a62efc3"},
// A credential only a person can give was made anyway (found provisioning the media machine, re-checked
// 2026-10-08): a Servarr app's API key, an indexer's key and a downloader's password are held by the provider's
// software, and for each consumer of them the send made a random pair credential, sealed it to both ends and
// placed it; the provider refused it, and a consumer retrying it could be banned (ADR 0337: such a credential is
// never made). A test of its own in the controller, written with only what it had before the fix — register,
// assign, planFor, declarationFor, AcceptSecretForPair, SecretsOf — laid over the commit before. Needs a store:
// MESH_TEST_POSTGRES. Fix is the merge of its pull request (mesh-controller #298), which also added the replay.
{ID: "R164", Issue: 164, Kind: InRepository, Repository: "mesh-controller", Fix: "584aa2d7d7a5a68b5087970feb93d0b1f1d4f503",
What: "the send made a random pair credential for a provision whose credential only a person can give, and " +
"placed it: the provider refused it",
Asserts: "for a provision whose offer says its credential is issued outside the mesh, the send makes no value " +
"and places none, the consumer's machine still composes, and a value a person gave for the pair is placed",
Package: "./cmd/mesh-controller", Test: "TestReplay164", Files: []string{"cmd/mesh-controller/replay164_test.go"}},
// A README-only merge made a new digest of an unchanged Go program (2026-10-13): a module built from its
// repository's root is compiled with the clone's .git in view, and Go stamped the commit into the binary — the
// node-engine on the laptop carried vcs.revision=7a41e120… — so a rebuild of an unchanged build source was a new
// program (novox/hq ADR 0267 rule 6). The builder run whole, Build to the packed bundle, at two commits differing
// only in README.md, with the toolchain's command run on this machine; written with only what the builder had
// before. Fix is the merge of its pull request (mesh-controller #292); Home is the commit that added the replay,
// reachable once its pull request merges.
{ID: "R363", Issue: 363, Kind: InRepository, Repository: "mesh-controller", Fix: "cd6757f1e3c818bc6043178bfb4edb42b8f3c63d",
What: "two builds of a Go module built from its repository's root, at commits differing only in its README, " +
"made two digests: the commit was stamped into the binary",
Asserts: "two builds of one Go module whose sources differ only outside its build source give one digest",
Package: "./internal/builder", Test: "TestReplay363", Files: []string{"internal/builder/replay363_test.go"},
Home: "mesh-controller", HomeRef: "94c928b1f5af099c328eb6950550b6380df24b31"},
// A module's name was taken over by another repository (2026-10-04): the photo app's module was registered from
// the app's own repository, and rebuilding it "to main" built the catalogue's stub of the same name instead. The
// controller registered the outcome, recorded the catalogue as the module's source, and the next push replaced the
// app with the stub: two sites' containers and five routes went. The fix (mesh-controller #174, hq ADR 0266 §7)
// registers an outcome only from the repository the module is registered from, unless asked at the controller's
// terminal. A test of its own in the controller, written with only what it had before — a mesh, RegisterModule,
// SourceOf, a kept build request, takeIn — laid over the commit before. Needs a store: MESH_TEST_POSTGRES. Fix is
// the merge of its pull request; Home is the commit that added the replay, reachable once its pull request merges.
{ID: "R239", Issue: 239, Kind: InRepository, Repository: "mesh-controller", Fix: "eb7a15808641f2075d9fd1660fa0e949b9cd7230",
What: "a build of a module's name from another repository was registered over the module's own, and the next " +
"push replaced the photo app with the catalogue's stub",
Asserts: "a build asked through a verb, of a module's name from a repository other than the one it is registered " +
"from, is refused naming both sources, and the module keeps its source and version",
Package: "./cmd/mesh-controller", Test: "TestReplay239AModuleNameIsNotTakenOverByAnotherRepository",
Files: []string{"cmd/mesh-controller/replay239_test.go"},
Home: "mesh-controller", HomeRef: "95556b8def3f3f21e3d037617d73f97606e6ea1f"},
// A new SDK rebuilt neither its toolchain nor the bundles compiled in it (2026-10-13): mesh-sdk 0.1.14 planned
// "builds mesh-sdk, sent nowhere". A built manifest is registered without its `build`, so mesh-tools' `build.on`
// naming the SDK's package and every bundle's toolchain were edges only before a module's first build — the
// fixes of issues 211 and 212 lost once modules were built. The fix (mesh-controller #296) reads them from what is
// kept: the package's name@version a build stood on, and the bundles' languages. A test of its own in the
// controller, written with only what it had before — RegisterModule, RecordBuild, theGraph, pullScope, reasoned
// as `delivery-plan` reasons — laid over main, where it plans [[mesh-sdk]]. Needs a store: MESH_TEST_POSTGRES.
// Fix is the merge of its pull request.
{ID: "R614", Issue: 614, Kind: InRepository, Repository: "mesh-controller",
Fix: "404b4409e70ed7365bb8a53862e4b197deb0bdc6",
What: "a new SDK was planned alone: neither the toolchain built on its package nor the TypeScript bundles " +
"compiled in that toolchain were rebuilt, so the SDK's fix reached no machine",
Asserts: "a change to the SDK plans the SDK, then the toolchain, then every bundle compiled in it, from modules " +
"registered as built and builds recorded with what they stood on",
Package: "./cmd/mesh-controller", Test: "TestReplay614ANewSDKPlansTheToolchainAndThenItsBundles",
Files: []string{"cmd/mesh-controller/replay614_test.go"}},
// A mesh-host change and a mesh-controller change of one delivery group that need each other could not both pass
// their own merge checks (2026-10-16, issue 867): each was read beside the other's repository at the commit the mesh
// runs, which holds neither change. The fix (mesh-controller #348, hq ADR 0382) asks mesh-delivery's groups once per
// check and reads each sibling's repository at its merge. TestReplay867 runs the whole path — the forge's
// announcement, the controller's ask, the build seat's check with a container runtime that runs on this machine —
// for the pair, written with only what the controller and the build seat had before. Needs a store:
// MESH_TEST_POSTGRES. R-issue867-recheck, R-issue867-recheck-verb and R-issue867-release-group are the three defects found live in the same fix, each with its own merge.
{ID: "R867", Issue: 867, Kind: InRepository, Repository: "mesh-controller",
Fix: "a47976be4e699c35681be87c48ecfedd793374cd",
What: "a mesh-host change and a mesh-controller change of one delivery group, each needing the other, could not " +
"both pass their own merge check: each was read beside the other at the commit the mesh runs",
Asserts: "each member of a delivery group passes its own merge check, read beside the other member's head " +
"merged onto its base",
Package: "./cmd/mesh-controller", Test: "TestReplay867", Files: []string{"cmd/mesh-controller/replay867_test.go"}},
// A recheck announces the pull request without its head branch, so its group was not found and the verdict said
// nothing of its siblings (mesh-controller #350): the group is found by repository and number.
{ID: "R-issue867-recheck", Issue: 867, Kind: InRepository, Repository: "mesh-controller",
Fix: "af60d2719c0c6c2f1ff44347037280d6ba5e72b4",
What: "a recheck of a group member read no siblings: it carries no head branch name, and the group was found by it",
Asserts: "a pull request announced as a recheck is, without its head branch, found in its group by repository " +
"and number, and passes beside its sibling",
Package: "./cmd/mesh-controller", Test: "TestReplay867Recheck", Files: []string{"cmd/mesh-controller/replay867_test.go"}},
// `delivery check` runs as a command of its own, where the controller is not the serving one: it asked mesh-delivery
// on no bus ("this controller is not on the bus") and read no siblings (mesh-controller #351).
{ID: "R-issue867-recheck-verb", Issue: 867, Kind: InRepository, Repository: "mesh-controller",
Fix: "a07dd78c3e90335bbe06970799e9b36a142f68c9",
What: "the recheck verb asked mesh-delivery for the group on no bus connection and read no siblings",
Asserts: "`delivery check` with one member, on a test bus with a mesh-delivery answering groups, learns the " +
"sibling, asking once",
Package: "./cmd/mesh-controller", Test: "TestTheRecheckVerbAsksMeshDeliveryOnABusOfItsOwn",
Files: []string{"cmd/mesh-controller/replay867_test.go"}},
// A release-branch group's composed check took the release judge, which cannot build a pair consistent only
// together (mesh-controller #352, hq ADR 0239 §14): the group's own controller judges, with the host member's
// validator. Both tests: the controller member primary, and the node-engine member primary with Judge validator.
{ID: "R-issue867-release-group", Issue: 867, Kind: InRepository, Repository: "mesh-controller",
Fix: "d70bb1f38867fad3595c96c7f7f2deb0b3ff501a",
What: "the composed check of a delivery group into a release branch was judged by the release branch's " +
"controller, which does not build with the pair's validator",
Asserts: "a group's composed check into a release branch is judged by the group's own controller with the " +
"mesh-host member's validator, and passes",
Package: "./cmd/mesh-controller", Test: "TestAGroupsComposition",
Files: []string{"cmd/mesh-controller/replay867_test.go"}},
}
// Find is the replay of that id, or of that issue.
+9
View File
@@ -2,6 +2,7 @@ package replays
import (
"regexp"
"slices"
"testing"
)
@@ -29,6 +30,14 @@ func TestEveryReplayIsWhole(t *testing.T) {
if len(r.Files) == 0 || len(r.With) == 0 {
t.Errorf("%s runs two repositories and lays the controller's half over its commit: %+v", r.ID, r)
}
case Bed:
if !regexp.MustCompile(`^test/integration/[a-z0-9-]+\.test\.ts$`).MatchString(r.Test) {
t.Errorf("%s is a bed and names no bed of the lab to raise: %q", r.ID, r.Test)
}
if !slices.ContainsFunc(r.With, func(a Also) bool { return a.Repository == "mesh-controller" }) ||
!slices.ContainsFunc(r.With, func(a Also) bool { return a.Repository == "mesh-catalog" }) {
t.Errorf("%s is a bed and does not pin the controller and the catalogue it raises: %+v", r.ID, r)
}
case Bus, Resolver:
default:
t.Errorf("%s is of no kind the prover runs: %q", r.ID, r.Kind)
+2
View File
@@ -1,3 +1,5 @@
//go:build integration
package replays
import (
+2
View File
@@ -1,3 +1,5 @@
//go:build integration
package replays
import (
+135
View File
@@ -0,0 +1,135 @@
//go:build integration
package replays
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// THE STORE GENESIS RAISES INITIALISES WHERE THE POSTGRES MODULE KEEPS IT (novox/hq ADR 0338 §5, issue 609).
//
// Genesis raises the store as the catalogue's postgres module declares its server: the module's image, its
// data in a directory made before the store starts and owned 999:70 as the module declares it, and its
// superuser password read from a file. The measurement the decision names: the module's image, started on
// such a directory, empty, initialises a store that answers with that password. The image is read from the
// catalogue's manifest, so the measurement moves with the module's pin.
//
// Needs a container runtime (its socket) and the catalogue (MESH_REPLAY_CATALOGUE, or the checkout beside
// the lab); skipped aloud without either.
func TestTheStoreGenesisRaisesInitialisesOnTheDirectoryGenesisMakes(t *testing.T) {
catalogue := os.Getenv("MESH_REPLAY_CATALOGUE")
if catalogue == "" {
catalogue = filepath.Join("..", "..", "mesh-catalog")
}
raw, err := os.ReadFile(filepath.Join(catalogue, "modules", "postgres", "module.json"))
if err != nil {
t.Skipf("no catalogue to read the store's image from (%v); MESH_REPLAY_CATALOGUE names one", err)
}
docker, ok := DockerFromEnv()
if !ok {
t.Skip("no container runtime: the store runs in a container")
}
var m struct {
Resources []struct {
ID, Type, Path, Owner, Mode, Image string
Env map[string]string
} `json:"resources"`
}
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
var image, pgdata, owner, mode string
for _, r := range m.Resources {
switch {
case r.Type == "container" && r.ID == "server":
image, pgdata = r.Image, r.Env["PGDATA"]
case r.Type == "directory" && r.ID == "store-data":
owner, mode = r.Owner, r.Mode
}
}
if image == "" || pgdata == "" || owner == "" {
t.Fatalf("the postgres module names no server image (%q), PGDATA (%q) or data directory owner (%q)", image, pgdata, owner)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
// Under the working directory: a check's workspace is mounted into its containers at the same path,
// so the runtime binds what this test made.
here, err := os.Getwd()
if err != nil {
t.Fatal(err)
}
dir, err := os.MkdirTemp(here, "store-measure-")
if err != nil {
t.Fatal(err)
}
for _, img := range []string{"alpine:3.20", image} {
if err := docker.Pull(ctx, img); err != nil {
t.Fatal(err)
}
}
// The directory as genesis makes it, and the password file as genesis writes it: as root, a helper
// container doing what the node-engine does on the machine.
prep, _, err := docker.Start(ctx, Run{Image: "alpine:3.20", Binds: []string{dir + ":/w"}, Cmd: []string{"sh", "-c",
"mkdir -p /w/store /w/state && chown " + owner + " /w/store && chmod " + orMode(mode) + " /w/store && " +
"printf measured-pw > /w/state/superuser.secret && chmod 0600 /w/state/superuser.secret"}})
if err != nil {
t.Fatal(err)
}
if code, said, err := docker.Wait(ctx, prep); err != nil || code != 0 {
t.Fatalf("the directory could not be made as genesis makes it: %v %d %s", err, code, said)
}
docker.Remove(ctx, prep)
t.Cleanup(func() {
clean, _, err := docker.Start(context.Background(), Run{Image: "alpine:3.20", Binds: []string{dir + ":/w"},
Cmd: []string{"sh", "-c", "rm -rf /w/store /w/state"}})
if err == nil {
_, _, _ = docker.Wait(context.Background(), clean)
docker.Remove(context.Background(), clean)
}
_ = os.RemoveAll(dir)
})
store, _, err := docker.Start(ctx, Run{Image: image,
Env: []string{"POSTGRES_PASSWORD_FILE=/run/secrets/superuser", "PGDATA=" + pgdata},
Binds: []string{dir + "/store:/var/lib/postgresql/data", dir + "/state/superuser.secret:/run/secrets/superuser:ro"}})
if err != nil {
t.Fatal(err)
}
defer docker.Remove(context.Background(), store)
answered := false
for i := 0; i < 90 && !answered; i++ {
// Over TCP, and with the password: the temporary server initdb runs listens on the socket only.
code, err := docker.Exec(ctx, store, "sh", "-c",
"PGPASSWORD=measured-pw psql -h 127.0.0.1 -U postgres -tAc 'select 1' | grep -qx 1")
answered = err == nil && code == 0
if !answered {
time.Sleep(2 * time.Second)
}
}
if !answered {
t.Fatalf("%s did not initialise a store that answers on a directory made %s %s, as genesis makes it:\n%s",
image, owner, orMode(mode), lastLinesOf(docker.Logs(ctx, store), 15))
}
}
func orMode(m string) string {
if m == "" {
return "0700"
}
return m
}
func lastLinesOf(s string, n int) string {
lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
if len(lines) > n {
lines = lines[len(lines)-n:]
}
return strings.Join(lines, "\n")
}
+41
View File
@@ -0,0 +1,41 @@
# GENESIS PAST ITS BROKER STEP: the bed of novox/hq ADR 0333 §5 (issue 590).
#
# The installer raises the bus from its bundle, then adopts it as the `nats` module at its broker step (step
# 18 of 24). Before ADR 0333 the bundle raised the bus under another name and on other paths than the module
# keeps it, so adopting it would have raised a second bus beside the first, and genesis stopped there. The
# bed runs the installer from bare to that step, looks at the bus, runs it on past the step, and looks again.
#
# Two machines on one public segment, both with a way out — the installer pulls the store, the bus and the
# toolchains over it, as a bare machine does:
# - anchor: the machine genesis raises;
# - forge: serves the control plane's, the base's, the SDK's and the catalogue's repositories at the
# commits under test over git's own protocol, so the installer builds exactly those and nothing on the
# workstation listens for the lab. On the anchor itself it would be a listener, and genesis refuses a
# machine something already listens on.
scenario: genesis-keeps-its-bus
segments:
hosting:
kind: public
cidr: [192.0.2.0/24]
machines:
anchor:
at: { segment: hosting, address: [192.0.2.10] }
egress: true
inbound: allow
# The foundation, the registry, the forge genesis raises, the base and the modules it builds.
memory: 8GiB
cpus: 4
disk: 60GiB
forge:
at: { segment: hosting, address: [192.0.2.30] }
egress: true
inbound: allow
memory: 1GiB
cpus: 1
disk: 10GiB
place:
all: [host, runtime]
+42
View File
@@ -0,0 +1,42 @@
# GENESIS on a runtime nobody told to trust a registry: the bed of replay R92 (novox/hq issue 92).
#
# The first real genesis pushed the control plane's image to the registry it had just raised, and the
# container runtime refused: "server gave HTTP response to HTTPS client". The mesh's registry speaks plain
# HTTP on purpose, and the trust that lets a runtime speak to it was written six steps after the push. The
# lab never saw it, because its base image trusts the documentation ranges as plain-HTTP registries before
# any bed starts (src/lifecycle/base.ts). The bed takes that trust back off the anchor before genesis runs,
# so the anchor's runtime is a bare machine's: it trusts loopback and nothing else.
#
# Two machines on one public segment, both with a way out — the installer pulls the store, the bus and the
# registry over it, as a bare machine does:
# - anchor: the machine genesis raises, given its registry at its own address, 192.0.2.10:5000;
# - forge: serves the control plane's repository at the commit under test over git's own protocol, so the
# installer builds exactly that and nothing on the workstation listens for the lab. On the anchor
# itself it would be a listener, and genesis refuses a machine something already listens on.
scenario: genesis-untrusted
segments:
hosting:
kind: public
cidr: [192.0.2.0/24]
machines:
anchor:
at: { segment: hosting, address: [192.0.2.10] }
egress: true
inbound: allow
# Genesis-single's anchor: the foundation, the registry, and two control planes during the pivot.
memory: 8GiB
cpus: 4
disk: 40GiB
forge:
at: { segment: hosting, address: [192.0.2.30] }
egress: true
inbound: allow
memory: 2GiB
cpus: 2
disk: 15GiB
place:
all: [host, runtime]
+19 -3
View File
@@ -24,6 +24,7 @@ import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { homedir } from "node:os";
import { dirname, join } from "node:path";
import type { Outcome, Skip } from "./outcomes.ts";
import { repositories } from "./repos.ts";
/** What a run was taken against, per repository. */
@@ -38,6 +39,11 @@ export interface Receipt {
against: Against;
/** The test files this run was pointed at. See {@link endToEnd}. */
ran: string[];
/** How many tests skipped, and each with its reason (novox/hq issue 693). Absent on older receipts. */
skipped?: number;
skips?: Skip[];
/** Why the run did not pass although nothing failed — no test passed, or one was not reached. */
notPassed?: string;
}
/**
@@ -104,6 +110,7 @@ export function record(
ran: string[],
env = process.env,
builtAgainst?: Against,
outcome?: Outcome,
): Receipt {
const receipt: Receipt = {
at: new Date().toISOString(),
@@ -115,6 +122,8 @@ export function record(
// never in the binaries. A verdict is only worth something attributed to one exact state.
against: builtAgainst ?? whatWasTested(env),
ran,
...(outcome ? { skipped: outcome.skipped, skips: outcome.skips } : {}),
...(outcome?.notPassed ? { notPassed: outcome.notPassed } : {}),
};
const path = receiptPath();
mkdirSync(dirname(path), { recursive: true });
@@ -166,9 +175,14 @@ export function judge(
const days = (now.getTime() - Date.parse(receipt.at)) / 86_400_000;
const lines: string[] = [];
// A run that proved nothing did not pass, though nothing failed (novox/hq issue 693): no test passed,
// or one skipped as not reached. A receipt from before the field is judged by its count alone.
const notPassed = receipt.failed > 0 ? "" : receipt.notPassed ?? (receipt.passed === 0 ? "no test passed" : "");
const outcome = receipt.failed > 0
? `last ran ${ago(days)} and ${receipt.failed} test(s) failed`
: `last passed ${ago(days)}, ${receipt.passed} test(s)`;
: notPassed
? `last ran ${ago(days)} and did not pass: ${notPassed}`
: `last passed ${ago(days)}, ${receipt.passed} test(s)`;
lines.push(`the end-to-end suite ${outcome}`);
let moved = false;
@@ -192,7 +206,9 @@ export function judge(
lines.push(` That run did not include ${endToEnd}, so it raised no machines.`);
}
if (receipt.failed > 0) {
for (const s of receipt.skips ?? []) lines.push(` skipped: ${s.test} — ${s.reason || "no reason given"}`);
if (receipt.failed > 0 || notPassed) {
lines.push(" Nothing has been proven end to end since.");
} else if (moved) {
lines.push(" What it proved was proven about code that has since changed.");
@@ -201,7 +217,7 @@ export function judge(
}
return {
current: endToEndRan && receipt.failed === 0 && !moved && days <= staleAfterDays,
current: endToEndRan && receipt.failed === 0 && !notPassed && !moved && days <= staleAfterDays,
lines,
};
}
+102 -9
View File
@@ -23,6 +23,88 @@ export const UPSTREAM_IMAGE = "images:archlinux/current";
const BUILDER = "mesh-lab-base-builder";
/**
* **The mirrors the builder installs from, several and in order** (novox/hq issue 700).
*
* The stock image names one mirror, mirrors.kernel.org, and a base build failed twice on it in one morning:
* timeouts while installing the runtime, then a 500 on wireguard-tools. pacman tries the next server in the
* list for each file it could not fetch, so a list of several turns one mirror's bad minute into a slower
* download instead of a failed build.
*
* A fixed list rather than ranking at build time: ranking (rankmirrors, reflector) needs a package installed
* first, from the one mirror that is the problem, and gives a different list on every build. These are
* Arch's own geo-routed mirror first, then well-kept European mirrors near the lab's machine, then the
* stock image's own. Each answered for core.db on 2026-10-14. A mirror that goes away costs a fallback,
* not a build, and is replaced here.
*/
export const MIRRORS = [
"https://geo.mirror.pkgbuild.com/$repo/os/$arch",
"https://mirror.netcologne.de/archlinux/$repo/os/$arch",
"https://ftp.fau.de/archlinux/$repo/os/$arch",
"https://archlinux.mirror.liteserver.nl/$repo/os/$arch",
"https://mirror.leaseweb.net/archlinux/$repo/os/$arch",
"https://mirrors.kernel.org/archlinux/$repo/os/$arch",
];
/** The mirrorlist the builder is given: every mirror, in order. */
export function mirrorlist(): string {
return "# Written by the mesh-lab base build (novox/hq issue 700): several mirrors, tried in order.\n" +
MIRRORS.map((m) => `Server = ${m}`).join("\n") + "\n";
}
/** The mirrors a failed pacman step names: "from <host>" in what it said, each once. */
export function failedMirrors(said: string): string[] {
const hosts = [...said.matchAll(/\bfrom ([a-z0-9.-]+\.[a-z]{2,})\b/gi)].map((m) => m[1]!.toLowerCase());
return [...new Set(hosts)];
}
/** How long to wait before each retry of a failed pacman step: three retries, backing off. */
export const PACMAN_BACKOFF_MS = [10_000, 30_000, 90_000];
/** What a base build runs through, so a test can stand in for the hypervisor and the clock. */
export interface BaseDeps {
incus: (args: string[], timeoutMs?: number) => Promise<unknown>;
incusOk: (args: string[], timeoutMs?: number) => Promise<string | null>;
succeeds: (args: string[], timeoutMs?: number) => Promise<boolean>;
sleep: (ms: number) => Promise<void>;
}
const realDeps: BaseDeps = {
incus, incusOk, succeeds, sleep: (ms) => new Promise((r) => setTimeout(r, ms)),
};
/**
* pacmanIn runs one pacman step in the builder, and **runs it again after a pause when it fails**, up to
* three more times, saying each time which mirror failed and what it said. Installing is idempotent — what is
* installed stays installed — so a retry repeats only what did not happen. The last failure is thrown, naming
* every mirror that failed along the way.
*/
export async function pacmanIn(name: string, args: string[], log: (m: string) => void, deps: BaseDeps = realDeps):
Promise<void> {
const failed: string[] = [];
for (let attempt = 0; ; attempt++) {
try {
await deps.incus(["exec", name, "--", "pacman", ...args], 600_000);
return;
} catch (err) {
// What pacman said, where incus kept it apart; otherwise the whole message.
const said = (err as { stderr?: string }).stderr?.trim() || (err as Error).message;
const mirrors = failedMirrors(said);
for (const m of mirrors) if (!failed.includes(m)) failed.push(m);
const where = mirrors.length > 0 ? `from ${mirrors.join(", ")}` : "naming no mirror";
const first = said.split("\n").find((l) => /error/i.test(l))?.trim() ?? said.split("\n")[0]?.trim() ?? "";
if (attempt >= PACMAN_BACKOFF_MS.length) {
throw new BaseImageError(`pacman ${args.join(" ")} failed ${attempt + 1} times; the mirrors that failed: ` +
`${failed.join(", ") || "none named"}. Last: ${said}`);
}
const wait = PACMAN_BACKOFF_MS[attempt]!;
log(` pacman ${args.join(" ")} failed ${where} (${first}); trying again in ${wait / 1000}s ` +
`(attempt ${attempt + 2} of ${PACMAN_BACKOFF_MS.length + 1})`);
await deps.sleep(wait);
}
}
}
export class BaseImageError extends Error {
constructor(message: string) {
super(message);
@@ -38,7 +120,9 @@ export class BaseImageError extends Error {
*/
export async function buildBaseImage(
log: (message: string) => void = () => {},
deps: BaseDeps = realDeps,
): Promise<{ alias: string; runtime: string }> {
const { incus, incusOk, succeeds } = deps;
await succeeds(["delete", "-f", BUILDER], 120_000);
log(` launching ${BUILDER} from ${UPSTREAM_IMAGE}, with a network`);
@@ -50,10 +134,19 @@ export async function buildBaseImage(
], 300_000);
try {
await waitForAgent(BUILDER);
await waitForAgent(BUILDER, deps);
// Several mirrors before anything is fetched (MIRRORS), and read back: a list that was not written is
// the one mirror again, and the failure would look like the one this prevents.
log(` giving pacman ${MIRRORS.length} mirrors`);
await incus(["exec", BUILDER, "--", "sh", "-c", `printf '%s' '${mirrorlist()}' > /etc/pacman.d/mirrorlist`], 60_000);
const written = await incusOk(["exec", BUILDER, "--", "cat", "/etc/pacman.d/mirrorlist"], 60_000);
if (written?.trim() !== mirrorlist().trim()) {
throw new BaseImageError(`the mirrorlist written into ${BUILDER} reads back as ${JSON.stringify(written)}`);
}
log(" installing a container runtime");
await incus(["exec", BUILDER, "--", "pacman", "-Sy", "--noconfirm", "docker"], 600_000);
await pacmanIn(BUILDER, ["-Sy", "--noconfirm", "docker"], log, deps);
// And the tools for the private network, for the same reason as the runtime: a sealed
// scenario cannot install them, so a lab that omits them cannot test connectivity at all —
@@ -63,27 +156,27 @@ export async function buildBaseImage(
// declaration; a lab that brought the interface up itself would be testing its own setup
// rather than the mesh's.
log(" installing the tools for the private network");
await incus(["exec", BUILDER, "--", "pacman", "-S", "--noconfirm", "wireguard-tools"], 600_000);
await pacmanIn(BUILDER, ["-S", "--noconfirm", "wireguard-tools"], log, deps);
// And git, for the same reason again: a machine that builds modules clones them, and a sealed
// scenario cannot install it. On a real build machine the mesh installs it as a package like
// anything else — the lab is the special case, because its machines reach no mirror.
log(" installing git, so a machine can build modules");
await incus(["exec", BUILDER, "--", "pacman", "-S", "--noconfirm", "git"], 600_000);
await pacmanIn(BUILDER, ["-S", "--noconfirm", "git"], log, deps);
// And nftables, because the mesh computes a machine's filtering and delivers it as a file
// that a service reflects — and neither the file nor the service can install what loads it.
// Installed and NOT enabled: whether a machine filters is the mesh's decision, and a lab that
// turned it on itself would be testing its own setup.
log(" installing nftables, so a machine can enforce what the mesh computed");
await incus(["exec", BUILDER, "--", "pacman", "-S", "--noconfirm", "nftables"], 600_000);
await pacmanIn(BUILDER, ["-S", "--noconfirm", "nftables"], log, deps);
// And dnsmasq, because a service is named under the machine it runs on — postgres.novox.internal
// — and only a resolver can answer a name the mesh was never told about. Installed and NOT
// started: whether a machine resolves for the mesh is the mesh's decision, and a lab that
// turned it on itself would be testing its own setup.
log(" installing dnsmasq, so a machine can answer names under another machine");
await incus(["exec", BUILDER, "--", "pacman", "-S", "--noconfirm", "dnsmasq"], 600_000);
await pacmanIn(BUILDER, ["-S", "--noconfirm", "dnsmasq"], log, deps);
// Trust the documentation ranges as plain-HTTP registries.
//
@@ -216,10 +309,10 @@ export async function baseImageExists(): Promise<boolean> {
* Wait for the guest agent, because `launch` returning means the VM started, not that anything
* inside it will answer.
*/
async function waitForAgent(name: string): Promise<void> {
async function waitForAgent(name: string, deps: BaseDeps = realDeps): Promise<void> {
for (let i = 0; i < 90; i++) {
if (await succeeds(["exec", name, "--", "true"], 10_000)) return;
await new Promise((r) => setTimeout(r, 2_000));
if (await deps.succeeds(["exec", name, "--", "true"], 10_000)) return;
await deps.sleep(2_000);
}
throw new BaseImageError(`${name} started and its agent never answered.`);
}
+114
View File
@@ -0,0 +1,114 @@
/**
* What a suite's tests did, beyond the runner's totals: which were skipped and why, and whether the run
* proved anything (novox/hq issue 693).
*
* **A run in which no test passed has proven nothing, whatever its exit code.** The runner exits 0 when
* every test skipped, and the lab recorded such a run as passed: the genesis bed stopped before its
* subject, skipped its only test as "NOT REACHED", and the run read "0 passed, 0 failed", passed.
* Failure must be loud, and a green run that verified nothing is the quietest failure there is.
*
* The default export is a reporter for node's test runner (`--test-reporter`): it writes one JSON line per
* skipped test, with its file, its name and the reason it gave, word for word. The reasons come from the
* runner's events rather than from the spec reporter's text, because a reason runs over several lines and a
* later test's output lands between them.
*/
import { readFileSync } from "node:fs";
/** One skipped test and the reason it gave ("" when it gave none). */
export interface Skip {
file: string;
test: string;
reason: string;
}
/** What a receipt keeps beside the totals: the skips, and why the run did not pass when it did not. */
export interface Outcome {
skipped: number;
skips: Skip[];
notPassed?: string;
}
/**
* The words a bed's test skips with when the run stopped before its subject: the subject was not reached,
* so nothing about it was proven (the beds say "<what> NOT REACHED: <where it stopped>").
*/
export const NOT_REACHED = "NOT REACHED";
/**
* Whether a run passed, and if not, why — the design limit of issue 693, in one place:
*
* - a test failed: not passed (the runner already says so);
* - **no test passed**: not passed, whatever was skipped — a run that skipped everything, or ran
* nothing, verified nothing;
* - **a test skipped as NOT REACHED**: not passed, even beside tests that passed — its subject was not
* reached, which is the bed failing to get where it was sent, not a skip anyone declared expected;
* - otherwise passed, its other skips listed with their reasons: a test skipped for a reason its file
* declares (no hypervisor, an optional bed not asked for) beside tests that passed is an expected
* skip, and the run still proved what passed.
*
* Counts the runner did not report (null) decide nothing here: that is reportOn's "no receipt" case.
*/
export function judgeRun(
counts: { passed: number | null; failed: number | null },
skips: Skip[],
): { passed: boolean; why: string } {
if (counts.failed !== null && counts.failed > 0) return { passed: false, why: `${counts.failed} test(s) failed` };
// In any case: "not reached" says the same as "NOT REACHED".
const unreached = skips.filter((s) => s.reason.toUpperCase().includes(NOT_REACHED));
if (counts.passed === 0) {
return {
passed: false,
why: skips.length > 0
? `no test passed: ${skips.length} skipped${unreached.length > 0 ? `, ${unreached.length} of them as ${NOT_REACHED}` : ""}`
: "no test passed: none ran",
};
}
if (unreached.length > 0) {
return { passed: false, why: `${unreached.length} test(s) skipped as ${NOT_REACHED}: their subject was not reached` };
}
return { passed: true, why: "" };
}
/** A reason on one line, for a log line and a record: its whitespace folded, and at most 500 characters. */
export function oneLine(reason: string): string {
const flat = reason.replace(/\s+/g, " ").trim();
return flat.length > 500 ? `${flat.slice(0, 499)}…` : flat;
}
/** The skips the reporter wrote to a file; none when it wrote nothing or the file cannot be read. */
export function readSkips(path: string): Skip[] {
let text: string;
try {
text = readFileSync(path, "utf8");
} catch {
return [];
}
const skips: Skip[] = [];
for (const line of text.split("\n")) {
if (!line.trim()) continue;
try {
const s = JSON.parse(line) as Partial<Skip>;
skips.push({ file: String(s.file ?? ""), test: String(s.test ?? ""), reason: String(s.reason ?? "") });
} catch {
// A torn last line, when the runner was killed mid-write: the lines before it still stand.
}
}
return skips;
}
interface Event {
type: string;
data: { name?: string; file?: string; skip?: string | boolean; details?: { type?: string } };
}
/** The reporter: one JSON line per skipped test (not per skipped suite: its tests are reported each). */
export default async function* skipsReporter(source: AsyncIterable<Event>): AsyncGenerator<string> {
for await (const event of source) {
if (event.type !== "test:pass" && event.type !== "test:fail") continue;
const { skip, details } = event.data;
if (skip === undefined || skip === false || details?.type === "suite") continue;
const said: Skip = { file: event.data.file ?? "", test: event.data.name ?? "", reason: typeof skip === "string" ? skip : "" };
yield `${JSON.stringify(said)}\n`;
}
}
+70 -16
View File
@@ -10,11 +10,16 @@
*/
import { spawn } from "node:child_process";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { isAbsolute, join, relative } from "node:path";
import { fileURLToPath } from "node:url";
import { endToEnd, record, whatWasTested } from "./lastrun.ts";
import { judgeRun, oneLine, readSkips, type Outcome, type Skip } from "./outcomes.ts";
import { rebuild } from "./rebuild.ts";
/** counted is what the runner said, or nulls when it said nothing recognisable. */
export function counted(output: string): { passed: number | null; failed: number | null } {
export function counted(output: string): { passed: number | null; failed: number | null; skipped: number | null } {
// The runner's own summary lines, each on a line of its own. Anchored, so a test *named*
// "pass 3" cannot be mistaken for the total — which is not a hypothetical worry in a suite whose
// tests are named in sentences.
@@ -23,13 +28,18 @@ export function counted(output: string): { passed: number | null; failed: number
// running this against the real runner — the fixture it was first written against was output I
// had imagined, which is a test that agrees with the mistake it was written beside.
const plain = output.replace(/\u001b\[[0-9;]*m/g, "");
// **The last such line, never the first** (novox/hq issue 693): the runner prints its summary after
// everything a test printed, so a test that prints "ℹ pass 5" before it skips would otherwise be read
// as five passes, and a run that proved nothing as passed.
const total = (what: RegExp) => {
const found = plain.match(what);
return found ? Number(found[1]) : null;
const found = [...plain.matchAll(what)];
const last = found[found.length - 1];
return last ? Number(last[1]) : null;
};
return {
passed: total(/^\s*(?:\u2139|#)\s*pass\s+(\d+)\s*$/m),
failed: total(/^\s*(?:\u2139|#)\s*fail\s+(\d+)\s*$/m),
passed: total(/^\s*(?:\u2139|#)\s*pass\s+(\d+)\s*$/gm),
failed: total(/^\s*(?:\u2139|#)\s*fail\s+(\d+)\s*$/gm),
skipped: total(/^\s*(?:\u2139|#)\s*skipped\s+(\d+)\s*$/gm),
};
}
@@ -57,16 +67,38 @@ export async function runSuite(args: string[]): Promise<number> {
// `test/integration/canary.test.ts` is still there and still runs when it is named; it is no
// longer raised on the way to everything else.
const { code, seen } = await runFiles(files);
console.log("\n" + reportOn(counted(seen), (p, f) => record(p, f, files, process.env, against)));
return code;
const { code, seen, skips } = await runFiles(files);
const counts = counted(seen);
console.log("\n" + reportOn(counts, skips, (p, f, outcome) => record(p, f, files, process.env, against, outcome)));
return exitOf(code, counts, skips);
}
/** Run some test files, passing their output through as it arrives. */
async function runFiles(files: string[]): Promise<{ code: number; seen: string }> {
/**
* The suite's exit: the runner's, unless it says 0 for a run that did not pass (novox/hq issue 693). The
* runner exits 0 when every test skipped; the lab takes this exit as the run's verdict, so a run that
* proved nothing must not end in 0.
*/
export function exitOf(code: number, counts: { passed: number | null; failed: number | null }, skips: Skip[]): number {
if (code !== 0) return code;
if (counts.passed === null || counts.failed === null) return code;
return judgeRun(counts, skips).passed ? 0 : 1;
}
/** The reporter that writes each skipped test and its reason (outcomes.ts), by its path. */
const skipsReporter = fileURLToPath(new URL("./outcomes.ts", import.meta.url));
/**
* Run some test files, passing their output through as it arrives, and collect what was skipped and why:
* the spec reporter to stdout as before, and the skips reporter beside it into a file of the run's own.
*/
async function runFiles(files: string[]): Promise<{ code: number; seen: string; skips: Skip[] }> {
const scratch = mkdtempSync(join(tmpdir(), "mesh-lab-suite-"));
const skipsFile = join(scratch, "skips.jsonl");
const running = spawn(
process.execPath,
["--test", "--test-concurrency=1", "--experimental-strip-types",
"--test-reporter=spec", "--test-reporter-destination=stdout",
`--test-reporter=${skipsReporter}`, `--test-reporter-destination=${skipsFile}`,
...files],
{ stdio: ["inherit", "pipe", "inherit"] },
);
@@ -81,7 +113,14 @@ async function runFiles(files: string[]): Promise<{ code: number; seen: string }
const code: number = await new Promise((resolve) => {
running.on("close", (c) => resolve(c ?? 1));
});
return { code, seen };
// Named as the files were given, relative to where the suite runs, rather than as the runner's absolute paths.
const named = (file: string) => {
const rel = isAbsolute(file) ? relative(process.cwd(), file) : file;
return rel.startsWith("..") ? file : rel;
};
const skips = readSkips(skipsFile).map((s) => ({ ...s, file: named(s.file) }));
rmSync(scratch, { recursive: true, force: true });
return { code, seen, skips };
}
/**
@@ -92,16 +131,31 @@ async function runFiles(files: string[]): Promise<{ code: number; seen: string }
* could check it — which is 04-ISSUES/005 in miniature, inside the fix for it.
*/
export function reportOn(
counts: { passed: number | null; failed: number | null },
write: (passed: number, failed: number) => { against: Record<string, string>; ran: string[] },
counts: { passed: number | null; failed: number | null; skipped?: number | null },
skips: Skip[],
write: (passed: number, failed: number, outcome: Outcome) => { against: Record<string, string>; ran: string[] },
): string {
if (counts.passed === null || counts.failed === null) {
// A run whose result could not be read is a run nobody can say anything about. Writing
// "0 failed" because nothing said otherwise is how a green record comes to mean nothing.
return "could not read what the runner reported; no receipt written";
}
const receipt = write(counts.passed, counts.failed);
// **A skip is said, with its reason, and a run that proved nothing says NOT PASSED** (novox/hq issue
// 693): "0 passed, 0 failed" read as a clean run. The lab module reads these lines into its record.
const skipped = counts.skipped ?? skips.length;
const judged = judgeRun({ passed: counts.passed, failed: counts.failed }, skips);
const outcome: Outcome = {
skipped,
skips: skips.map((s) => ({ ...s, reason: oneLine(s.reason) })),
...(judged.passed ? {} : { notPassed: judged.why }),
};
const receipt = write(counts.passed, counts.failed, outcome);
const against = Object.entries(receipt.against).map(([n, c]) => `${n} ${c}`).join(", ");
return `recorded: ${receipt.ran.join(", ")} — ${counts.passed} passed, ` +
`${counts.failed} failed, against ${against || "nothing in git"}`;
const lines = [`recorded: ${receipt.ran.join(", ")} — ${counts.passed} passed, ` +
`${counts.failed} failed, ${skipped} skipped, against ${against || "nothing in git"}`];
if (!judged.passed) lines.push(`NOT PASSED: ${judged.why}`);
// As the runner's own spec line has it, "<test> (<file>) # <reason>": a test's name may hold any
// punctuation, so the file in brackets is what the lab module reads the line apart by.
for (const s of outcome.skips) lines.push(`skipped: ${s.test} (${s.file}) # ${s.reason || "no reason given"}`);
return lines.join("\n");
}
+100
View File
@@ -0,0 +1,100 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { IncusError } from "../src/incus/client.ts";
import {
buildBaseImage, failedMirrors, MIRRORS, mirrorlist, PACMAN_BACKOFF_MS, pacmanIn, type BaseDeps,
} from "../src/lifecycle/base.ts";
import { BASE_IMAGE_ALIAS } from "../src/lifecycle/place.ts";
/**
* Defends novox/hq issue 700: the base build installed from the one mirror the stock image names,
* mirrors.kernel.org, and failed twice in a morning when that mirror timed out and then answered 500.
*/
const KERNEL_500 = "error: failed retrieving file 'wireguard-tools-1.0.20260223-1-x86_64.pkg.tar.zst' from " +
"mirrors.kernel.org : The requested URL returned error: 500\nwarning: failed to retrieve some files\n" +
"error: failed to commit transaction (failed to retrieve some files)";
/** A hypervisor for one base build: pacman fails `fails` times per step, then installs. */
function aBuilder(fails: number, said = KERNEL_500) {
const ran: string[] = [];
const slept: number[] = [];
let mirrorlistWritten = "";
const tries = new Map<string, number>();
const deps: BaseDeps = {
incus: async (args) => {
ran.push(args.join(" "));
const at = args.indexOf("pacman");
if (at >= 0) {
const step = args.slice(at).join(" ");
const n = (tries.get(step) ?? 0) + 1;
tries.set(step, n);
if (n <= fails) throw new IncusError(args, said, 1);
}
const printf = args.join(" ").match(/printf '%s' '([\s\S]*)' > \/etc\/pacman.d\/mirrorlist/);
if (printf) mirrorlistWritten = printf[1]!;
return { stdout: "", stderr: "" };
},
incusOk: async (args) => {
const c = args.join(" ");
if (c.endsWith("cat /etc/pacman.d/mirrorlist")) return mirrorlistWritten;
if (c.includes("{{.ServerVersion}}")) return "29.8.2";
if (c.includes("InsecureRegistryCIDRs")) return "[127.0.0.0/8 192.0.2.0/24]";
if (c.startsWith("image list")) return BASE_IMAGE_ALIAS;
return "tool 1.0";
},
succeeds: async () => true,
sleep: async (ms) => { slept.push(ms); },
};
return { deps, ran, slept };
}
test("the builder is given several mirrors, the stock one last, before anything is installed", async () => {
const hosts = MIRRORS.map((m) => new URL(m.replace("$repo", "core").replace("$arch", "x86_64")).host);
assert.ok(new Set(hosts).size >= 4, `too few mirrors: ${hosts.join(", ")}`);
assert.equal(hosts.at(-1), "mirrors.kernel.org");
assert.ok(!mirrorlist().includes("'"), "the mirrorlist is written inside single quotes and holds one");
assert.equal(mirrorlist().split("\n").filter((l) => l.startsWith("Server = ")).length, MIRRORS.length);
const b = aBuilder(0);
await buildBaseImage(() => {}, b.deps);
const wrote = b.ran.findIndex((c) => c.includes("/etc/pacman.d/mirrorlist"));
const firstPacman = b.ran.findIndex((c) => c.includes(" pacman "));
assert.ok(wrote >= 0 && wrote < firstPacman, "pacman ran before the mirrorlist was written");
});
test("a pacman step that fails is tried again after a pause, saying which mirror failed", async () => {
const said: string[] = [];
const b = aBuilder(2);
await pacmanIn("builder", ["-S", "--noconfirm", "wireguard-tools"], (m) => said.push(m), b.deps);
assert.equal(b.ran.length, 3);
assert.deepEqual(b.slept, PACMAN_BACKOFF_MS.slice(0, 2));
assert.ok(PACMAN_BACKOFF_MS[1]! > PACMAN_BACKOFF_MS[0]!, "no backoff");
assert.equal(said.length, 2);
assert.match(said[0]!, /failed from mirrors\.kernel\.org \(error: failed retrieving file 'wireguard-tools/);
assert.match(said[0]!, /trying again in 10s \(attempt 2 of 4\)/);
});
test("a step that keeps failing stops the build, naming every mirror that failed", async () => {
const b = aBuilder(99, "error: too many errors from mirrors.kernel.org\nerror: failed retrieving file 'x' from ftp.fau.de : 500");
await assert.rejects(
pacmanIn("builder", ["-Sy", "--noconfirm", "docker"], () => {}, b.deps),
/failed 4 times; the mirrors that failed: mirrors\.kernel\.org, ftp\.fau\.de/,
);
assert.equal(b.ran.length, PACMAN_BACKOFF_MS.length + 1);
});
test("a whole base build rides out a mirror's bad minute on every package", async () => {
const b = aBuilder(1);
const built = await buildBaseImage(() => {}, b.deps);
assert.equal(built.alias, BASE_IMAGE_ALIAS);
const pacman = b.ran.filter((c) => c.includes(" pacman "));
assert.equal(pacman.length, 10, pacman.join("\n"));
});
test("the mirrors a failure names are read from what pacman said, each once", () => {
assert.deepEqual(failedMirrors(KERNEL_500), ["mirrors.kernel.org"]);
assert.deepEqual(failedMirrors("Connection timed out after 10000 milliseconds\nerror: too many errors from mirrors.kernel.org"),
["mirrors.kernel.org"]);
assert.deepEqual(failedMirrors("error: target not found: dockr"), []);
});
+76
View File
@@ -0,0 +1,76 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { busNotKept, runningFrom, streamsFrom, written, type Seen } from "./integration/bus-kept.ts";
/**
* The judgement of the bed that raises genesis past its broker step (novox/hq ADR 0333 §5, issue 590):
* exactly one bus container, it is mesh-broker-nats, a stream written before the step is still there, and
* the first node is still connected. Held here without a lab, so the bed's verdict is the judgement's and
* not whatever a bed happened to compare.
*/
const UPSTREAM = "nats@sha256:" + "b".repeat(64);
const MODULES = "192.0.2.10:5000/nats@sha256:" + "c".repeat(64);
const PORTS = "127.0.0.1:8222->8222/tcp, 0.0.0.0:5671->4222/tcp";
function jsz(streams: Array<[string, string, number, number]>): string {
return JSON.stringify({ account_details: [{ name: "MESH", stream_detail: streams.map(([name, created, messages, last]) =>
({ name, created, state: { messages, last_seq: last } })) }] });
}
const before: Seen = {
running: runningFrom(`mesh-store\tpostgres@sha256:aa\t0.0.0.0:5432->5432/tcp\nmesh-broker-nats\t${UPSTREAM}\t${PORTS}\n` +
"mesh-controller\t192.0.2.10:5000/mesh-controller@sha256:dd\t\n"),
streams: streamsFrom(jsz([["KV_mesh-controller_lease", "2026-10-13T00:10:00Z", 3, 7], ["EMPTY", "2026-10-13T00:10:01Z", 0, 0]])),
};
const kept: Seen = {
running: runningFrom(`mesh-store\tpostgres@sha256:aa\t0.0.0.0:5432->5432/tcp\nmesh-broker-nats\t${MODULES}\t${PORTS}\n` +
"mesh-controller\t192.0.2.10:5000/mesh-controller@sha256:dd\t\n"),
streams: streamsFrom(jsz([["KV_mesh-controller_lease", "2026-10-13T00:10:00Z", 3, 9]])),
};
const SAID = "broker — the foundation's bus, adopted as the nats module\n restarting mesh-broker-nats — once, onto the nats module's image";
const HERE = "anchor here last 3s ago";
test("a bus adopted in place is kept: one bus, the module's image, its streams, the node connected", () => {
assert.deepEqual(busNotKept(before, kept, HERE, SAID), []);
});
test("only a stream holding something counts as written before the step", () => {
assert.deepEqual(written(before).map((s) => s.name), ["KV_mesh-controller_lease"]);
});
test("a second bus beside the first is not kept: the fault genesis had before ADR 0333", () => {
const two: Seen = { ...kept, running: [...kept.running,
{ name: "mesh-broker", image: UPSTREAM, ports: "0.0.0.0:5671->4222/tcp" }] };
const wrong = busNotKept(before, two, HERE, SAID);
assert.equal(wrong.length, 1);
assert.match(wrong[0] as string, /2 bus containers.*mesh-broker \(/);
});
test("a bus under another name, or none, is not kept", () => {
const renamed: Seen = { ...kept, running: kept.running.map((r) => r.name === "mesh-broker-nats" ? { ...r, name: "nats-server" } : r) };
assert.ok(busNotKept(before, renamed, HERE, SAID).some((w) => /mesh-broker-nats does not run/.test(w)));
const none: Seen = { ...kept, running: kept.running.filter((r) => r.name !== "mesh-broker-nats") };
assert.ok(busNotKept(before, none, HERE, SAID).some((w) => /0 bus containers/.test(w)));
});
test("a bus still on the bundle's image was not adopted", () => {
const same: Seen = { ...kept, running: before.running };
assert.ok(busNotKept(before, same, HERE, SAID).some((w) => /still runs .* the image the bundle raised/.test(w)));
assert.ok(busNotKept(before, kept, HERE, "broker — adopted").some((w) => /did not say the bus restarts/.test(w)));
});
test("a stream gone or made again is not kept", () => {
const gone: Seen = { ...kept, streams: [] };
assert.ok(busNotKept(before, gone, HERE, SAID).some((w) => /KV_mesh-controller_lease .* is gone/.test(w)));
const again: Seen = { ...kept, streams: streamsFrom(jsz([["KV_mesh-controller_lease", "2026-10-13T00:40:00Z", 1, 1]])) };
assert.ok(busNotKept(before, again, HERE, SAID).some((w) => /was made again/.test(w)));
});
test("a node the mesh has not heard from since the step is not connected", () => {
assert.ok(busNotKept(before, kept, "anchor out of touch 4m", SAID).some((w) => /has not heard from the first node/.test(w)));
assert.ok(busNotKept(before, kept, "", SAID).some((w) => /no record/.test(w)));
});
+50
View File
@@ -0,0 +1,50 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { execFileSync } from "node:child_process";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { carry } from "./integration/forge.ts";
/**
* A bed's forge takes the lab's checkouts as they are: shallow (novox/hq issue 689). The lab module clones
* every repository with `--depth`, and the genesis bed's forge refused the push ("shallow update not
* allowed"), so the bed never reached its installer. Held here with a real git and no lab.
*/
const git = (cwd: string, ...args: string[]) =>
execFileSync("git", ["-C", cwd, ...args], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"],
env: { ...process.env, GIT_AUTHOR_NAME: "lab", GIT_AUTHOR_EMAIL: "lab@example.org",
GIT_COMMITTER_NAME: "lab", GIT_COMMITTER_EMAIL: "lab@example.org" } }).trim();
test("a shallow checkout, as the lab clones it, is carried into the bed's forge and served from it", () => {
const dir = mkdtempSync(join(tmpdir(), "mesh-lab-689-"));
try {
const origin = join(dir, "origin");
execFileSync("git", ["init", "-q", "-b", "main", origin]);
for (let i = 0; i < 5; i++) {
writeFileSync(join(origin, "file"), `${i}\n`);
git(origin, "add", "file");
git(origin, "commit", "-q", "-m", `commit ${i}`);
}
// As the lab module clones: shallow, over a transport that honours the depth.
const checkout = join(dir, "checkout");
execFileSync("git", ["clone", "-q", "--depth", "2", "--branch", "main", `file://${origin}`, checkout]);
assert.equal(git(checkout, "rev-parse", "--is-shallow-repository"), "true");
const bare = join(dir, "forge", "repo.git");
const head = carry(checkout, bare, ["main", "lab"]);
assert.equal(head, git(origin, "rev-parse", "HEAD"));
assert.equal(git(bare, "rev-parse", "refs/heads/main"), head);
assert.equal(git(bare, "rev-parse", "refs/heads/lab"), head);
// What the bed's machines do with it: clone over git's protocol, and find the commit under test.
const served = join(dir, "served");
execFileSync("git", ["clone", "-q", "--branch", "main", `file://${bare}`, served]);
assert.equal(git(served, "rev-parse", "HEAD"), head);
assert.equal(git(served, "show", "HEAD:file"), "4");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
@@ -16,7 +16,7 @@
*
* MESH_LAB_INCUS='sudo -n incus'
* MESH_LAB_HOST_BINARY=.../mesh-host/mesh-host
* MESH_LAB_BUNDLE=.../mesh-host/examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../mesh-host/examples/foundation-first-node.lock
* MESH_LAB_CATALOG=.../mesh-catalog/modules
*/
import { test, before, after } from "node:test";
+3 -6
View File
@@ -34,7 +34,7 @@
*
* MESH_LAB_INCUS='sudo -n incus'
* MESH_LAB_HOST_BINARY=<mesh-host>/mesh-host MESH_LAB_BOOTSTRAP_BINARY=<mesh-host>/mesh-bootstrap
* MESH_LAB_BUNDLE=<mesh-host>/examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=<mesh-host>/examples/foundation-first-node.lock
* MESH_LAB_CATALOG=<mesh-catalog>/modules MESH_LAB_MODULES=<mesh-controller>/examples/modules
* MESH_TOOLS=<mesh-tools> MESH_SDK=<mesh-sdk> (default: the checkouts beside this one)
* MESH_LAB_KEEP=1 leave it standing MESH_LAB_WARM=1 iterate from the adopted foundation
@@ -55,6 +55,7 @@ import { catalogueRoot } from "../../src/repos.ts";
import { ready, returnTo, keep } from "../../src/warm.ts";
import { labIsUsable, destroyAll, foundationBundle, catalogueModule, catalogueManifest } from "./harness.ts";
import { genesis, type GenesisOptions } from "./genesis.ts";
import { carry } from "./forge.ts";
const SCENARIO = "adoption";
const CONTROL = "anchor";
@@ -246,11 +247,7 @@ async function raiseForge(): Promise<Record<string, string>> {
const heads: Record<string, string> = {};
try {
for (const [name, checkout] of Object.entries(REPOS)) {
const bare = join(dir, `${name}.git`);
execFileSync("git", ["init", "-q", "--bare", bare]);
execFileSync("git", ["-C", checkout, "push", "-q", "--force", bare,
"HEAD:refs/heads/main", "HEAD:refs/heads/lab"], { stdio: "pipe" });
heads[name] = execFileSync("git", ["-C", checkout, "rev-parse", "HEAD"], { encoding: "utf8" }).trim();
heads[name] = carry(checkout, join(dir, `${name}.git`), ["main", "lab"]);
}
const tar = join(tmpdir(), `mesh-lab-forge-${process.pid}.tar`);
execFileSync("tar", ["-cf", tar, "-C", dir, "."]);
+1 -1
View File
@@ -36,7 +36,7 @@
* in for the authenticated cross-node call a manager node would make). The node-private-key stub of
* the earlier cut is GONE — the host uses its own real key.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* Build both runtime images into the local daemon first:
* scripts/build-module-runtime.sh anthropic-manager /tmp/anthropic-manager.tar
* scripts/build-module-runtime.sh anthropic-consumer /tmp/anthropic-consumer.tar
+1 -1
View File
@@ -10,7 +10,7 @@
* It needs the host binary, the foundation bundle, and the runtime image the scenario loads:
*
* MESH_LAB_HOST_BINARY=.../mesh-host
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-runtime-image.sh builds mesh-runtime-audit:development into the local daemon,
* which scenarios/audit-node.yml stocks — so no MESH_LAB_RUNTIME here; the host pulls it.
*/
+1 -1
View File
@@ -13,7 +13,7 @@
* would say nothing at all about removal — which is the half issue 129 asked for by name.
*
* MESH_LAB_HOST_BINARY=.../mesh-host
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* MESH_LAB_CATALOG=.../mesh-catalog/modules
* step-ca's image is upstream, pinned by the catalogue, pulled by the machine over its uplink.
* ca-trust carries no image: a script, a unit, and the machine's own systemd.
@@ -17,7 +17,7 @@
*
* All are assigned to the one anchor, pushed ONCE, and the node converges ONCE with every one up.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh {mongodb,unifi,postgres} build the runtime images into the local
* daemon; scenarios/catalogue-apps.yml stocks them. mongo:7, lscr.io/linuxserver/unifi-controller
* and synesthesiam/marytts must be in the local daemon to be stocked.
@@ -23,7 +23,7 @@
*
* All is assigned to the one anchor, pushed ONCE, and the node converges ONCE with both modules up.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh {sonarr,radarr} build the runtime images into the local daemon;
* scenarios/catalogue-media.yml stocks them. lscr.io/linuxserver/{sonarr,radarr} must be in the
* local daemon; the service images are pulled from the internet. Each *arr runtime is given a lab
@@ -22,7 +22,7 @@
* the seed), and mosquitto's provisioner — running in the assigned runtime — creates a scoped client
* for a contribution the mesh delivered, which then authenticates with the password the mesh minted.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh mosquitto builds mesh-runtime-mosquitto:development (carrying
* mosquitto_ctrl and the compiled bootstrap entrypoint) into the local daemon, which
* scenarios/catalogue-mqtt.yml stocks. eclipse-mosquitto:2 must be in the local daemon to be
@@ -18,7 +18,7 @@
* path is fulfilled by creating the consumer's login with the mesh-minted password — it seals nothing
* and needs no seal key (novox/hq ADR 0048, issue 032-provider-runtime-has-no-seal-key).
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh {postgres,redis,minio,plex} build the four runtime images into the
* local daemon; scenarios/catalogue-small.yml stocks them. postgres:17-alpine, redis:7-alpine and
* minio/minio:latest is pulled from the internet by the node itself.
@@ -22,7 +22,7 @@
* It needs a host binary and the foundation bundle:
*
* MESH_LAB_HOST_BINARY=.../mesh-host
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
*
* HELPER — stock the two runtimes into the local daemon before the run (some may already be there):
* scripts/build-module-runtime.sh postgres /tmp/postgres.tar
@@ -25,7 +25,7 @@
* registry by digest; the host pulls and runs it on the cadence.
*
* MESH_LAB_HOST_BINARY=.../mesh-host (feat/apply-schedule — the scheduler that fires the step)
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* MESH_LAB_MODULES=.../mesh-controller/examples/modules (feat/schedule-container — the parser that
* carries `schedule` through). scenarios/schedule-tick.yml stocks alpine:latest (which must be in
* the local daemon) and serves it by digest; there is no runtime image — schedtest is a bare tick.
@@ -17,7 +17,7 @@
* A tool would only fail if it were actually invoked without real creds — which this bed does not do,
* because the point is exactly that serving does not require them.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh confluence builds mesh-runtime-confluence:development into the
* local daemon; scenarios/tools-confluence.yml stocks it. There is no service image.
*/
@@ -16,7 +16,7 @@
* A tool would only fail if it were actually invoked without real creds — which this bed does not do,
* because the point is exactly that serving does not require them.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh gitlab builds mesh-runtime-gitlab:development into the local
* daemon; scenarios/tools-gitlab.yml stocks it. There is no service image — gitlab is tools-only.
*/
@@ -22,7 +22,7 @@
* It needs a host binary and the foundation bundle:
*
* MESH_LAB_HOST_BINARY=.../mesh-host
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
*
* HELPER — stock the three runtimes into the local daemon before the run (some may already be there):
* scripts/build-module-runtime.sh postgres /tmp/postgres.tar
+1 -1
View File
@@ -22,7 +22,7 @@
* The manifests are the catalogue's own (../mesh-catalog/modules/{mesh-vault,redis}/module.json), with
* the runtime artifact named as the image the lab built, exactly as the other assigned-* beds do.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh mesh-vault / redis build the two runtime images into the local
* daemon; scenarios/vault-node.yml stocks them.
*/
@@ -19,7 +19,7 @@
* by the firewall the nftables module derives (issues 055/056/057 in one bed).
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BOOTSTRAP_BINARY=.../mesh-bootstrap
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock (the TEMPLATE)
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock (the TEMPLATE)
* MESH_LAB_CATALOG=.../mesh-catalog/modules
* MESH_LAB_SOURCE=git://<forge>/mesh-controller.git MESH_LAB_SOURCE_REF=<commit>
* MESH_LAB_BUILD_REF=<branch or commit for module builds, default main>
+112
View File
@@ -0,0 +1,112 @@
/**
* Whether genesis kept its bus through the broker step (novox/hq ADR 0333 §5, issue 590).
*
* Pure, so the judgement is tested without a lab (test/bus-kept.test.ts) and the bed only gathers what it
* judges: the containers the anchor runs, the streams its bus holds, and what the mesh says of the node,
* once before the broker step and once after it.
*/
/** One running container, as `docker ps --format '{{.Names}}\t{{.Image}}\t{{.Ports}}'` says it. */
export interface Running {
name: string;
image: string;
ports: string;
}
/** One stream on the bus, as the server's monitor says it (`/jsz?accounts=true&streams=true`). */
export interface Stream {
account: string;
name: string;
created: string;
messages: number;
lastSeq: number;
}
/** What the anchor showed at one moment. */
export interface Seen {
running: Running[];
streams: Stream[];
}
/** The container the bundle raises the bus as, and the nats module keeps it as (ADR 0333 §1). */
export const BUS = "mesh-broker-nats";
/** `docker ps --format '{{.Names}}\t{{.Image}}\t{{.Ports}}'`, read. */
export function runningFrom(ps: string): Running[] {
return ps.split("\n").map((l) => l.trim()).filter(Boolean).map((l) => {
const [name = "", image = "", ports = ""] = l.split("\t");
return { name, image, ports };
});
}
/** The server's `/jsz?accounts=true&streams=true`, read: every stream of every account. */
export function streamsFrom(jsz: string): Stream[] {
const parsed = JSON.parse(jsz) as {
account_details?: Array<{
name?: string;
stream_detail?: Array<{ name?: string; created?: string; state?: { messages?: number; last_seq?: number } }>;
}>;
};
const out: Stream[] = [];
for (const account of parsed.account_details ?? []) {
for (const s of account.stream_detail ?? []) {
out.push({
account: account.name ?? "", name: s.name ?? "", created: s.created ?? "",
messages: s.state?.messages ?? 0, lastSeq: s.state?.last_seq ?? 0,
});
}
}
return out;
}
/**
* A bus container: one that serves the bus's port inside it, or carries a bus's name. Both, because a
* second bus raised by a module under another name is exactly what this looks for, and so is one raised on
* another port under the bus's name.
*/
export function busContainers(running: Running[]): Running[] {
return running.filter((r) => /->4222\/tcp/.test(r.ports) || r.name.startsWith("mesh-broker"));
}
/** The streams written before the step: the ones holding anything. */
export function written(seen: Seen): Stream[] {
return seen.streams.filter((s) => s.messages > 0 || s.lastSeq > 0);
}
/**
* Every way the bus was not kept, in sentences; empty when it was. `node` is the mesh's line for the first
* node from `node list`; `said` is what the installer printed on the run that crossed the step.
*/
export function busNotKept(before: Seen, after: Seen, node: string, said: string): string[] {
const wrong: string[] = [];
const buses = busContainers(after.running);
if (buses.length !== 1) {
wrong.push(`${buses.length} bus containers run after the broker step, not one: ` +
(buses.map((b) => `${b.name} (${b.image}, ${b.ports || "nothing published"})`).join("; ") || "none"));
}
const bus = after.running.find((r) => r.name === BUS);
if (!bus) {
wrong.push(`${BUS} does not run after the broker step`);
}
const was = before.running.find((r) => r.name === BUS);
if (bus && was && bus.image === was.image) {
wrong.push(`${BUS} still runs ${bus.image}, the image the bundle raised it from: the module's image never ` +
`took its place (ADR 0333 §3)`);
}
if (!/restarting\s+mesh-broker-nats/.test(said)) {
wrong.push("the installer did not say the bus restarts onto the module's image (ADR 0333 §3)");
}
for (const s of written(before)) {
const now = after.streams.find((a) => a.account === s.account && a.name === s.name);
if (!now) {
wrong.push(`the stream ${s.name} (${s.account}), holding ${s.messages} message(s) before the step, is gone`);
} else if (now.created !== s.created || now.lastSeq < s.lastSeq) {
wrong.push(`the stream ${s.name} (${s.account}) was made again: created ${s.created}, last ${s.lastSeq} ` +
`before the step; created ${now.created}, last ${now.lastSeq} after it`);
}
}
if (!/^\S+\s+here\b/.test(node.trim())) {
wrong.push(`the mesh has not heard from the first node since the step: ${node.trim() || "it has no record of it"}`);
}
return wrong;
}
+1 -1
View File
@@ -17,7 +17,7 @@
* It needs the host binary and the foundation bundle, like the mesh walk, plus a runtime image:
*
* MESH_LAB_HOST_BINARY=.../mesh-host
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* MESH_LAB_RUNTIME=.../mesh-runtime-audit.tar (docker save of the runtime+audit-logger image;
* built by scripts/build-runtime-image.sh)
*
+31
View File
@@ -0,0 +1,31 @@
/**
* A bed's forge: a checkout's HEAD carried into a fresh bare repository, which a git daemon then serves.
*
* **A shallow checkout is carried as it is** (novox/hq issue 689). The lab module checks every repository
* out shallow, and a fresh bare repository refuses a push whose history ends in a cut ("shallow update not
* allowed"), so the genesis bed raised its anchor and forge and then skipped. The bare repository accepts
* the cut (`receive.shallowUpdate`) and serves the history the checkout holds: the bed serves the commit
* under test, and nothing it raises reads further back than the run's checkout does. Unshallowing first is
* not an option in a lab run: the forge credential is gone once the run's clones are done, so a private
* repository could not be fetched again.
*/
import { execFileSync } from "node:child_process";
/**
* Initialise `bare` and push `checkout`'s HEAD into it under each of `branches`. Returns the commit served.
*/
export function carry(checkout: string, bare: string, branches: string[] = ["main"]): string {
execFileSync("git", ["init", "-q", "--bare", bare]);
// No collection after the push: one run in the background rewrites the objects while they are carried.
execFileSync("git", ["-C", bare, "config", "receive.autogc", "false"]);
// The lab's checkouts are shallow, and a bare repository refuses a shallow push without this.
execFileSync("git", ["-C", bare, "config", "receive.shallowUpdate", "true"]);
const [first, ...rest] = branches;
execFileSync("git", ["-C", checkout, "push", "-q", "--force", bare, `HEAD:refs/heads/${first}`], { stdio: "pipe" });
const head = execFileSync("git", ["-C", checkout, "rev-parse", "HEAD"], { encoding: "utf8" }).trim();
// The other branches are set in the bare repository itself: a shallow push of several refs at once has
// receive-pack fail every ref after the first ("remote failed to report status").
for (const branch of rest) execFileSync("git", ["-C", bare, "update-ref", `refs/heads/${branch}`, head]);
return head;
}
+1 -1
View File
@@ -31,7 +31,7 @@
* MESH_LAB_INCUS='sudo -n incus'
* MESH_LAB_HOST_BINARY=.../mesh-host/mesh-host
* MESH_LAB_BOOTSTRAP_BINARY=.../mesh-host/mesh-bootstrap
* MESH_LAB_BUNDLE=.../mesh-host/examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../mesh-host/examples/foundation-first-node.lock
* MESH_LAB_CATALOG=.../mesh-catalog/modules
* MESH_LAB_SOURCE=<forge>/mesh-controller.git MESH_LAB_SOURCE_REF=<commit>
* MESH_LAB_KEEP=1 to leave it standing afterwards
@@ -0,0 +1,265 @@
/**
* GENESIS KEEPS ITS BUS through the broker step (novox/hq ADR 0333 §5, issue 590).
*
* The installer raises the bus from its bundle and, at its broker step (step 18 of 24), adopts it as the
* catalogue's `nats` module. Until ADR 0333 the bundle raised the bus as `mesh-broker` on docker volumes,
* and the module keeps it as `mesh-broker-nats` on directories of the machine: adopting it would have raised
* a second bus beside the first, so the installer refused there and genesis stopped. ADR 0333 has the bundle
* raise the bus where the module keeps it, so the adoption replaces the server's image once and keeps
* everything else.
*
* The bed runs the installer from bare twice. The first run is given a catalogue without the `nats` module,
* so it stops at the broker step by name, before anything is registered — the moment just before the step,
* with everything genesis wrote on the bus by then. The bed reads the bus there: its containers, and every
* stream holding anything. The second run, given the whole catalogue, finds the steps before done and goes
* on through the broker step. The bed reads the bus again, and passes only when (ADR 0333 §5):
*
* - exactly one bus container runs, and it is mesh-broker-nats, on the module's image now;
* - every stream that held something before the step is still there, the same stream, as long or longer;
* - the mesh has heard from the first node since.
*
* The judgement is test/integration/bus-kept.ts, held without a lab by test/bus-kept.test.ts.
*
* **A bed that could not reach the step says so and does not fail**: a first run that stopped earlier than
* the broker step — a pull the internet refused, a build that broke — proves nothing about the step, and the
* test is skipped with the reason. A second run that stops AT the broker step is the fault, and fails. One
* that stops after it has crossed it: what it stopped on is reported, and the bus is judged.
*
* MESH_LAB_HOST_BINARY=<mesh-host>/mesh-host MESH_LAB_BOOTSTRAP_BINARY=<mesh-host>/mesh-bootstrap
* MESH_LAB_BUNDLE=<mesh-host>/examples/foundation-first-node.lock
* MESH_LAB_CATALOG=<mesh-catalog>/modules
* MESH_LAB_REPOS=<a directory holding mesh-controller, mesh-tools, mesh-tools-go, mesh-sdk and mesh-catalog checkouts>
* — served by the forge machine at each checkout's HEAD; the catalogue's is the one built from
* MESH_LAB_KEEP=1 leave it standing
*/
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { cpSync, existsSync, mkdtempSync, rmSync } from "node:fs";
import { execFileSync } from "node:child_process";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { loadScenario } from "../../src/declaration/parse.ts";
import { raise } from "../../src/lifecycle/raise.ts";
import { destroy, exec, push } from "../../src/lifecycle/operate.ts";
import { bootstrapBinaryPath, hostBinaryPath } from "../../src/lifecycle/place.ts";
import { labIsUsable, destroyAll, foundationBundle } from "./harness.ts";
import { genesis, type GenesisResult } from "./genesis.ts";
import { carry } from "./forge.ts";
import { BUS, busContainers, busNotKept, runningFrom, streamsFrom, written, type Seen } from "./bus-kept.ts";
const SCENARIO = "genesis-keeps-its-bus";
const ANCHOR = "anchor";
const FORGE = "forge";
const FORGE_ADDRESS = "192.0.2.30";
const REGISTRY = "192.0.2.10:5000";
/** The repositories the installer clones, served by the forge at each checkout's HEAD. */
// mesh-tools-go is found by the installer beside mesh-tools (hq issue 616): genesis builds the Go toolchain too.
const REPOS = ["mesh-controller", "mesh-tools", "mesh-tools-go", "mesh-sdk", "mesh-catalog"] as const;
/** The step this bed is about, as the installer names it. */
const BROKER = "broker";
/** The installer's steps before it, in its order: a first run stopped at one of these never reached it. */
const BEFORE_BROKER = ["preflight", "load", "build", "bundle", "apply", "verify", "enrol", "operator", "registry",
"publish", "control-plane", "retire", "builder", "packages", "sdk", "base", "store"];
const capability = await labIsUsable();
const binary = hostBinaryPath();
const installer = bootstrapBinaryPath();
const bundle = process.env["MESH_LAB_BUNDLE"] ?? "";
const catalogDir = process.env["MESH_LAB_CATALOG"] ?? "";
const repos = process.env["MESH_LAB_REPOS"] ?? "";
const KEEP = !!process.env["MESH_LAB_KEEP"];
const FIXED_ID = process.env["MESH_LAB_INSTANCE_ID"] ?? (KEEP ? "genesis-keeps-its-bus-live" : undefined);
const missingRepo = REPOS.find((r) => !repos || !existsSync(resolve(repos, r, ".git")));
const skip =
!capability.usable ? capability.why :
!binary ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" :
!installer ? "MESH_LAB_BOOTSTRAP_BINARY is not set to a built mesh-bootstrap" :
!bundle || !existsSync(bundle) ? "MESH_LAB_BUNDLE is not set to the genesis bundle" :
!catalogDir || !existsSync(join(catalogDir, "nats", "module.json")) ? "MESH_LAB_CATALOG is not set to mesh-catalog/modules" :
missingRepo ? `MESH_LAB_REPOS holds no ${missingRepo} checkout` :
false;
let instanceId = "";
/** Why the bed never reached the step, or "" when it did. */
let unreached = "";
let first: GenesisResult | undefined;
let second: GenesisResult | undefined;
let seenBefore: Seen | undefined;
let seenAfter: Seen | undefined;
let nodeLine = "";
const heads: Record<string, string> = {};
const report: string[] = [];
async function on(machine: string, command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> {
const { stdout } = await exec(instanceId, machine, ["sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`], timeoutMs);
const marker = stdout.lastIndexOf("__exit=");
if (marker < 0) return { out: stdout, ok: false };
return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" };
}
async function must(machine: string, command: string, timeoutMs?: number): Promise<string> {
const { out, ok } = await on(machine, command, timeoutMs);
if (!ok) throw new Error(`${machine}: ${command}\n${out}`);
return out;
}
/** Each repository at its checkout's HEAD, served from the forge machine over git's protocol as `main`. */
async function raiseForge(): Promise<void> {
const dir = mkdtempSync(join(tmpdir(), "mesh-lab-590-forge-"));
try {
for (const repo of REPOS) {
heads[repo] = carry(resolve(repos, repo), join(dir, `${repo}.git`));
}
const tar = join(tmpdir(), `mesh-lab-590-forge-${process.pid}.tar`);
execFileSync("tar", ["-cf", tar, "-C", dir, "."]);
await push(instanceId, FORGE, tar, "/tmp/forge.tar");
rmSync(tar, { force: true });
} finally {
rmSync(dir, { recursive: true, force: true });
}
// Owned by the daemon's user, or git refuses to serve it ("dubious ownership") and says nothing.
await must(FORGE, `mkdir -p /srv/git && tar --no-same-owner -xf /tmp/forge.tar -C /srv/git && chown -R root:root /srv/git && ` +
`git daemon --base-path=/srv/git --export-all --reuseaddr --detach --listen=${FORGE_ADDRESS} --pid-file=/run/git-daemon.pid`);
for (let i = 0; ; i++) {
if ((await on(ANCHOR, `git ls-remote git://${FORGE_ADDRESS}/mesh-catalog.git main`)).ok) break;
if (i >= 30) throw new Error(`the anchor cannot reach the forge at git://${FORGE_ADDRESS}/`);
await new Promise((r) => setTimeout(r, 2_000));
}
}
/** What the anchor shows of its bus: the containers it runs, and every stream the bus holds. */
async function look(): Promise<Seen> {
const ps = await must(ANCHOR, `docker ps --format '{{.Names}}\t{{.Image}}\t{{.Ports}}'`);
// The monitor, published on the anchor's loopback by the bundle and by the module alike.
let jsz = "";
for (let i = 0; ; i++) {
const asked = await on(ANCHOR, `curl -sf --max-time 10 'http://127.0.0.1:8222/jsz?accounts=true&streams=true'`);
if (asked.ok) { jsz = asked.out; break; }
if (i >= 30) throw new Error(`the bus's monitor does not answer on the anchor: ${asked.out}`);
await new Promise((r) => setTimeout(r, 2_000));
}
return { running: runningFrom(ps), streams: streamsFrom(jsz) };
}
function stoppedAt(r: GenesisResult): string {
return r.step.replace(/^step \d+ of \d+, /, "");
}
before(async () => {
if (skip) return;
const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), {
onProgress: (m) => console.log(`raise: ${m}`),
...(FIXED_ID ? { instanceId: FIXED_ID } : {}),
});
instanceId = raised.instanceId;
console.log(`INSTANCE ${instanceId}${KEEP ? " (KEEP — will be left standing)" : ""}`);
try {
await raiseForge();
report.push(` forge ${REPOS.map((r) => `${r} ${heads[r]?.slice(0, 8)}`).join(", ")}`);
} catch (err) {
unreached = `the forge was not raised, so the installer never ran: ${(err as Error).message}`;
return;
}
const forge = (repo: string) => `git://${FORGE_ADDRESS}/${repo}.git`;
const run = (catalogue: string) => genesis({
instanceId,
node: ANCHOR,
installer: installer as string,
catalogDir: catalogue,
catalogueModules: ["distribution", "build-agent", "postgres"],
bundleTemplate: foundationBundle(bundle, []),
registry: REGISTRY,
source: forge("mesh-controller"),
sourceRef: heads["mesh-controller"] as string,
toolsSource: forge("mesh-tools"),
sdkSource: forge("mesh-sdk"),
catalogSource: forge("mesh-catalog"),
// A refusal is the answer, and a re-run is the second half of this bed, not a retry.
attempts: 1,
verify: false,
log: (m) => console.log(m),
});
// 1. From bare to the broker step, and no further: the catalogue the installer reads has no nats module.
const withoutTheBus = mkdtempSync(join(tmpdir(), "mesh-lab-590-catalogue-"));
try {
const nats = join(catalogDir, "nats");
cpSync(catalogDir, withoutTheBus, { recursive: true, filter: (src) => src !== nats && !src.startsWith(nats + "/") });
first = await run(withoutTheBus);
} catch (err) {
unreached = `the installer could not be run: ${(err as Error).message}`;
return;
} finally {
rmSync(withoutTheBus, { recursive: true, force: true });
}
report.push(...first.report);
if (first.ok || stoppedAt(first) !== BROKER) {
unreached = first.ok
? "the first run finished without the nats module, so the broker step never stopped it"
: `the first run stopped at ${first.step || "no step it named"}, before the broker step:\n${first.why}`;
return;
}
if (!/nats module's manifest could not be read/.test(first.said)) {
unreached = `the first run stopped at the broker step for another reason than the missing module:\n${first.why}`;
return;
}
seenBefore = await look();
report.push(` before the step ${busContainers(seenBefore.running).map((b) => `${b.name} on ${b.image}`).join("; ")}`);
report.push(` ${written(seenBefore).length} stream(s) holding something: ` +
written(seenBefore).map((s) => `${s.name} (last ${s.lastSeq})`).join(", "));
if (written(seenBefore).length === 0) {
unreached = "nothing was written on the bus before the broker step, so whether its streams survive it says nothing";
return;
}
// 2. The whole catalogue: the steps before are done already, and the broker step adopts the bus.
try {
second = await run(catalogDir);
} catch (err) {
unreached = `the installer could not be run again: ${(err as Error).message}`;
return;
}
report.push(...second.report);
if (!second.ok && BEFORE_BROKER.includes(stoppedAt(second))) {
unreached = `the second run stopped at ${second.step}, before the broker step it was run again for:\n${second.why}`;
return;
}
seenAfter = await look();
report.push(` after the step ${busContainers(seenAfter.running).map((b) => `${b.name} on ${b.image}`).join("; ") || "no bus"}`);
const nodes = await on(ANCHOR, `docker exec mesh-controller /mesh-controller node list`, 60_000);
nodeLine = nodes.out.split("\n").map((l) => l.trim()).find((l) => l.startsWith(`${ANCHOR} `)) ?? "";
report.push(` node list ${nodeLine || nodes.out.trim()}`);
}, { timeout: 7_200_000 });
after(async () => {
if (KEEP) {
console.log(`\nLEFT STANDING: ${instanceId} — not destroyed (MESH_LAB_KEEP).`);
return;
}
if (instanceId) await destroy(instanceId);
await destroyAll(`${SCENARIO}-`);
}, { timeout: 900_000 });
test("genesis adopts its bus as the nats module in place: one bus, its streams kept, the node connected", {
skip, timeout: 7_200_000,
}, async (t) => {
console.log(report.join("\n"));
if (unreached) {
t.skip(`ADR 0333 §5 NOT REACHED: ${unreached}`);
return;
}
const ran = second as GenesisResult;
// Stopped AT the broker step: the adoption itself refused or failed — the fault, not a missed bed.
assert.ok(ran.ok || stoppedAt(ran) !== BROKER,
`genesis stopped at its broker step and did not adopt the bus:\n${ran.why}`);
if (!ran.ok) {
console.log(`\nthe second run crossed the broker step and stopped later, at ${ran.step} — not this bed's ` +
`question:\n${ran.why}`);
}
const wrong = busNotKept(seenBefore as Seen, seenAfter as Seen, nodeLine, ran.said);
assert.deepEqual(wrong, [], `genesis did not keep its bus through the broker step:\n - ${wrong.join("\n - ")}`);
console.log(`\nVERDICT: one bus, ${BUS}, on the module's image; ${written(seenBefore as Seen).length} stream(s) ` +
`kept; ${ANCHOR} connected.`);
});
+1 -1
View File
@@ -12,7 +12,7 @@
* MESH_LAB_INCUS='sudo -n incus'
* MESH_LAB_HOST_BINARY=.../mesh-host/mesh-host
* MESH_LAB_BOOTSTRAP_BINARY=.../mesh-host/mesh-bootstrap
* MESH_LAB_BUNDLE=.../mesh-host/examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../mesh-host/examples/foundation-first-node.lock
* MESH_LAB_CATALOG=.../mesh-catalog/modules
* MESH_LAB_KEEP=1 to leave it standing afterwards
*/
+1 -1
View File
@@ -112,7 +112,7 @@ export async function genesis(o: GenesisOptions): Promise<GenesisResult> {
const registry = o.registry ?? "127.0.0.1:5000";
// The registry's and the builder's. Not the control plane's: its manifest lives in its own
// repository and genesis takes it from the build it runs (novox/hq ADR 0069, 04-ISSUES/072).
const modules = o.catalogueModules ?? ["distribution", "builder"];
const modules = o.catalogueModules ?? ["distribution", "build-agent"];
const catalogueOnMachine = o.catalogueOnMachine ?? "/opt/mesh-catalog";
const log = o.log ?? (() => {});
+7 -11
View File
@@ -23,14 +23,14 @@ import { mustBeHandedOver, pinnedInto, referenceFor, repositoryOf, type HeldImag
/**
* The example bundle in mesh-host names a registry that no longer exists.
*
* `examples/foundation-first-node-nats.lock` was written **for a target**, and the target was the lab: it
* `examples/foundation-first-node.lock` was written **for a target**, and the target was the lab: it
* pins `192.0.2.250:5000/…` because that is where the registry the lab used to raise served from.
* That registry is gone, so those three references name nothing.
*
* Two of them are ordinary third-party images and belong to the internet. Rather than invent
* digests here, they are the ones the mesh's own modules already pin — mesh-catalog's `postgres`
* and `lavinmq` — so the foundation's store and broker are literally the images the mesh runs. The
* third, mesh-controller, exists in no registry at all and becomes the ID the machine holds it under.
* Two of them are ordinary third-party images and belong to the internet. The store's is the one
* mesh-catalog's `postgres` module already pins, so the foundation's store is literally the image the
* mesh runs; the bus's is below. The third, mesh-controller, exists in no registry at all and becomes
* the ID the machine holds it under.
*
* **The bundle itself should be fixed in mesh-host**, and this substitution deleted with it. It is
* here because the file lives in another repository and because a fixture that lies about where an
@@ -38,11 +38,9 @@ import { mustBeHandedOver, pinnedInto, referenceFor, repositoryOf, type HeldImag
*/
const UPSTREAM_STORE =
"postgres@sha256:7456ef82e5f5bc43d997f4781bbd7c0d6389bff397564649a356e206ba473aee";
const UPSTREAM_BROKER =
"cloudamqp/lavinmq@sha256:3eb54c12916d700a978c2ea86e6362cd4974b0e3189508718006d4e6d341246b";
/**
* And the bus, for `foundation-first-node-nats.lock` — the bundle the mesh raises since it stopped
* speaking AMQP (novox/hq ADR 0131). The digest is the bundle's own: what the registry served was a
* And the bus, for `foundation-first-node.lock` — the one genesis bundle, which raises the bus the
* mesh runs on since it stopped speaking AMQP (novox/hq ADR 0131, ADR 0331). The digest is the bundle's own: what the registry served was a
* copy of the upstream image, so the same digest resolves on Docker Hub, and this is a prefix being
* removed rather than a reference being replaced.
*/
@@ -59,8 +57,6 @@ const UPSTREAM_BUS =
export function foundationBundle(path: string, held: HeldImage[]): string {
let text = readFileSync(path, "utf8");
text = text.replaceAll(/[A-Za-z0-9_.:-]+\/postgres@sha256:[0-9a-f]{64}/g, UPSTREAM_STORE);
text = text.replaceAll(
/[A-Za-z0-9_.:-]+\/cloudamqp\/lavinmq@sha256:[0-9a-f]{64}/g, UPSTREAM_BROKER);
text = text.replaceAll(/[A-Za-z0-9_.:-]+:[0-9]+\/nats@sha256:[0-9a-f]{64}/g, UPSTREAM_BUS);
return pinnedInto(text, held);
}
@@ -13,7 +13,7 @@
* It needs a host binary, the foundation bundle and the control plane's image:
*
* MESH_LAB_HOST_BINARY=.../mesh-host
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* (mesh-controller:development, from mesh-controller's `make image`)
*/
import { test, before, after } from "node:test";
+1 -1
View File
@@ -25,7 +25,7 @@
* its vhost — the provider named the vhost after the login — and nothing is hardcoded; the provider's
* `serves` carries the port so the consumer references `${bound:amqp:port}`.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
*
* HELPER — stock the two runtimes into the local daemon before the run (some may already be there):
* scripts/build-module-runtime.sh lavinmq /tmp/lavinmq.tar
+1 -1
View File
@@ -15,7 +15,7 @@
* asserts the templated URL and that a request to it reaches the running server (ollama answers
* /v1/models even with no model pulled — the wiring is what is proven, not a model's output).
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* No module runtime image is built — both modules are pure declaration.
*/
@@ -11,7 +11,7 @@
* Mint on one side and create on the other agreeing, with no shared key and nothing placed by the
* test, is the entire provider/consumer contract working as one thing.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh redis builds mesh-runtime-redis:development, which
* scenarios/redis-node.yml stocks.
*/
+1 -1
View File
@@ -9,7 +9,7 @@
* It needs a host binary and the foundation bundle:
*
* MESH_LAB_HOST_BINARY=.../mesh-host
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
*
* The bundle's image references are rewritten to the ones this scenario's own registry serves.
* A digest belongs to whatever registry serves it, so a committed bundle names a registry that is
+3 -3
View File
@@ -31,7 +31,7 @@
* MESH_LAB_INCUS='sudo -n incus'
* MESH_LAB_HOST_BINARY=.../mesh-host/mesh-host
* MESH_LAB_BOOTSTRAP_BINARY=.../mesh-host/mesh-bootstrap
* MESH_LAB_BUNDLE=.../mesh-host/examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../mesh-host/examples/foundation-first-node.lock
* MESH_LAB_CATALOG=.../mesh-catalog/modules
* MESH_LAB_SOURCE=<forge>/mesh-controller.git MESH_LAB_SOURCE_REF=<commit>
* MESH_LAB_KEEP=1 to leave it standing afterwards
@@ -109,7 +109,7 @@ const CONTROL_PLANE = { module: "mesh-controller", repo: "mesh-controller", path
* the registry and the builder — are here too: they are carried in, and a mesh missing any of them
* is not one.
*/
const MUST_HOLD = ["mesh-controller", "distribution", "builder", "mesh-tools", "postgres",
const MUST_HOLD = ["mesh-controller", "distribution", "build-agent", "mesh-tools", "postgres",
"mesh-catalog", "lavinmq", "mesh-vault", "amqp-ping"];
const MUST_RUN = ["mesh-controller", "mesh-registry", "mesh-broker", "mesh-store",
"mesh-postgres", "mesh-catalog", "mesh-lavinmq", "mesh-vault", "amqp-ping"];
@@ -898,7 +898,7 @@ before(async () => {
// does. novox/hq issue 050 — on a fresh mesh the modules built before the catalogue existed
// are exactly the ones it needed in order to exist, so the hole is always the foundation.
const missing = MUST_HOLD.filter((m) =>
!["distribution", "builder"].includes(m) && !held.includes(m));
!["distribution", "build-agent"].includes(m) && !held.includes(m));
assert.deepEqual(missing, [],
`the catalogue does not hold ${missing.join(", ")} — the mesh built them and its own ` +
`record has no trace of it (novox/hq issue 050):\n${held}`);
+1 -1
View File
@@ -12,7 +12,7 @@
* ordinary sealed-delivery path — with NO manager, NO refresh, NO access/refresh split, NO usage. The
* whole of OpenAI's integration in the control plane is one registry line (vendor -> static-key).
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* Build the consumer runtime image into the local daemon first:
* scripts/build-module-runtime.sh openai-consumer /tmp/openai-consumer.tar
*/
+235
View File
@@ -0,0 +1,235 @@
/**
* REPLAY R92 — genesis pushes to its own registry on a runtime nobody told to trust it (novox/hq issue 92).
*
* On the first real genesis the push of the control plane's image into the registry genesis had just
* raised was refused: `Get "https://<address>/v2/": http: server gave HTTP response to HTTPS client`. The
* registry speaks plain HTTP on purpose (ADR 0082), and the trust that lets a runtime speak to it was
* written by the `docker` module, installed six steps after the push. The fix (mesh-host #91) has genesis
* write that trust into the runtime's configuration itself, and confirm it through `docker info`, before
* anything is pushed.
*
* **The lab hid it.** Its base image trusts the documentation ranges as plain-HTTP registries before any bed
* starts (src/lifecycle/base.ts), so every genesis bed pushed to a registry its runtime already trusted. This
* bed takes that trust back off the anchor — the base's file removed, the runtime restarted, and the runtime
* asked that it now trusts loopback and nothing else — and then runs the installer as an operator does, with
* `--registry 192.0.2.10:5000`.
*
* What it holds: genesis gets past its publish step, the registry it raised serves the control plane's image
* it pushed by the address it was given, and the runtime trusts that registry by the mesh's name too, which
* the builder pushes to later. Before the fix the push is refused, and the bed fails naming it.
*
* **A bed that could not reach the step it replays says so and does not fail**: a genesis stopped before
* its registry step — a pull the internet refused, a build that broke — proves nothing about the order this
* replays, and the test is skipped with the reason, which the prover reads as an error, never as the fault
* reproduced. Genesis is run once and not asked to finish: the steps after the control plane read
* repositories this bed does not serve, and are not what it replays.
*
* MESH_LAB_HOST_BINARY=<mesh-host>/mesh-host MESH_LAB_BOOTSTRAP_BINARY=<mesh-host>/mesh-bootstrap
* MESH_LAB_BUNDLE=<mesh-host>/examples/foundation-first-node.lock (the installer refuses the bus one)
* MESH_LAB_CATALOG=<mesh-catalog>/modules MESH_REPLAY_CONTROLLER=<a mesh-controller checkout>
* MESH_LAB_KEEP=1 leave it standing
*
* Run by `go run ./replays/cmd/prove R92`, which builds all of that at the fix and at the commit before it.
*/
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { existsSync, mkdtempSync, rmSync } from "node:fs";
import { execFileSync } from "node:child_process";
import { tmpdir } from "node:os";
import { join, resolve } from "node:path";
import { loadScenario } from "../../src/declaration/parse.ts";
import { raise } from "../../src/lifecycle/raise.ts";
import { destroy, exec, push } from "../../src/lifecycle/operate.ts";
import { bootstrapBinaryPath, hostBinaryPath } from "../../src/lifecycle/place.ts";
import { labIsUsable, destroyAll, foundationBundle } from "./harness.ts";
import { genesis, type GenesisResult } from "./genesis.ts";
import { carry } from "./forge.ts";
const SCENARIO = "genesis-untrusted";
const ANCHOR = "anchor";
const FORGE = "forge";
const ANCHOR_ADDRESS = "192.0.2.10";
const FORGE_ADDRESS = "192.0.2.30";
/** The registry as the operator gives it: the anchor's own address and the registry's port. */
const REGISTRY = `${ANCHOR_ADDRESS}:5000`;
/** And as the mesh names it: the node, the mesh's internal suffix, the same port. */
const REGISTRY_BY_NAME = `${ANCHOR}.internal:5000`;
/** The installer's steps up to and including the one this replays, in its order. */
const UP_TO_PUBLISH = ["preflight", "load", "build", "bundle", "apply", "verify", "enrol", "operator", "registry", "publish"];
/** The steps before the registry's: a stop there never reached what this replays. */
const BEFORE_REGISTRY = UP_TO_PUBLISH.slice(0, UP_TO_PUBLISH.indexOf("registry"));
const capability = await labIsUsable();
const binary = hostBinaryPath();
const installer = bootstrapBinaryPath();
const bundle = process.env["MESH_LAB_BUNDLE"] ?? "";
const catalogDir = process.env["MESH_LAB_CATALOG"] ?? "";
const controller = process.env["MESH_REPLAY_CONTROLLER"] ?? "";
const KEEP = !!process.env["MESH_LAB_KEEP"];
const FIXED_ID = process.env["MESH_LAB_INSTANCE_ID"] ?? (KEEP ? "genesis-untrusted-live" : undefined);
const skip =
!capability.usable ? capability.why :
!binary ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" :
!installer ? "MESH_LAB_BOOTSTRAP_BINARY is not set to a built mesh-bootstrap" :
!bundle || !existsSync(bundle) ? "MESH_LAB_BUNDLE is not set to a foundation template" :
!catalogDir || !existsSync(catalogDir) ? "MESH_LAB_CATALOG is not set to mesh-catalog/modules" :
!controller || !existsSync(resolve(controller, ".git")) ? "MESH_REPLAY_CONTROLLER is not set to a mesh-controller checkout" :
false;
let instanceId = "";
/** Why the bed never reached the step it replays, or "" when it did. */
let unreached = "";
let result: GenesisResult | undefined;
let untrusted = "";
const report: string[] = [];
async function on(machine: string, command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> {
const { stdout } = await exec(instanceId, machine, ["sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`], timeoutMs);
const marker = stdout.lastIndexOf("__exit=");
if (marker < 0) return { out: stdout, ok: false };
return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" };
}
async function must(machine: string, command: string, timeoutMs?: number): Promise<string> {
const { out, ok } = await on(machine, command, timeoutMs);
if (!ok) throw new Error(`${machine}: ${command}\n${out}`);
return out;
}
/** The control plane's repository at the checkout's HEAD, served from the forge machine over git's protocol. */
async function raiseForge(): Promise<string> {
const dir = mkdtempSync(join(tmpdir(), "mesh-lab-r92-forge-"));
let head: string;
try {
head = carry(controller, join(dir, "mesh-controller.git"));
const tar = join(tmpdir(), `mesh-lab-r92-forge-${process.pid}.tar`);
execFileSync("tar", ["-cf", tar, "-C", dir, "."]);
await push(instanceId, FORGE, tar, "/tmp/forge.tar");
rmSync(tar, { force: true });
} finally {
rmSync(dir, { recursive: true, force: true });
}
// Owned by the daemon's user, or git refuses to serve it ("dubious ownership") and says nothing.
await must(FORGE, `mkdir -p /srv/git && tar --no-same-owner -xf /tmp/forge.tar -C /srv/git && chown -R root:root /srv/git && ` +
`git daemon --base-path=/srv/git --export-all --reuseaddr --detach --listen=${FORGE_ADDRESS} --pid-file=/run/git-daemon.pid`);
for (let i = 0; ; i++) {
if ((await on(ANCHOR, `git ls-remote git://${FORGE_ADDRESS}/mesh-controller.git main`)).ok) break;
if (i >= 30) throw new Error(`the anchor cannot reach the forge at git://${FORGE_ADDRESS}/`);
await new Promise((r) => setTimeout(r, 2_000));
}
return head;
}
/**
* The anchor's runtime made a bare machine's: the base's trust of the documentation ranges taken off, and
* the runtime asked that it trusts no registry but loopback. Before genesis, so nothing runs that a restart
* would stop. Answers what the runtime said.
*/
async function untrust(): Promise<string> {
await must(ANCHOR, `rm -f /etc/docker/daemon.json && systemctl restart docker`, 180_000);
for (let i = 0; ; i++) {
if ((await on(ANCHOR, `docker info >/dev/null`)).ok) break;
if (i >= 60) throw new Error("the anchor's runtime did not answer after its trust was taken off");
await new Promise((r) => setTimeout(r, 2_000));
}
const config = (await must(ANCHOR, `docker info --format '{{json .RegistryConfig}}'`)).trim();
const parsed = JSON.parse(config) as { InsecureRegistryCIDRs?: string[]; IndexConfigs?: Record<string, { Secure?: boolean }> };
const cidrs = parsed.InsecureRegistryCIDRs ?? [];
const insecureNames = Object.entries(parsed.IndexConfigs ?? {}).filter(([, c]) => c.Secure === false).map(([n]) => n);
if (cidrs.some((c) => c !== "127.0.0.0/8" && c !== "::1/128") || insecureNames.length > 0) {
throw new Error(`the anchor's runtime still trusts more than loopback after the base's trust was taken off: ${config}`);
}
return `trusts ${cidrs.join(", ") || "nothing"} as plain-HTTP registries, and no registry by name`;
}
before(async () => {
if (skip) return;
const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), {
onProgress: (m) => console.log(`raise: ${m}`),
...(FIXED_ID ? { instanceId: FIXED_ID } : {}),
});
instanceId = raised.instanceId;
console.log(`INSTANCE ${instanceId}${KEEP ? " (KEEP — will be left standing)" : ""}`);
let head = "";
try {
head = await raiseForge();
report.push(` forge git://${FORGE_ADDRESS}/mesh-controller.git at ${head.slice(0, 8)}`);
untrusted = await untrust();
report.push(` anchor's runtime ${untrusted}`);
} catch (err) {
unreached = `the bed was not prepared, so the installer never ran: ${(err as Error).message}`;
return;
}
const forge = (repo: string) => `git://${FORGE_ADDRESS}/${repo}.git`;
try {
result = await genesis({
instanceId,
node: ANCHOR,
installer: installer as string,
catalogDir,
// The registry's manifest is all the steps this replays read from the catalogue.
catalogueModules: ["distribution"],
bundleTemplate: foundationBundle(bundle, []),
registry: REGISTRY,
source: forge("mesh-controller"),
sourceRef: head,
// Named because the installer requires them; served by nobody, since the steps that clone them
// come after the one replayed here.
toolsSource: forge("mesh-tools"),
sdkSource: forge("mesh-sdk"),
catalogSource: forge("mesh-catalog"),
attempts: 1,
verify: false,
log: (m) => console.log(m),
});
} catch (err) {
unreached = `the installer could not be run: ${(err as Error).message}`;
return;
}
report.push(...result.report);
const stoppedAt = result.step.replace(/^step \d+ of \d+, /, "");
if (!result.ok && (BEFORE_REGISTRY.includes(stoppedAt) || !result.step)) {
unreached = `genesis stopped at ${result.step || "no step it named"}, before its registry step, so the order ` +
`this replays was never reached:\n${result.why}`;
}
}, { timeout: 3_600_000 });
after(async () => {
if (KEEP) {
console.log(`\nLEFT STANDING: ${instanceId} — not destroyed (MESH_LAB_KEEP).`);
return;
}
if (instanceId) await destroy(instanceId);
await destroyAll(`${SCENARIO}-`);
}, { timeout: 900_000 });
test("genesis pushes to the registry it raised on a runtime nobody told to trust it", {
skip, timeout: 3_600_000,
}, async (t) => {
console.log(report.join("\n"));
if (unreached) {
t.skip(`R92 NOT REACHED: ${unreached}`);
return;
}
const ran = result as GenesisResult;
const stoppedAt = ran.step.replace(/^step \d+ of \d+, /, "");
// The incident's own words, and the step it stopped at.
assert.doesNotMatch(ran.said, /server gave HTTP response to HTTPS client/,
`the runtime refused to push to ${REGISTRY} over plain HTTP — the registry was not trusted before the push ` +
`(stopped at ${ran.step || "no step"}):\n${ran.why}`);
assert.ok(ran.ok || !UP_TO_PUBLISH.includes(stoppedAt),
`genesis stopped at ${ran.step}, before its control plane was published to ${REGISTRY}:\n${ran.why}`);
// The registry agreeing: the image genesis pushed, by the address it was given.
const tags = (await on(ANCHOR, `curl -s --max-time 10 http://${REGISTRY}/v2/mesh-controller/tags/list`)).out.trim();
assert.match(tags, /"genesis"/, `${REGISTRY} does not serve the control plane genesis pushed: ${tags || "no answer"}`);
// And the runtime trusts the registry by the mesh's name as well: the builder pushes there later.
const config = (await must(ANCHOR, `docker info --format '{{json .RegistryConfig.IndexConfigs}}'`)).trim();
const byName = (JSON.parse(config) as Record<string, { Secure?: boolean }>)[REGISTRY_BY_NAME];
assert.ok(byName && byName.Secure === false,
`the runtime does not trust ${REGISTRY_BY_NAME}, the name the mesh pushes to its registry by: ${config}`);
});
+1 -1
View File
@@ -25,7 +25,7 @@
* publicly-trusted certificate and answering an HTTP-01 challenge at the name — is proven separately
* by certificates.test.ts, which drives the same proxy binary against a real ACME server (Pebble).
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-route-proxy-image.sh builds mesh-route-proxy:development into the local daemon;
* scenarios/route-forwarding.yml stocks it and alpine:latest, and serves both by digest.
*/
@@ -12,7 +12,7 @@
* the container was replaced (a new container id) and the config on disk carries the new value.
* It builds the host from source (no --no-build), because the behaviour under test is the host's.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh grafana builds mesh-runtime-grafana:development, which
* scenarios/grafana-node.yml stocks.
*/
+1 -1
View File
@@ -14,7 +14,7 @@
* It needs a host binary and the foundation bundle:
*
* MESH_LAB_HOST_BINARY=.../mesh-host
* MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
*/
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
+1 -1
View File
@@ -20,7 +20,7 @@
* references of OURS are rewritten to the IDs the machine holds, and the co-located
* host-port collisions are remapped at load time (see REMAP).
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
*/
import { test, before, after } from "node:test";
+2 -2
View File
@@ -50,7 +50,7 @@
* (whole-mesh-full-live) and NOT torn down — it is left standing and browsable. Without it the bed
* behaves like every other: raise in before(), destroy in after().
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* MESH_LAB_BOOTSTRAP_BINARY=.../mesh-bootstrap MESH_LAB_CATALOG=.../mesh-catalog/modules
*/
@@ -119,7 +119,7 @@ const CATALOGUE_ON_MACHINE = "/opt/mesh-catalog";
* a third, this list is where the bed finds out, by the installer saying which manifest it could
* not read.
*/
const CATALOGUE_MODULES = ["distribution", "builder"];
const CATALOGUE_MODULES = ["distribution", "build-agent"];
/**
* Where this mesh keeps its own images, as the anchor reaches it.
+1 -1
View File
@@ -28,7 +28,7 @@
* host ports here (container ports unchanged); the provider ports the consumers actually connect to
* (postgres 5432, minio 9000, mongodb 27017, mssql 1433) are left as-is. See REMAP below.
*
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node-nats.lock
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
* scripts/build-module-runtime.sh builds one runtime per module that has code; the route-proxy image
* is built by scripts/build-route-proxy-image.sh; scenarios/whole-mesh-novox.yml stocks them all
* alongside every server image.
+179
View File
@@ -0,0 +1,179 @@
// A run in which no test passed has proven nothing, and says so (novox/hq issue 693).
//
// Seen on the lab: the genesis bed stopped before its subject, skipped its only test as "NOT REACHED",
// and the suite printed "0 passed, 0 failed" and exited 0, so the lab recorded the run as passed.
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { judge, type Receipt, endToEnd } from "../src/lastrun.ts";
import { judgeRun, oneLine, type Skip } from "../src/outcomes.ts";
import { counted, exitOf, reportOn } from "../src/suite.ts";
// The suite as a person runs it: not as a child of this test runner, which would have it report to us.
const suiteEnv = (state: string): NodeJS.ProcessEnv => {
const env: NodeJS.ProcessEnv = { ...process.env, XDG_STATE_HOME: state, NO_COLOR: "1" };
delete env["NODE_TEST_CONTEXT"];
return env;
};
const lab = fileURLToPath(new URL("..", import.meta.url));
const unreached: Skip = {
file: "test/integration/genesis-keeps-its-bus.test.ts",
test: "genesis adopts its bus",
reason: "ADR 0333 §5 NOT REACHED: the first run stopped at step 16 of 24, base, before the broker step:\n docker exited 1",
};
const noBed: Skip = { file: "test/integration/mesh.test.ts", test: "a builder builds", reason: "set MESH_LAB_BUILDER to a built mesh-builder" };
test("a run whose only test skipped did not pass", () => {
const said = judgeRun({ passed: 0, failed: 0 }, [unreached]);
assert.equal(said.passed, false);
assert.match(said.why, /no test passed: 1 skipped, 1 of them as NOT REACHED/);
});
test("a run that ran no test at all did not pass", () => {
assert.deepEqual(judgeRun({ passed: 0, failed: 0 }, []), { passed: false, why: "no test passed: none ran" });
});
test("a NOT REACHED skip beside tests that passed is still not passed", () => {
const said = judgeRun({ passed: 3, failed: 0 }, [unreached]);
assert.equal(said.passed, false);
assert.match(said.why, /1 test\(s\) skipped as NOT REACHED/);
});
// The design limit: a skip its file declares expected, beside tests that passed, leaves the run passed.
test("an expected skip beside tests that passed leaves the run passed", () => {
assert.deepEqual(judgeRun({ passed: 21, failed: 0 }, [noBed]), { passed: true, why: "" });
});
test("the suite's exit is not 0 for a run that did not pass, and is the runner's otherwise", () => {
assert.equal(exitOf(0, { passed: 0, failed: 0 }, [unreached]), 1);
assert.equal(exitOf(0, { passed: 21, failed: 0 }, [noBed]), 0);
assert.equal(exitOf(1, { passed: 20, failed: 1 }, []), 1);
assert.equal(exitOf(0, { passed: null, failed: null }, []), 0, "unread totals are reportOn's case, not this one");
});
test("the recorded line says what skipped and why, and NOT PASSED", () => {
let kept: unknown;
const said = reportOn({ passed: 0, failed: 0, skipped: 1 }, [unreached], (_p, _f, outcome) => {
kept = outcome;
return { against: { "mesh-lab": "7ed7da1" }, ran: [unreached.file] };
});
assert.match(said, /— 0 passed, 0 failed, 1 skipped, against mesh-lab 7ed7da1/);
assert.match(said, /^NOT PASSED: no test passed: 1 skipped, 1 of them as NOT REACHED$/m);
assert.match(said, /^skipped: genesis adopts its bus \(test\/integration\/genesis-keeps-its-bus\.test\.ts\) # ADR 0333 §5 NOT REACHED: the first run stopped at step 16 of 24, base, before the broker step: docker exited 1$/m);
assert.deepEqual(kept, {
skipped: 1,
skips: [{ ...unreached, reason: oneLine(unreached.reason) }],
notPassed: "no test passed: 1 skipped, 1 of them as NOT REACHED",
});
});
// A test that prints a summary-shaped line before it skips must not stand for the runner's summary.
test("the runner's last summary is read, not a line a test printed before it", () => {
const output = "\u2139 pass 5\n\u2139 fail 0\n\u2139 skipped 0\n\ufe63 genesis adopts its bus (1ms) # NOT REACHED\n" +
"\u2139 tests 1\n\u2139 pass 0\n\u2139 fail 0\n\u2139 skipped 1\n";
const counts = counted(output);
assert.deepEqual(counts, { passed: 0, failed: 0, skipped: 1 });
assert.equal(exitOf(0, counts, [unreached]), 1);
});
test("NOT REACHED is matched in any case", () => {
const said = judgeRun({ passed: 2, failed: 0 }, [{ ...noBed, reason: "the broker step was not reached" }]);
assert.equal(said.passed, false);
});
test("the runner's skipped total is read", () => {
assert.deepEqual(counted("ℹ tests 1\nℹ pass 0\nℹ fail 0\nℹ skipped 1\n"), { passed: 0, failed: 0, skipped: 1 });
});
test("a receipt of a run that proved nothing is not current, and says why", () => {
const receipt: Receipt = {
at: "2026-10-14T00:33:11Z", passed: 0, failed: 0, against: { "mesh-lab": "aaa" }, ran: [endToEnd],
skipped: 1, skips: [unreached], notPassed: "no test passed: 1 skipped",
};
const said = judge(receipt, new Date("2026-10-14T01:00:00Z"), { "mesh-lab": "aaa" });
assert.equal(said.current, false, said.lines.join("\n"));
const text = said.lines.join("\n");
assert.match(text, /did not pass: no test passed: 1 skipped/);
assert.match(text, /skipped: genesis adopts its bus — ADR 0333 §5 NOT REACHED/);
// An older receipt, from before the fields, with nothing passed, is judged by its count.
const older = judge({ at: receipt.at, passed: 0, failed: 0, against: receipt.against, ran: [endToEnd] },
new Date("2026-10-14T01:00:00Z"), { "mesh-lab": "aaa" });
assert.equal(older.current, false);
});
// The real runner, end to end: the suite command against a file whose only test skips as NOT REACHED,
// the way the genesis bed does.
test("the suite exits non-zero for a file whose only test skipped as NOT REACHED, and records why", () => {
const dir = mkdtempSync(join(tmpdir(), "mesh-lab-693-"));
try {
const file = join(dir, "unreached.test.ts");
writeFileSync(file, [
`import { test } from "node:test";`,
`test("genesis adopts its bus", (t) => {`,
` console.log("a line the bed printed");`,
` t.skip("ADR 0333 §5 NOT REACHED: the first run stopped at step 16 of 24, base:\\nNotFound: content digest");`,
`});`,
].join("\n"));
const ran = spawnSync(process.execPath, ["--experimental-strip-types", join(lab, "src", "cli.ts"), "suite", "--no-build", file], {
cwd: lab, encoding: "utf8", env: suiteEnv(dir),
});
assert.equal(ran.status, 1, `the suite exited ${ran.status}:\n${ran.stdout}${ran.stderr}`);
assert.match(ran.stdout, /0 passed, 0 failed, 1 skipped/);
assert.match(ran.stdout, /^NOT PASSED: no test passed: 1 skipped, 1 of them as NOT REACHED$/m);
assert.match(ran.stdout, /^skipped: genesis adopts its bus \(.*unreached\.test\.ts\) # ADR 0333 §5 NOT REACHED: the first run stopped at step 16 of 24, base: NotFound: content digest$/m);
const receipt = JSON.parse(readFileSync(join(dir, "mesh-lab", "last-run.json"), "utf8")) as Receipt;
assert.equal(receipt.skipped, 1);
assert.equal(receipt.notPassed, "no test passed: 1 skipped, 1 of them as NOT REACHED");
assert.match(receipt.skips?.[0]?.reason ?? "", /NOT REACHED/);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test("the suite does not take a summary a test printed for the runner's", () => {
const dir = mkdtempSync(join(tmpdir(), "mesh-lab-693-"));
try {
const file = join(dir, "forged.test.ts");
writeFileSync(file, [
`import { test } from "node:test";`,
`test("genesis adopts its bus", (t) => {`,
` console.log("\\u2139 pass 5\\n\\u2139 fail 0");`,
` t.skip("ADR 0333 §5 NOT REACHED: stopped at step 16");`,
`});`,
].join("\n"));
const ran = spawnSync(process.execPath, ["--experimental-strip-types", join(lab, "src", "cli.ts"), "suite", "--no-build", file], {
cwd: lab, encoding: "utf8", env: suiteEnv(dir),
});
assert.equal(ran.status, 1, `the suite exited ${ran.status}:\n${ran.stdout}${ran.stderr}`);
assert.match(ran.stdout, /0 passed, 0 failed, 1 skipped/);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test("the suite still exits 0 for a file that passes a test and skips another for a declared reason", () => {
const dir = mkdtempSync(join(tmpdir(), "mesh-lab-693-"));
try {
const file = join(dir, "declared.test.ts");
writeFileSync(file, [
`import { test } from "node:test";`,
`test("a module lands", () => {});`,
`test("a builder builds", { skip: "set MESH_LAB_BUILDER to a built mesh-builder" }, () => {});`,
].join("\n"));
const ran = spawnSync(process.execPath, ["--experimental-strip-types", join(lab, "src", "cli.ts"), "suite", "--no-build", file], {
cwd: lab, encoding: "utf8", env: suiteEnv(dir),
});
assert.equal(ran.status, 0, `the suite exited ${ran.status}:\n${ran.stdout}${ran.stderr}`);
assert.match(ran.stdout, /1 passed, 0 failed, 1 skipped/);
assert.doesNotMatch(ran.stdout, /NOT PASSED/);
assert.match(ran.stdout, /^skipped: a builder builds \(.*declared\.test\.ts\) # set MESH_LAB_BUILDER to a built mesh-builder$/m);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
+7 -6
View File
@@ -5,7 +5,7 @@ import { counted, reportOn } from "../src/suite.ts";
// The totals come from the runner's own summary, and from nothing else.
test("the runner's totals are read from its summary", () => {
const said = counted("✔ something (1ms)\nℹ tests 22\nℹ pass 22\nℹ fail 0\n");
assert.deepEqual(said, { passed: 22, failed: 0 });
assert.deepEqual(said, { passed: 22, failed: 0, skipped: null });
});
// A test *named* like a total must not be mistaken for one. The summary is a line of its own, and
@@ -18,7 +18,7 @@ test("a test named like a total is not a total", () => {
// A failing run is read as a failing run.
test("failures are read", () => {
const said = counted("ℹ pass 21\nℹ fail 1\n");
assert.deepEqual(said, { passed: 21, failed: 1 });
assert.deepEqual(said, { passed: 21, failed: 1, skipped: null });
});
// **No totals is not zero failures.** A run whose result could not be read is a run nobody can say
@@ -36,7 +36,7 @@ test("output with no summary yields no totals rather than a clean bill", () => {
// check it — 04-ISSUES/005 in miniature, inside the fix for it.
test("a run whose result could not be read writes nothing", () => {
let wrote = false;
const said = reportOn({ passed: null, failed: null }, () => {
const said = reportOn({ passed: null, failed: null }, [], () => {
wrote = true;
return { against: {}, ran: [] };
});
@@ -46,12 +46,13 @@ test("a run whose result could not be read writes nothing", () => {
test("a run that was read is recorded, with what it was read against", () => {
let got: [number, number] | null = null;
const said = reportOn({ passed: 22, failed: 0 }, (p, f) => {
const said = reportOn({ passed: 22, failed: 0 }, [], (p, f) => {
got = [p, f];
return { against: { "mesh-lab": "abc1234" }, ran: ["test/integration/mesh.test.ts"] };
});
assert.deepEqual(got, [22, 0]);
assert.match(said, /mesh\.test\.ts — 22 passed, 0 failed, against mesh-lab abc1234/);
assert.match(said, /mesh\.test\.ts — 22 passed, 0 failed, 0 skipped, against mesh-lab abc1234/);
assert.doesNotMatch(said, /NOT PASSED/);
});
// The runner's real output, colours and all.
@@ -64,5 +65,5 @@ test("the runner's totals are read from output as it actually arrives", () => {
"\u001b[34m\u2139 pass 22\u001b[39m\n" +
"\u001b[34m\u2139 fail 0\u001b[39m\n" +
"\u001b[34m\u2139 duration_ms 98.9\u001b[39m\n";
assert.deepEqual(counted(real), { passed: 22, failed: 0 });
assert.deepEqual(counted(real), { passed: 22, failed: 0, skipped: null });
});