A suite run in which no test passed exits non-zero and names its skips (issue 693) #105

Merged

mesh-admin merged 2 commits from fix/693-a-run-that-proved-nothing-is-not-passed into main 2026-10-14 05:05:36 +00:00

Conversation 1 Commits 2 Files Changed 5

+389 -25

mesh-admin commented 2026-10-14 01:14:20 +00:00

Contributor

Copy Link

Copy Source

Fixes issue 693 (a lab run whose only test skipped is recorded as passed).

What

• src/outcomes.ts: a reporter for node's test runner that writes each skipped test with its reason, word for word, and judgeRun, the one place the rule lives.
• src/suite.ts: the suite runs the spec reporter to stdout as before plus the skips reporter into a file of its own; reads the runner's skipped total; after the recorded: line it prints NOT PASSED: <why> and one skipped: <test> (<file>) # <reason> line per skip; its exit is 1 when the runner said 0 for a run that did not pass.
• src/lastrun.ts: the receipt keeps skipped, skips and notPassed; last-run reads a run that proved nothing as not current, with why (an older receipt with 0 passed too).

Why

The runner exits 0 when every test skipped. run-2026-10-14T00-09-38-410Z and run-2026-10-14T00-21-44-014Z on shanks skipped the genesis bed's only test as "ADR 0333 §5 NOT REACHED", printed "0 passed, 0 failed" and were recorded as passed.

The design limit (chosen here)

• Not passed when no test passed (all skipped, or none ran).
• Not passed when a test skipped as NOT REACHED, even beside passes: the beds' own words for "the subject was not reached".
• Passed when tests passed and others skipped for another, declared reason (no hypervisor, set MESH_LAB_BUILDER ...); each such skip is still listed with its reason in the log and the receipt.
• Totals the runner did not print decide nothing new (unchanged: no receipt, the runner's exit).

Tested

• test/outcomes.test.ts: the rule, the recorded lines, the receipt and last-run, and two real-runner runs of cli.ts suite --no-build: a file whose only test skips as NOT REACHED exits 1 with NOT PASSED and its reason (on main it exits 0, "0 passed, 0 failed"); a file that passes one and skips one for a declared reason exits 0 and lists the skip.
• sh merge-check.sh on g14: typecheck, 179 unit tests pass, replays/asks Go parts pass.

The lab module's side (lab_status) is mesh-catalog's pull request for the same issue; each stands alone.

Fixes issue 693 (a lab run whose only test skipped is recorded as passed). ## What - `src/outcomes.ts`: a reporter for node's test runner that writes each skipped test with its reason, word for word, and `judgeRun`, the one place the rule lives. - `src/suite.ts`: the suite runs the spec reporter to stdout as before plus the skips reporter into a file of its own; reads the runner's `skipped` total; after the `recorded:` line it prints `NOT PASSED: <why>` and one `skipped: <test> (<file>) # <reason>` line per skip; its exit is 1 when the runner said 0 for a run that did not pass. - `src/lastrun.ts`: the receipt keeps `skipped`, `skips` and `notPassed`; `last-run` reads a run that proved nothing as not current, with why (an older receipt with 0 passed too). ## Why The runner exits 0 when every test skipped. run-2026-10-14T00-09-38-410Z and run-2026-10-14T00-21-44-014Z on shanks skipped the genesis bed's only test as "ADR 0333 §5 NOT REACHED", printed "0 passed, 0 failed" and were recorded as passed. ## The design limit (chosen here) - Not passed when **no test passed** (all skipped, or none ran). - Not passed when **a test skipped as NOT REACHED**, even beside passes: the beds' own words for "the subject was not reached". - **Passed** when tests passed and others skipped for another, declared reason (no hypervisor, `set MESH_LAB_BUILDER ...`); each such skip is still listed with its reason in the log and the receipt. - Totals the runner did not print decide nothing new (unchanged: no receipt, the runner's exit). ## Tested - `test/outcomes.test.ts`: the rule, the recorded lines, the receipt and `last-run`, and two real-runner runs of `cli.ts suite --no-build`: a file whose only test skips as NOT REACHED exits 1 with NOT PASSED and its reason (on main it exits 0, "0 passed, 0 failed"); a file that passes one and skips one for a declared reason exits 0 and lists the skip. - `sh merge-check.sh` on g14: typecheck, 179 unit tests pass, replays/asks Go parts pass. The lab module's side (lab_status) is mesh-catalog's pull request for the same issue; each stands alone.

mesh-admin added 1 commit 2026-10-14 01:14:20 +00:00

A suite run in which no test passed exits non-zero and names its skips (issue 693) ... 8635560993

The runner exits 0 when every test skipped, so a bed that stopped before its
subject and skipped its only test as NOT REACHED was recorded as passed, "0
passed, 0 failed". The suite now collects each skip and its reason through a
reporter of its own, prints NOT PASSED and a skipped line per skip after its
recorded line, keeps them in the receipt, and exits 1 when no test passed or a
test was not reached. A skip for a declared reason beside passes stays passed.

mesh-admin commented 2026-10-14 01:14:31 +00:00

Author

Contributor

Copy Link

Copy Source

Delivery novox/mesh-lab@5ba88c69fa80 — delivered since 2026-10-14T05:06:01Z

Delivery plan — builds nothing: the change touches no module of the mesh's graph

Group fix/693-a-run-that-proved-nothing-is-not-passed, in order: novox/mesh-catalog@acf8a76646 → novox/mesh-lab@5ba88c69fa

• composed together: pass — every machine composes with the change as it did without (4 of 4 compose)

Transitions

• 2026-10-14 01:28 (new) → proposed (announced): novox/mesh-lab#105's head announced
• 2026-10-14 01:32 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
• 2026-10-14 01:32 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
• 2026-10-14 05:06 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
• 2026-10-14 05:06 published → delivered (done): nothing for a walk to move

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

<!-- mesh-delivery:view --> **Delivery** `novox/mesh-lab@5ba88c69fa80` — **delivered** since 2026-10-14T05:06:01Z **Delivery plan** — builds nothing: the change touches no module of the mesh's graph **Group** `fix/693-a-run-that-proved-nothing-is-not-passed`, in order: novox/mesh-catalog@acf8a76646de → novox/mesh-lab@5ba88c69fa80 - composed together: pass — every machine composes with the change as it did without (4 of 4 compose) **Transitions** - 2026-10-14 01:28 (new) → proposed (announced): novox/mesh-lab#105's head announced - 2026-10-14 01:32 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-14 01:32 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-14 05:06 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-14 05:06 published → delivered (done): nothing for a walk to move The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin reviewed 2026-10-14 01:26:42 +00:00

mesh-admin left a comment

Author

Contributor

Copy Link

Copy Source

Verdict: sound, mergeable at 8635560993aa; one robustness follow-up (non-blocking, not a regression).

Independent review for issue 693 (a lab run whose only test skipped is recorded as passed), delivery group fix/693-a-run-that-proved-nothing-is-not-passed.

Checks at this head: mesh/merge-gate success ("PASS — the change touches no module of the mesh's graph"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh (go)"), mesh/delivery success ("ready: waits for its merge"), mesh/delivery-group success ("ready: every member ready, and composed together they pass; composed check pass — every machine composes with the change as it did without (4 of 4 compose)").

1. The rule. judgeRun is sound and kept in one place: a failure stays a failure (exitOf never lowers a non-zero runner exit), no pass means not passed, a NOT REACHED skip means not passed even beside passes, and a declared skip beside passes stays passed. The match is case-sensitive "NOT REACHED"; a bed that writes "not reached" would read as a declared skip — worth a line in the beds' convention. A describe.skip suite reports none of its tests, so it reads "no test passed: none ran" with "0 skipped": right verdict, slightly wrong wording.

2. Hostile log text. Skips come from the runner's events through the reporter, not from text — good. But counted() still takes the FIRST summary-shaped line in the output, and a test's stdout reaches that output. I ran a file whose only test prints "ℹ pass 5" on a line of its own and then skips with "no hypervisor": the suite recorded "5 passed, 0 failed, 1 skipped", printed no NOT PASSED and exited 0. So a run that proved nothing reads as passed, in the exit and in last-run.json. The same weakness exists on main, and the lab module (mesh-catalog #287) reads the LAST summary and catches this case, so this does not block. Follow-up: take the last match in counted(), as the module does.

3. Older receipts: judge() treats a receipt without notPassed and with passed 0 as not current; tested.

4. Fails on main: with main's src and the PR's fixture (one test, t.skip("... NOT REACHED ...")), cli.ts suite --no-build exits 0 and prints "0 passed, 0 failed"; at this head it exits 1 with NOT PASSED and the reason. test/outcomes.test.ts on main fails at import. At this head, niced: typecheck clean, npm test 179 pass, 0 fail, 2 skipped.

Merge order: this one first, then mesh-catalog #287 (each stands alone; the module reads both old and new logs).

Reviewed by an independent reviewer on g14; no lab bed was run.

Verdict: sound, mergeable at 8635560993aa; one robustness follow-up (non-blocking, not a regression). Independent review for issue 693 (a lab run whose only test skipped is recorded as passed), delivery group fix/693-a-run-that-proved-nothing-is-not-passed. Checks at this head: mesh/merge-gate success ("PASS — the change touches no module of the mesh's graph"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh (go)"), mesh/delivery success ("ready: waits for its merge"), mesh/delivery-group success ("ready: every member ready, and composed together they pass; composed check pass — every machine composes with the change as it did without (4 of 4 compose)"). 1. The rule. judgeRun is sound and kept in one place: a failure stays a failure (exitOf never lowers a non-zero runner exit), no pass means not passed, a NOT REACHED skip means not passed even beside passes, and a declared skip beside passes stays passed. The match is case-sensitive "NOT REACHED"; a bed that writes "not reached" would read as a declared skip — worth a line in the beds' convention. A describe.skip suite reports none of its tests, so it reads "no test passed: none ran" with "0 skipped": right verdict, slightly wrong wording. 2. Hostile log text. Skips come from the runner's events through the reporter, not from text — good. But counted() still takes the FIRST summary-shaped line in the output, and a test's stdout reaches that output. I ran a file whose only test prints "ℹ pass 5" on a line of its own and then skips with "no hypervisor": the suite recorded "5 passed, 0 failed, 1 skipped", printed no NOT PASSED and exited 0. So a run that proved nothing reads as passed, in the exit and in last-run.json. The same weakness exists on main, and the lab module (mesh-catalog #287) reads the LAST summary and catches this case, so this does not block. Follow-up: take the last match in counted(), as the module does. 3. Older receipts: judge() treats a receipt without notPassed and with passed 0 as not current; tested. 4. Fails on main: with main's src and the PR's fixture (one test, t.skip("... NOT REACHED ...")), cli.ts suite --no-build exits 0 and prints "0 passed, 0 failed"; at this head it exits 1 with NOT PASSED and the reason. test/outcomes.test.ts on main fails at import. At this head, niced: typecheck clean, npm test 179 pass, 0 fail, 2 skipped. Merge order: this one first, then mesh-catalog #287 (each stands alone; the module reads both old and new logs). Reviewed by an independent reviewer on g14; no lab bed was run.

jschoubben added 1 commit 2026-10-14 01:28:18 +00:00

Read the runner's last summary and NOT REACHED in any case (issue 693, review 253) ... 5ba88c69fa

A test that printed a summary-shaped line before it skipped was read as the
runner's totals: "ℹ pass 5" made a run that proved nothing pass.

mesh-admin reviewed 2026-10-14 05:03:10 +00:00

mesh-admin left a comment

Author

Contributor

Copy Link

Copy Source

Verdict: sound, mergeable at 5ba88c69fa80; the follow-ups of review 253 are answered.

Re-review for issue 693 (a lab run whose only test skipped is recorded as passed).

Checks at this head: mesh/merge-gate success ("PASS — the change touches no module of the mesh's graph"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh (go)"), mesh/delivery success ("ready: waits for its merge"), mesh/delivery-group success ("ready: every member ready, and composed together they pass; composed check pass — every machine composes with the change as it did without (4 of 4 compose)").

1. counted() now reads the last summary line. I re-ran my probe, a file whose only test prints "ℹ pass 5" and then skips with "no hypervisor": the suite now records "0 passed, 0 failed, 1 skipped", prints "NOT PASSED: no test passed: 1 skipped" and exits 1 (at 8635560 it exited 0).
2. NOT REACHED is matched in any case: a file with one pass and a skip "bus not reached: stopped" exits 1 with "NOT PASSED: 1 test(s) skipped as NOT REACHED".
3. The new tests fail against the old source: with 8635560's src, "the runner's last summary is read, not a line a test printed before it", "the suite does not take a summary a test printed for the runner's" and "NOT REACHED is matched in any case" fail (3 fail).

At this head, niced: typecheck clean; npm test 182 pass, 0 fail, 2 skipped.

Merge order: this one first, then mesh-catalog #287.

Reviewed by an independent reviewer on g14; no lab bed ran.

Verdict: sound, mergeable at 5ba88c69fa80; the follow-ups of review 253 are answered. Re-review for issue 693 (a lab run whose only test skipped is recorded as passed). Checks at this head: mesh/merge-gate success ("PASS — the change touches no module of the mesh's graph"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (typescript), replays/merge-check.sh (go), asks/merge-check.sh (go)"), mesh/delivery success ("ready: waits for its merge"), mesh/delivery-group success ("ready: every member ready, and composed together they pass; composed check pass — every machine composes with the change as it did without (4 of 4 compose)"). 1. counted() now reads the last summary line. I re-ran my probe, a file whose only test prints "ℹ pass 5" and then skips with "no hypervisor": the suite now records "0 passed, 0 failed, 1 skipped", prints "NOT PASSED: no test passed: 1 skipped" and exits 1 (at 8635560 it exited 0). 2. NOT REACHED is matched in any case: a file with one pass and a skip "bus not reached: stopped" exits 1 with "NOT PASSED: 1 test(s) skipped as NOT REACHED". 3. The new tests fail against the old source: with 8635560's src, "the runner's last summary is read, not a line a test printed before it", "the suite does not take a summary a test printed for the runner's" and "NOT REACHED is matched in any case" fail (3 fail). At this head, niced: typecheck clean; npm test 182 pass, 0 fail, 2 skipped. Merge order: this one first, then mesh-catalog #287. Reviewed by an independent reviewer on g14; no lab bed ran.

mesh-admin merged commit c0b1225f6d into main 2026-10-14 05:05:36 +00:00

mesh-admin deleted branch fix/693-a-run-that-proved-nothing-is-not-passed 2026-10-14 05:05:36 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

mesh-admin

Labels

No items

No labels

Milestone

No items

No Milestone

Assignees

Clear assignees

jschoubben miel.truyen

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-lab#105