lab: a run whose suite proved nothing reads as failed, with its skips (issue 693) #287

2026-10-14T01:14:28Z

mesh-admin commented

2026-10-14 01:14:28 +00:00

Fixes issue 693 (a lab run whose only test skipped is recorded as passed).

What

modules/lab/tools/runs.ts:

outcomeOf(log) reads what a run's log says of its tests: the runner's summary totals (pass, fail, skipped), each skip with its reason, and why the run did not pass. It reads the lab suite's own NOT PASSED: and skipped: <test> (<file>) # <reason> lines (mesh-lab pull request for the same issue) and, for a lab of before, the runner's spec skip lines (first line of the reason).
settled(): readStatus (so lab_status) gives every ended run an outcome, kept in its status.json; a run whose exit said 0 while it did not pass reads as failed, its exit left as it was.

Why

lab_status took the suite's exit as the verdict, and the runner exits 0 when every test skipped. run-2026-10-14T00-09-38-410Z and run-2026-10-14T00-21-44-014Z on shanks skipped the genesis bed's only test as NOT REACHED and read passed, exit 0.

The design limit

The same as the lab suite's: not passed when no test passed, or when a test skipped as NOT REACHED; passed with its skips listed when tests passed beside skips for a declared reason. Read from the log, so it also holds for a run of an older mesh-lab branch, and the two runs above read as failed once this is delivered to shanks.

Tested

modules/lab/test/runs.test.ts: the real log tail of run-2026-10-14T00-21-44-014Z (state passed, exit 0) reads as failed with no test passed: 1 skipped, 1 of them as NOT REACHED and the reason, kept in status.json; the new suite lines are read, a test name with an em dash and all; a run with a pass and a declared skip stays passed; a run with no test reads not passed. On main the file fails (no outcomeOf/settled).
node --test on the module's tests (28 pass), tsc strict on runs.ts and the test, MESH_CHECK_MODULES=lab sh merge-check.sh (Go part; the TypeScript part runs in its toolchain). No manifest change.

Fixes issue 693 (a lab run whose only test skipped is recorded as passed). ## What `modules/lab/tools/runs.ts`: - `outcomeOf(log)` reads what a run's log says of its tests: the runner's summary totals (pass, fail, skipped), each skip with its reason, and why the run did not pass. It reads the lab suite's own `NOT PASSED:` and `skipped: <test> (<file>) # <reason>` lines (mesh-lab pull request for the same issue) and, for a lab of before, the runner's spec skip lines (first line of the reason). - `settled()`: `readStatus` (so `lab_status`) gives every ended run an `outcome`, kept in its `status.json`; a run whose exit said 0 while it did not pass reads as `failed`, its `exit` left as it was. ## Why lab_status took the suite's exit as the verdict, and the runner exits 0 when every test skipped. run-2026-10-14T00-09-38-410Z and run-2026-10-14T00-21-44-014Z on shanks skipped the genesis bed's only test as NOT REACHED and read `passed`, exit 0. ## The design limit The same as the lab suite's: not passed when no test passed, or when a test skipped as NOT REACHED; passed with its skips listed when tests passed beside skips for a declared reason. Read from the log, so it also holds for a run of an older mesh-lab branch, and the two runs above read as failed once this is delivered to shanks. ## Tested - `modules/lab/test/runs.test.ts`: the real log tail of run-2026-10-14T00-21-44-014Z (state passed, exit 0) reads as failed with `no test passed: 1 skipped, 1 of them as NOT REACHED` and the reason, kept in status.json; the new suite lines are read, a test name with an em dash and all; a run with a pass and a declared skip stays passed; a run with no test reads not passed. On main the file fails (no `outcomeOf`/`settled`). - `node --test` on the module's tests (28 pass), tsc strict on runs.ts and the test, `MESH_CHECK_MODULES=lab sh merge-check.sh` (Go part; the TypeScript part runs in its toolchain). No manifest change.

mesh-admin added 1 commit 2026-10-14 01:14:29 +00:00

lab: a run whose suite proved nothing reads as failed, with its skips (issue 693)

mesh/merge-gate head c8255eb15d12 merged into trunk ebf7f3773dc7 — pass: builds lab → shanks; no bus step; every machine composes with the change as it did…

Details

mesh/repo-check head c8255eb15d12 merged into trunk ebf7f3773dc7 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…

Details

mesh/delivery-group group fix/693-a-run-that-proved-nothing-is-not-passed checking: 1 of 2 member(s) ready

Details

mesh/delivery superseded: a newer head of the same pull request

Details

c8255eb15d

lab_status took the suite's exit as the verdict, and the runner exits 0 when
every test skipped: a genesis bed that skipped its only test as NOT REACHED was
recorded as passed. A run that ended now has its outcome read from its log and
kept in its status - totals, each skip with its reason, and why it did not pass
- and one whose exit said 0 while no test passed reads as failed. Read from the
lab's own lines when it prints them, from the runner's summary for a lab of
before.

mesh-admin commented

2026-10-14 01:15:01 +00:00

Delivery novox/mesh-catalog@acf8a76646de — delivered since 2026-10-14T05:09:27Z

Delivery plan — builds lab → shanks; no bus step

tier 0: lab
shanks: receives lab
lab keeps data (ADR 0233): what it holds is backed up before it moves
grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine

Group fix/693-a-run-that-proved-nothing-is-not-passed, in order: novox/mesh-catalog@acf8a76646 → novox/mesh-lab@5ba88c69fa

composed together: pass — every machine composes with the change as it did without (4 of 4 compose)

Machines

lab on shanks: passed — healthy 1 times over 55s, by a leaf module's rule: one reading (novox/hq ADR 0304)
lab on the rest running it: sent

Transitions

2026-10-14 01:28 (new) → proposed (announced): novox/mesh-catalog#287's head announced
2026-10-14 01:30 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-14 01:30 checked → rejected (refused): the gate failed or could not run, or the repository's own check failed
2026-10-14 01:33 rejected → proposed (recheck): issue 693: the check could not run (facts snapshot unreadable, artifact store unreachable); not a verdict on the head
2026-10-14 01:35 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-14 01:35 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-14 05:07 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
2026-10-14 05:07 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit
2026-10-14 05:09 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@acf8a76646de` — **delivered** since 2026-10-14T05:09:27Z **Delivery plan** — builds lab → shanks; no bus step - tier 0: lab - shanks: receives lab - lab keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine **Group** `fix/693-a-run-that-proved-nothing-is-not-passed`, in order: novox/mesh-catalog@acf8a76646de → novox/mesh-lab@5ba88c69fa80 - composed together: pass — every machine composes with the change as it did without (4 of 4 compose) **Machines** - lab on shanks: passed — healthy 1 times over 55s, by a leaf module's rule: one reading (novox/hq ADR 0304) - lab on the rest running it: sent **Transitions** - 2026-10-14 01:28 (new) → proposed (announced): novox/mesh-catalog#287's head announced - 2026-10-14 01:30 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-14 01:30 checked → rejected (refused): the gate failed or could not run, or the repository's own check failed - 2026-10-14 01:33 rejected → proposed (recheck): issue 693: the check could not run (facts snapshot unreadable, artifact store unreachable); not a verdict on the head - 2026-10-14 01:35 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-14 01:35 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-14 05:07 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-14 05:07 published → delivering (go): its walk started: let go by this owner in its group's order, by a person, or on the controller's own path — not a failed walk that carried a later commit for it, after which it waits for a walk of its own commit - 2026-10-14 05:09 delivering → delivered (done): its walk is done, carrying its own commit: every machine of its deploy plan passed, or was left as its policy says The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin commented

2026-10-14 01:19:25 +00:00

Merge check at c8255eb1
Judged: head c8255eb15d12 merged into trunk ebf7f3773dc7; read beside it: mesh-controller df9e5728a925, mesh-controller-main df9e5728a925, mesh-host 1a2677882380, mesh-tools 954e1a53eef9

mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)
mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds lab → shanks; no bus step

tier 0: lab
shanks: receives lab
lab keeps data (ADR 0233): what it holds is backed up before it moves
grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine

Run by the build seat on novox as build-1791940498081968541 (builds --log build-1791940498081968541).

--- the gate: lab
lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339)
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:08:55Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T01:08:55Z

modules the change moves:
  - lab (changed)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): lab
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.024s
words: 12 retired words and 70 daemons' lines, none in the 46813 strings of 718 files
ok  	git.novox.be/novox/mesh-catalog/checks/agenthome	0.015s
agent-home: no archive and no file written into under the agent's home
ok  	git.novox.be/novox/mesh-catalog/checks/secrets	0.014s
secrets: no setting can replace a vault secret in a mergeable file (1 known, nested)
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.008s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 66, as kept
ok  	git.novox.be/novox/mesh-catalog/checks/deps	0.881s
modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it
--- its merge-check-typescript.sh (typescript toolchain)
type-checking modules/lab (TypeScript)
testing modules/lab (TypeScript)
✔ a checkout with a lock file at its root is installed there (3.066347ms)
✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (1.246413ms)
✔ a package.json with no lock file is named, never installed (0.657082ms)
✔ a run installs and builds mesh-tools in node-tools, where its lock file is (32.203303ms)
✔ a run installs a checkout with its lock file at the root there, and builds only when asked (4.993391ms)
✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (13.491711ms)
✔ a run checks out mesh-tools-go and says where its checkouts are (0.851572ms)
✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.816202ms)
✔ a run clones through git's credential store, and its script carries no secret (0.967502ms)
✔ git presents the stored credential to the forge, and only when one is given (105.748522ms)
✔ a stopped script removes the credential, whatever ends it (37.167105ms)
✔ the script removes the credential as soon as its clones are done (0.402831ms)
✔ credentials a crashed run left are removed, but not the running one's (0.775402ms)
✔ the lab asks the package registry for read only (0.336571ms)
✔ each check gets a fresh directory of its own (0.396111ms)
﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.08254ms) # running as root, which no directory mode stops from removing anything
✔ leftover checks are cleared, whoever runs the check (0.787922ms)
✔ the lab's tools bundle names its bound credential and binding in its env (0.16201ms)
✔ a run names the package registry by the address the registry gives its tarballs (18.186122ms)
✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (5.166162ms)
✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.448211ms)
✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (1.289073ms)
✔ the public root is read from the binding, and only an https root at the npm path is taken (1.780644ms)
✔ a forge setting that is not http(s) trusts no origin of its own (0.590542ms)
✔ a run whose only test skipped as NOT REACHED, from a lab of before, reads as failed with the reason (10.818204ms)
✔ the lab's own NOT PASSED and skipped lines are read, a test name's punctuation and all (17.881751ms)
✔ a run that passed tests and skipped one for a declared reason stays passed, its skip listed (44.336592ms)
✔ a run that ran no test at all is not passed (0.238751ms)
ℹ tests 28
ℹ suites 0
ℹ pass 27
ℹ fail 0
ℹ cancelled 0
ℹ skipped 1
ℹ todo 0
ℹ duration_ms 426.497076

**Merge check** at `c8255eb1` Judged: head `c8255eb15d12` merged into trunk `ebf7f3773dc7`; read beside it: mesh-controller `df9e5728a925`, mesh-controller-main `df9e5728a925`, mesh-host `1a2677882380`, mesh-tools `954e1a53eef9` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: lab) - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds lab → shanks; no bus step - tier 0: lab - shanks: receives lab - lab keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine Run by the build seat on novox as `build-1791940498081968541` (`builds --log build-1791940498081968541`). ``` --- the gate: lab lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339) long-running resources without health: 0 files asking for a setting without saying whether it is trusted: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:08:55Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-14T01:08:55Z modules the change moves: - lab (changed) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): lab --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.024s words: 12 retired words and 70 daemons' lines, none in the 46813 strings of 718 files ok git.novox.be/novox/mesh-catalog/checks/agenthome 0.015s agent-home: no archive and no file written into under the agent's home ok git.novox.be/novox/mesh-catalog/checks/secrets 0.014s secrets: no setting can replace a vault secret in a mergeable file (1 known, nested) ok git.novox.be/novox/mesh-catalog/checks/paths 0.008s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 66, as kept ok git.novox.be/novox/mesh-catalog/checks/deps 0.881s modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it --- its merge-check-typescript.sh (typescript toolchain) type-checking modules/lab (TypeScript) testing modules/lab (TypeScript) ✔ a checkout with a lock file at its root is installed there (3.066347ms) ✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (1.246413ms) ✔ a package.json with no lock file is named, never installed (0.657082ms) ✔ a run installs and builds mesh-tools in node-tools, where its lock file is (32.203303ms) ✔ a run installs a checkout with its lock file at the root there, and builds only when asked (4.993391ms) ✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (13.491711ms) ✔ a run checks out mesh-tools-go and says where its checkouts are (0.851572ms) ✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.816202ms) ✔ a run clones through git's credential store, and its script carries no secret (0.967502ms) ✔ git presents the stored credential to the forge, and only when one is given (105.748522ms) ✔ a stopped script removes the credential, whatever ends it (37.167105ms) ✔ the script removes the credential as soon as its clones are done (0.402831ms) ✔ credentials a crashed run left are removed, but not the running one's (0.775402ms) ✔ the lab asks the package registry for read only (0.336571ms) ✔ each check gets a fresh directory of its own (0.396111ms) ﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.08254ms) # running as root, which no directory mode stops from removing anything ✔ leftover checks are cleared, whoever runs the check (0.787922ms) ✔ the lab's tools bundle names its bound credential and binding in its env (0.16201ms) ✔ a run names the package registry by the address the registry gives its tarballs (18.186122ms) ✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (5.166162ms) ✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.448211ms) ✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (1.289073ms) ✔ the public root is read from the binding, and only an https root at the npm path is taken (1.780644ms) ✔ a forge setting that is not http(s) trusts no origin of its own (0.590542ms) ✔ a run whose only test skipped as NOT REACHED, from a lab of before, reads as failed with the reason (10.818204ms) ✔ the lab's own NOT PASSED and skipped lines are read, a test name's punctuation and all (17.881751ms) ✔ a run that passed tests and skipped one for a declared reason stays passed, its skip listed (44.336592ms) ✔ a run that ran no test at all is not passed (0.238751ms) ℹ tests 28 ℹ suites 0 ℹ pass 27 ℹ fail 0 ℹ cancelled 0 ℹ skipped 1 ℹ todo 0 ℹ duration_ms 426.497076 ```

mesh-admin reviewed

2026-10-14 01:26:52 +00:00

mesh-admin left a comment

Verdict: sound, mergeable at c8255eb15d12; one robustness follow-up (non-blocking; it can only make a run read worse, never better).

Independent review for issue 693 (a lab run whose only test skipped is recorded as passed), delivery group fix/693-a-run-that-proved-nothing-is-not-passed.

Checks at this head: mesh/merge-gate success ("PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)"), mesh/delivery success ("ready: waits for its merge"), mesh/delivery-group success ("ready: every member ready, and composed together they pass; composed check pass — every machine composes with the change as it did without (4 of 4 compose)").

The rule matches mesh-lab's judgeRun. settled() only ever moves passed to failed, never failed to passed, and leaves running, stopped and lost runs alone, so no real failure can read as a pass.
Hostile log text. outcomeOf reads the LAST runner summary, which defeats a fake "ℹ pass N" printed by a test (I checked: a fake "ℹ pass 7" before the real "ℹ pass 0" still reads not passed). But the "NOT PASSED:" and "skipped: (.ts) # " patterns match anywhere in run.log, first match wins, and a test's stdout reaches run.log. I ran outcomeOf on a genuinely passing log (pass 1, fail 0) with a test-printed "NOT PASSED: printed by a test" line: it returned notPassed "printed by a test", so a real pass would be rewritten to failed with a forged reason, and a printed fake "skipped: x (y.ts) # NOT REACHED" line is taken as a skip too. That is loud rather than silent, so it does not block. Follow-up: read the suite's lines only after the last "recorded:" line, which the suite prints after the runner has exited.
Older runs: an old log without the suite's lines falls back to the runner's spec "﹣ () # " lines and the last totals; the shanks run-2026-10-14T00-21-44-014Z tail is a test and reads failed with the reason. A run with no summary in its log (failed at setup, or cut short) gets no notPassed and keeps its state. The rewrite of status.json happens once per run (outcome then set). Every old run that passed with 0 tests passed will now read failed; that is the intended rule.
Fails on main: test/runs.test.ts against main's tools/runs.ts fails at import ("does not provide an export named 'outcomeOf'"). At this head, niced: 28 pass, 0 fail.

Merge order: mesh-lab #105 first, then this one (each stands alone).

Reviewed by an independent reviewer on g14; no lab bed was run.

Verdict: sound, mergeable at c8255eb15d12; one robustness follow-up (non-blocking; it can only make a run read worse, never better). Independent review for issue 693 (a lab run whose only test skipped is recorded as passed), delivery group fix/693-a-run-that-proved-nothing-is-not-passed. Checks at this head: mesh/merge-gate success ("PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)"), mesh/delivery success ("ready: waits for its merge"), mesh/delivery-group success ("ready: every member ready, and composed together they pass; composed check pass — every machine composes with the change as it did without (4 of 4 compose)"). 1. The rule matches mesh-lab's judgeRun. settled() only ever moves passed to failed, never failed to passed, and leaves running, stopped and lost runs alone, so no real failure can read as a pass. 2. Hostile log text. outcomeOf reads the LAST runner summary, which defeats a fake "ℹ pass N" printed by a test (I checked: a fake "ℹ pass 7" before the real "ℹ pass 0" still reads not passed). But the "NOT PASSED:" and "skipped: <test> (<file>.ts) # <reason>" patterns match anywhere in run.log, first match wins, and a test's stdout reaches run.log. I ran outcomeOf on a genuinely passing log (pass 1, fail 0) with a test-printed "NOT PASSED: printed by a test" line: it returned notPassed "printed by a test", so a real pass would be rewritten to failed with a forged reason, and a printed fake "skipped: x (y.ts) # NOT REACHED" line is taken as a skip too. That is loud rather than silent, so it does not block. Follow-up: read the suite's lines only after the last "recorded:" line, which the suite prints after the runner has exited. 3. Older runs: an old log without the suite's lines falls back to the runner's spec "﹣ <test> (<ms>) # <reason>" lines and the last totals; the shanks run-2026-10-14T00-21-44-014Z tail is a test and reads failed with the reason. A run with no summary in its log (failed at setup, or cut short) gets no notPassed and keeps its state. The rewrite of status.json happens once per run (outcome then set). Every old run that passed with 0 tests passed will now read failed; that is the intended rule. 4. Fails on main: test/runs.test.ts against main's tools/runs.ts fails at import ("does not provide an export named 'outcomeOf'"). At this head, niced: 28 pass, 0 fail. Merge order: mesh-lab #105 first, then this one (each stands alone). Reviewed by an independent reviewer on g14; no lab bed was run.

jschoubben added 1 commit 2026-10-14 01:28:24 +00:00

lab: read the suite's verdict only after its recorded line, NOT REACHED in any case (issue 693, review 254)

mesh/merge-gate head acf8a76646de merged into trunk ebf7f3773dc7 — pass: builds lab → shanks; no bus step; every machine composes with the change as it did…

Details

mesh/repo-check head acf8a76646de merged into trunk ebf7f3773dc7 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…

Details

mesh/delivery delivered

Details

mesh/delivery-group group fix/693-a-run-that-proved-nothing-is-not-passed delivered: every member is delivered

Details

acf8a76646

A test's own output, printed before the suite reports, could carry a line
shaped like the suite's NOT PASSED or skipped line and stand for its verdict.

mesh-admin commented

2026-10-14 01:30:44 +00:00

Merge check at acf8a766

mesh/merge-gate: ERROR — the check could not run: the facts snapshot cannot be read, and a check without it judges nothing: cannot reach the artifact store at novox.internal:5100: Get "http://novox.internal:5100/v2/facts/manifests/latest": dial tcp 10.10.0.1:5100: connect: connection refused (modules: lab)
mesh/repo-check: ERROR — the check could not run: the facts snapshot cannot be read, and a check without it judges nothing: cannot reach the artifact store at novox.internal:5100: Get "http://novox.internal:5100/v2/facts/manifests/latest": dial tcp 10.10.0.1:5100: connect: connection refused

Delivery plan — builds lab → shanks; no bus step

tier 0: lab
shanks: receives lab
lab keeps data (ADR 0233): what it holds is backed up before it moves
grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine

Run by the build seat on shanks as build-1791941339811628263 (builds --log build-1791941339811628263).

**Merge check** at `acf8a766` - `mesh/merge-gate`: **ERROR** — the check could not run: the facts snapshot cannot be read, and a check without it judges nothing: cannot reach the artifact store at novox.internal:5100: Get "http://novox.internal:5100/v2/facts/manifests/latest": dial tcp 10.10.0.1:5100: connect: connection refused (modules: lab) - `mesh/repo-check`: **ERROR** — the check could not run: the facts snapshot cannot be read, and a check without it judges nothing: cannot reach the artifact store at novox.internal:5100: Get "http://novox.internal:5100/v2/facts/manifests/latest": dial tcp 10.10.0.1:5100: connect: connection refused **Delivery plan** — builds lab → shanks; no bus step - tier 0: lab - shanks: receives lab - lab keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine Run by the build seat on shanks as `build-1791941339811628263` (`builds --log build-1791941339811628263`).

mesh-admin referenced this pull request

2026-10-14 01:32:18 +00:00

lab: name the base image's age, refuse a stale one, and rebuild it (issue 698) #288

mesh-admin commented

2026-10-14 01:35:22 +00:00

Merge check at acf8a766
Judged: head acf8a76646de merged into trunk ebf7f3773dc7; read beside it: mesh-controller 2959c8143536, mesh-controller-main 2959c8143536, mesh-host 1a2677882380, mesh-tools 954e1a53eef9

mesh/merge-gate: PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)
mesh/repo-check: PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)

Delivery plan — builds lab → shanks; no bus step

tier 0: lab
shanks: receives lab
lab keeps data (ADR 0233): what it holds is backed up before it moves
grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine

Run by the build seat on ace as build-1791941610047531020 (builds --log build-1791941610047531020).

--- the gate: lab
lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339)
long-running resources without health: 0
files asking for a setting without saying whether it is trusted: 1
1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here
judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:31:20Z says they run
composing every machine as the mesh is
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
composing every machine with the change
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued
merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose)
judged by controller development build against the facts of 2026-10-14T01:31:20Z

modules the change moves:
  - lab (changed)

machines:
  - a machine (f35): composes
  - holds mesh-dns-resolver (idt): composes
  - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes
  - a machine (yvbpsm): composes

a merge would rebuild 1 module(s) in 1 tier(s): lab
--- its merge-check.sh (go toolchain)
ok  	git.novox.be/novox/mesh-catalog/checks/words	0.002s
words: 12 retired words and 70 daemons' lines, none in the 46814 strings of 718 files
ok  	git.novox.be/novox/mesh-catalog/checks/agenthome	0.012s
agent-home: no archive and no file written into under the agent's home
ok  	git.novox.be/novox/mesh-catalog/checks/secrets	0.012s
secrets: no setting can replace a vault secret in a mergeable file (1 known, nested)
ok  	git.novox.be/novox/mesh-catalog/checks/paths	0.006s
paths: no module names another's state on its machine (3 known, listed to go)
long-running resources without health: 66, as kept
ok  	git.novox.be/novox/mesh-catalog/checks/deps	0.196s
modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it
--- its merge-check-typescript.sh (typescript toolchain)
type-checking modules/lab (TypeScript)
testing modules/lab (TypeScript)
✔ a checkout with a lock file at its root is installed there (2.02676ms)
✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (0.701789ms)
✔ a package.json with no lock file is named, never installed (0.418467ms)
✔ a run installs and builds mesh-tools in node-tools, where its lock file is (10.123335ms)
✔ a run installs a checkout with its lock file at the root there, and builds only when asked (4.143829ms)
✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (10.099925ms)
✔ a run checks out mesh-tools-go and says where its checkouts are (0.603744ms)
✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.733007ms)
✔ a run clones through git's credential store, and its script carries no secret (1.349757ms)
✔ git presents the stored credential to the forge, and only when one is given (9.640059ms)
✔ a stopped script removes the credential, whatever ends it (10.4919ms)
✔ the script removes the credential as soon as its clones are done (0.302769ms)
✔ credentials a crashed run left are removed, but not the running one's (0.760699ms)
✔ the lab asks the package registry for read only (0.3192ms)
✔ each check gets a fresh directory of its own (0.37279ms)
﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.057478ms) # running as root, which no directory mode stops from removing anything
✔ leftover checks are cleared, whoever runs the check (0.717869ms)
✔ the lab's tools bundle names its bound credential and binding in its env (0.142378ms)
✔ a run names the package registry by the address the registry gives its tarballs (17.372273ms)
✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (4.280755ms)
✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.344297ms)
✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (0.970695ms)
✔ the public root is read from the binding, and only an https root at the npm path is taken (1.247365ms)
✔ a forge setting that is not http(s) trusts no origin of its own (0.368804ms)
✔ a run whose only test skipped as NOT REACHED, from a lab of before, reads as failed with the reason (1.234912ms)
✔ the lab's own NOT PASSED and skipped lines are read, a test name's punctuation and all (0.585691ms)
✔ a run that passed tests and skipped one for a declared reason stays passed, its skip listed (0.550043ms)
✔ a run that ran no test at all is not passed (0.094097ms)
✔ a line a test printed that looks like the suite's is not read as the suite's (0.106911ms)
✔ not reached is matched in any case (0.095619ms)
ℹ tests 30
ℹ suites 0
ℹ pass 29
ℹ fail 0
ℹ cancelled 0
ℹ skipped 1
ℹ todo 0
ℹ duration_ms 169.90205

**Merge check** at `acf8a766` Judged: head `acf8a76646de` merged into trunk `ebf7f3773dc7`; read beside it: mesh-controller `2959c8143536`, mesh-controller-main `2959c8143536`, mesh-host `1a2677882380`, mesh-tools `954e1a53eef9` - `mesh/merge-gate`: **PASS** — every machine composes with the change as it did without (4 of 4 compose) (modules: lab) - `mesh/repo-check`: **PASS** — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript) **Delivery plan** — builds lab → shanks; no bus step - tier 0: lab - shanks: receives lab - lab keeps data (ADR 0233): what it holds is backed up before it moves - grants step: if this changes what the bus's user list says (a new tool, subject or user), novox is sent that list alone, every build there kept, before lab is judged on its first machine Run by the build seat on ace as `build-1791941610047531020` (`builds --log build-1791941610047531020`). ``` --- the gate: lab lab: ok, keeps work (cache); WARNING: runtime-env ask(s) for a setting and do(es) not say whether it is trusted, so it counts as trusted: set at the terminal alone; say "trusted" false where nothing trusts it (novox/hq issue 339) long-running resources without health: 0 files asking for a setting without saying whether it is trusted: 1 1 manifest(s) checked. Judged against the seats this binary carries; a claim on one of the mesh's own seats is judged fully at registration, and a seat declared by a module not given here reads as unknown. Identities judged on a 6-character machine name, against the bounds of the providers given here judging 1 module(s) changed against 4 machine(s), as the snapshot of 2026-10-14T01:31:20Z says they run composing every machine as the mesh is the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued composing every machine with the change the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued the bus's user list leaves out 5 user(s) the mesh has minted no credential for: controller, node.f35, node.idt, node.ygcyr, node.yvbpsm. Each is a user that cannot connect until one is issued merge gate: PASS — every machine composes with the change as it did without (4 of 4 compose) judged by controller development build against the facts of 2026-10-14T01:31:20Z modules the change moves: - lab (changed) machines: - a machine (f35): composes - holds mesh-dns-resolver (idt): composes - holds git, holds mesh-artifact-store, holds mesh-broker, holds mesh-catalog, holds mesh-delivery, holds mesh-dns-resolver, holds mesh-store, holds mesh-vault, holds npm-package-registry, the control node, the hub (ygcyr): composes - a machine (yvbpsm): composes a merge would rebuild 1 module(s) in 1 tier(s): lab --- its merge-check.sh (go toolchain) ok git.novox.be/novox/mesh-catalog/checks/words 0.002s words: 12 retired words and 70 daemons' lines, none in the 46814 strings of 718 files ok git.novox.be/novox/mesh-catalog/checks/agenthome 0.012s agent-home: no archive and no file written into under the agent's home ok git.novox.be/novox/mesh-catalog/checks/secrets 0.012s secrets: no setting can replace a vault secret in a mergeable file (1 known, nested) ok git.novox.be/novox/mesh-catalog/checks/paths 0.006s paths: no module names another's state on its machine (3 known, listed to go) long-running resources without health: 66, as kept ok git.novox.be/novox/mesh-catalog/checks/deps 0.196s modules/lab is TypeScript: merge-check-typescript.sh type-checks and tests it --- its merge-check-typescript.sh (typescript toolchain) type-checking modules/lab (TypeScript) testing modules/lab (TypeScript) ✔ a checkout with a lock file at its root is installed there (2.02676ms) ✔ a checkout whose package lives one directory down is installed there, as mesh-tools' node-tools (0.701789ms) ✔ a package.json with no lock file is named, never installed (0.418467ms) ✔ a run installs and builds mesh-tools in node-tools, where its lock file is (10.123335ms) ✔ a run installs a checkout with its lock file at the root there, and builds only when asked (4.143829ms) ✔ a run fails at its setup, naming the checkout, when nothing of it can be installed (10.099925ms) ✔ a run checks out mesh-tools-go and says where its checkouts are (0.603744ms) ✔ the forge credential is the bound account and its sealed secret, for the forge's own origin (0.733007ms) ✔ a run clones through git's credential store, and its script carries no secret (1.349757ms) ✔ git presents the stored credential to the forge, and only when one is given (9.640059ms) ✔ a stopped script removes the credential, whatever ends it (10.4919ms) ✔ the script removes the credential as soon as its clones are done (0.302769ms) ✔ credentials a crashed run left are removed, but not the running one's (0.760699ms) ✔ the lab asks the package registry for read only (0.3192ms) ✔ each check gets a fresh directory of its own (0.37279ms) ﹣ what earlier checks left is cleared, and what cannot be is named rather than tripped over (0.057478ms) # running as root, which no directory mode stops from removing anything ✔ leftover checks are cleared, whoever runs the check (0.717869ms) ✔ the lab's tools bundle names its bound credential and binding in its env (0.142378ms) ✔ a run names the package registry by the address the registry gives its tarballs (17.372273ms) ✔ a registry that does not give an address the lab takes leaves the forge's, and the reason is said (4.280755ms) ✔ a run's npm configuration names the registry it was given, and the forge's own address only without one (0.344297ms) ✔ a tarball is taken at the forge's public root, and at no other port or scheme of its host (0.970695ms) ✔ the public root is read from the binding, and only an https root at the npm path is taken (1.247365ms) ✔ a forge setting that is not http(s) trusts no origin of its own (0.368804ms) ✔ a run whose only test skipped as NOT REACHED, from a lab of before, reads as failed with the reason (1.234912ms) ✔ the lab's own NOT PASSED and skipped lines are read, a test name's punctuation and all (0.585691ms) ✔ a run that passed tests and skipped one for a declared reason stays passed, its skip listed (0.550043ms) ✔ a run that ran no test at all is not passed (0.094097ms) ✔ a line a test printed that looks like the suite's is not read as the suite's (0.106911ms) ✔ not reached is matched in any case (0.095619ms) ℹ tests 30 ℹ suites 0 ℹ pass 29 ℹ fail 0 ℹ cancelled 0 ℹ skipped 1 ℹ todo 0 ℹ duration_ms 169.90205 ```

mesh-admin referenced this pull request

2026-10-14 05:02:19 +00:00

lab: name the base image's age, refuse a stale one, and rebuild it (issue 698) #288

mesh-admin reviewed

2026-10-14 05:03:17 +00:00

mesh-admin left a comment

Verdict: sound, mergeable at acf8a76646de; the follow-up of review 254 is answered.

Re-review for issue 693 (a lab run whose only test skipped is recorded as passed).

Checks at this head: mesh/merge-gate success ("PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)"), mesh/delivery success ("ready: waits for its merge"), mesh/delivery-group success ("ready: every member ready, and composed together they pass; composed check pass — every machine composes with the change as it did without (4 of 4 compose)").

outcomeOf reads the suite's "NOT PASSED:" and "skipped:" lines only after the last "recorded:" line, which the suite prints after the runner has exited. I re-ran my probe, a genuinely passing log (pass 1, fail 0) with a test-printed "NOT PASSED: printed by a test", a fake "skipped: x (y.ts) # NOT REACHED" and even a fake "recorded:" line before the real one: it now returns no notPassed and no skips, so the run stays passed.
NOT REACHED is matched in any case.
The new tests ("a line a test printed that looks like the suite's is not read as the suite's", "not reached is matched in any case") fail against c8255eb's tools/runs.ts and pass at this head. Remaining, not blocking: for a log from an older lab with no suite lines, the fallback still reads the runner's spec skip lines from the whole log, and that can only make a run read worse.

At this head, niced: modules/lab 30 pass, 0 fail. Merged locally with #288 at 0fcc560: clean, 44 pass, 0 fail.

Merge order: mesh-lab #105 first, then this one; #288 merges cleanly with it in either order.

Reviewed by an independent reviewer on g14; no lab bed ran.

Verdict: sound, mergeable at acf8a76646de; the follow-up of review 254 is answered. Re-review for issue 693 (a lab run whose only test skipped is recorded as passed). Checks at this head: mesh/merge-gate success ("PASS — every machine composes with the change as it did without (4 of 4 compose) (modules: lab)"), mesh/repo-check success ("PASS — its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-check-typescript.sh (typescript)"), mesh/delivery success ("ready: waits for its merge"), mesh/delivery-group success ("ready: every member ready, and composed together they pass; composed check pass — every machine composes with the change as it did without (4 of 4 compose)"). 1. outcomeOf reads the suite's "NOT PASSED:" and "skipped:" lines only after the last "recorded:" line, which the suite prints after the runner has exited. I re-ran my probe, a genuinely passing log (pass 1, fail 0) with a test-printed "NOT PASSED: printed by a test", a fake "skipped: x (y.ts) # NOT REACHED" and even a fake "recorded:" line before the real one: it now returns no notPassed and no skips, so the run stays passed. 2. NOT REACHED is matched in any case. 3. The new tests ("a line a test printed that looks like the suite's is not read as the suite's", "not reached is matched in any case") fail against c8255eb's tools/runs.ts and pass at this head. Remaining, not blocking: for a log from an older lab with no suite lines, the fallback still reads the runner's spec skip lines from the whole log, and that can only make a run read worse. At this head, niced: modules/lab 30 pass, 0 fail. Merged locally with #288 at 0fcc560: clean, 44 pass, 0 fail. Merge order: mesh-lab #105 first, then this one; #288 merges cleanly with it in either order. Reviewed by an independent reviewer on g14; no lab bed ran.

mesh-admin merged commit e846c289c8 into main

2026-10-14 05:05:40 +00:00

mesh-admin deleted branch fix/693-a-run-that-proved-nothing-is-not-passed

2026-10-14 05:05:40 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#287