A run rebuilds what it tests, and leaves a receipt saying what it covered

The danger is not that the suite breaks. It is that nobody notices it
stopped running (novox/hq 04-ISSUES/005). The harness this replaces had
not built for two and a half months and nothing said so — and this suite
needs a hypervisor, so it inherits exactly that: it runs when somebody
remembers, and remembering is not a mechanism.

So running, recording, and rebuilding are one act:

- the host binary, control-plane image and builder are rebuilt from
  source first. The last two both parse manifests; building one and not
  the other left a binary eleven hours old refusing a field the mesh had
  just renamed, found by a full run.
- a receipt lands in XDG state — outside git, because the question is
  whether *this machine* has run it, and a receipt in git would be a
  claim about everybody's machine made by whoever committed last.
- `last-run` judges it and exits non-zero when it no longer counts.

Three faults found by running the thing rather than reading it, each now
held by a test confirmed to fail without it:

- counted() passed every test while parsing nothing. The runner colours
  its summary even into a pipe; the fixtures were clean text that had
  been imagined rather than captured. A fixture that agrees with the
  mistake proves the mistake.
- a receipt for `suite test/lastrun.test.ts` was indistinguishable from
  one for the real thing — 005's own symptom, rebuilt inside its remedy.
  The receipt now records what ran.
- a tree with uncommitted work reported the bare commit, claiming
  coverage of code nobody can check out. Nothing else could tell: the
  hash is identical either way.

Proven on real machines: 22/22, against all three repositories.
This commit is contained in:
2026-08-31 15:02:19 +02:00
parent e1317c9a69
commit 033ad7ec69
11 changed files with 762 additions and 15 deletions
+68
View File
@@ -0,0 +1,68 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { counted, reportOn } from "../src/suite.ts";
// The totals come from the runner's own summary, and from nothing else.
test("the runner's totals are read from its summary", () => {
const said = counted("✔ something (1ms)\nℹ tests 22\nℹ pass 22\nℹ fail 0\n");
assert.deepEqual(said, { passed: 22, failed: 0 });
});
// A test *named* like a total must not be mistaken for one. The summary is a line of its own, and
// the pattern says so — otherwise a test called "pass 3" would rewrite the record.
test("a test named like a total is not a total", () => {
const said = counted("✔ a machine reports pass 3 things (1ms)\nℹ pass 22\nℹ fail 0\n");
assert.equal(said.passed, 22, "a test name was read as the total");
});
// A failing run is read as a failing run.
test("failures are read", () => {
const said = counted("ℹ pass 21\nℹ fail 1\n");
assert.deepEqual(said, { passed: 21, failed: 1 });
});
// **No totals is not zero failures.** A run whose result could not be read is a run nobody can say
// anything about, and writing "0 failed" because nothing said otherwise is how a green record
// comes to mean nothing — which is the whole of 04-ISSUES/005.
test("output with no summary yields no totals rather than a clean bill", () => {
const said = counted("the runner crashed before it said anything\n");
assert.equal(said.passed, null);
assert.equal(said.failed, null);
});
// No receipt rather than a guessed one.
//
// The rule that keeps the record meaning something, and it was first written where nothing could
// check it — 04-ISSUES/005 in miniature, inside the fix for it.
test("a run whose result could not be read writes nothing", () => {
let wrote = false;
const said = reportOn({ passed: null, failed: null }, () => {
wrote = true;
return { against: {}, ran: [] };
});
assert.equal(wrote, false, "a receipt was written for a run nobody could read");
assert.match(said, /no receipt written/);
});
test("a run that was read is recorded, with what it was read against", () => {
let got: [number, number] | null = null;
const said = reportOn({ passed: 22, failed: 0 }, (p, f) => {
got = [p, f];
return { against: { "mesh-lab": "abc1234" }, ran: ["test/integration/mesh.test.ts"] };
});
assert.deepEqual(got, [22, 0]);
assert.match(said, /mesh\.test\.ts — 22 passed, 0 failed, against mesh-lab abc1234/);
});
// The runner's real output, colours and all.
//
// Captured from `node --test` writing into a pipe rather than written by hand: the first version of
// counted() passed every test and read nothing, because the fixtures were clean text and the runner
// emits escape codes. A fixture that agrees with the mistake proves the mistake.
test("the runner's totals are read from output as it actually arrives", () => {
const real = "\u001b[34m\u2139 suites 0\u001b[39m\n" +
"\u001b[34m\u2139 pass 22\u001b[39m\n" +
"\u001b[34m\u2139 fail 0\u001b[39m\n" +
"\u001b[34m\u2139 duration_ms 98.9\u001b[39m\n";
assert.deepEqual(counted(real), { passed: 22, failed: 0 });
});