novox/mesh-lab
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

A run rebuilds what it tests, and leaves a receipt saying what it covered

Browse Source 
The danger is not that the suite breaks. It is that nobody notices it
stopped running (novox/hq 04-ISSUES/005). The harness this replaces had
not built for two and a half months and nothing said so — and this suite
needs a hypervisor, so it inherits exactly that: it runs when somebody
remembers, and remembering is not a mechanism.

So running, recording, and rebuilding are one act:

- the host binary, control-plane image and builder are rebuilt from
  source first. The last two both parse manifests; building one and not
  the other left a binary eleven hours old refusing a field the mesh had
  just renamed, found by a full run.
- a receipt lands in XDG state — outside git, because the question is
  whether *this machine* has run it, and a receipt in git would be a
  claim about everybody's machine made by whoever committed last.
- `last-run` judges it and exits non-zero when it no longer counts.

Three faults found by running the thing rather than reading it, each now
held by a test confirmed to fail without it:

- counted() passed every test while parsing nothing. The runner colours
  its summary even into a pipe; the fixtures were clean text that had
  been imagined rather than captured. A fixture that agrees with the
  mistake proves the mistake.
- a receipt for `suite test/lastrun.test.ts` was indistinguishable from
  one for the real thing — 005's own symptom, rebuilt inside its remedy.
  The receipt now records what ran.
- a tree with uncommitted work reported the bare commit, claiming
  coverage of code nobody can check out. Nothing else could tell: the
  hash is identical either way.

Proven on real machines: 22/22, against all three repositories.
This commit is contained in:
jschoubben
2026-08-31 15:02:19 +02:00
parent e1317c9a69
commit 033ad7ec69
11 changed files with 762 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/integration/mesh.test.ts
+11 -11
View File
@@ -1396,17 +1396,6 @@ test("a service is reached by a name under the machine it runs on", {
await mesh("push");
await new Promise((r) => setTimeout(r, 25_000));
// `on`, not `must`: `is-active` exits non-zero for a unit that failed, so `must` would throw
// before the assertion below — taking every diagnostic with it. That happened, and the run said
// only "failed".
for (const machine of ["anchor", "laptop"]) {
const state = await on(machine, `systemctl is-active dnsmasq.service`);
if (state.out.trim() === "active") continue;
assert.fail(`the resolver is not running on ${machine} (${state.out.trim()}):\n\n` +
`its config:\n${(await on(machine, `cat /etc/dnsmasq.conf`)).out}\n` +
`${await diagnose(machine)}`);
}
// Everything this test could want to know, gathered in one place.
//
// Three times now a diagnostic has not run because the thing before it threw: `must` on a
@@ -1424,6 +1413,17 @@ test("a service is reached by a name under the machine it runs on", {
`asked directly:\n${(await on(machine,
`timeout 5 resolvectl query postgres.anchor.internal 2>&1 || echo "no answer"`)).out}`;
// `on`, not `must`: `is-active` exits non-zero for a unit that failed, so `must` would throw
// before the assertion below — taking every diagnostic with it. That happened, and the run said
// only "failed".
for (const machine of ["anchor", "laptop"]) {
const state = await on(machine, `systemctl is-active dnsmasq.service`);
if (state.out.trim() === "active") continue;
assert.fail(`the resolver is not running on ${machine} (${state.out.trim()}):\n\n` +
`its config:\n${(await on(machine, `cat /etc/dnsmasq.conf`)).out}\n` +
`${await diagnose(machine)}`);
}
// Through the machine's own resolver, by the path an application actually takes: nsswitch, then
// files, then DNS. `dig` would ask a server directly and prove less — the resolv.conf module is
// half of what is being tested, and only this path goes through it.
test/lastrun.test.ts
+167
View File
@@ -0,0 +1,167 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { execFileSync } from "node:child_process";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { endToEnd, headOf, judge, type Receipt } from "../src/lastrun.ts";
const now = new Date("2026-08-31T12:00:00Z");
const passing = (at: string, against: Record<string, string>): Receipt =>
({ at, passed: 22, failed: 0, against, ran: [endToEnd] });
// A machine that has never run it is told so, rather than told nothing.
//
// novox/hq 04-ISSUES/005: the harness it replaces had not built for two and a half months and
// nothing said so. Silence and success must never look alike.
test("a machine that has never run the suite is told so", () => {
const said = judge(null, now, { "mesh-lab": "aaa" });
assert.equal(said.current, false);
assert.match(said.lines.join("\n"), /never run/);
});
// The one that matters: it passed, and against code nobody runs any more.
test("a run against code that has since changed is not current", () => {
const said = judge(
passing("2026-08-31T11:00:00Z", { "mesh-lab": "aaa", "mesh-control": "bbb" }),
now,
{ "mesh-lab": "aaa", "mesh-control": "ccc" },
);
assert.equal(said.current, false, "a run against changed code was reported as current");
const text = said.lines.join("\n");
assert.match(text, /mesh-control\s+at bbb, now at ccc/, text);
assert.match(text, /code that has since changed/, text);
});
// Passing, recent, and against exactly this code is the only thing that counts.
test("a recent run against this code is current", () => {
const said = judge(
passing("2026-08-31T11:00:00Z", { "mesh-lab": "aaa" }),
now,
{ "mesh-lab": "aaa" },
);
assert.equal(said.current, true, said.lines.join("\n"));
assert.match(said.lines.join("\n"), /unchanged/);
});
// Old is a different complaint from moved, and says so — otherwise somebody goes looking for a
// change that did not happen.
test("a run that is merely old says that, not that something changed", () => {
const said = judge(
passing("2026-08-01T11:00:00Z", { "mesh-lab": "aaa" }),
now,
{ "mesh-lab": "aaa" },
);
assert.equal(said.current, false);
const text = said.lines.join("\n");
assert.match(text, /Nothing has changed since/, text);
assert.doesNotMatch(text, /has since changed/, text);
});
// A failed run is recorded, and does not count as coverage.
test("a run that failed is not coverage", () => {
const said = judge(
{
at: "2026-08-31T11:00:00Z",
passed: 21,
failed: 1,
against: { "mesh-lab": "aaa" },
ran: [endToEnd],
},
now,
{ "mesh-lab": "aaa" },
);
assert.equal(said.current, false);
assert.match(said.lines.join("\n"), /1 test\(s\) failed/);
assert.match(said.lines.join("\n"), /Nothing has been proven end to end since/);
});
// A repository the run never accounted for is named, rather than passing silently: a receipt that
// says nothing about something is not a receipt that clears it.
test("a repository the run did not account for is named", () => {
const said = judge(
passing("2026-08-31T11:00:00Z", { "mesh-lab": "aaa" }),
now,
{ "mesh-lab": "aaa", "mesh-host": "ddd" },
);
assert.equal(said.current, false);
assert.match(said.lines.join("\n"), /mesh-host\s+was not accounted for/);
});
// A green run of something else is not a green run of this.
//
// The suite takes paths, so it can be pointed at one quick unit file. Without recording what it
// ran, that receipt and a receipt for the real thing are the same document — which is the whole
// fault of novox/hq 04-ISSUES/005, reintroduced by the fix for it.
test("a run that raised no machines is not end-to-end coverage", () => {
const said = judge(
{
at: "2026-08-31T11:00:00Z",
passed: 6,
failed: 0,
against: { "mesh-lab": "aaa" },
ran: ["test/lastrun.test.ts"],
},
now,
{ "mesh-lab": "aaa" },
);
assert.equal(said.current, false, "a unit run was accepted as end-to-end coverage");
assert.match(said.lines.join("\n"), /raised no machines/);
});
// A receipt written before the mesh recorded what it ran claims nothing, and is read as claiming
// nothing — not as claiming everything.
test("a receipt from before this was recorded is not read as covering everything", () => {
const old = { at: "2026-08-31T11:00:00Z", passed: 22, failed: 0, against: { "mesh-lab": "aaa" } };
const said = judge(old as unknown as Receipt, now, { "mesh-lab": "aaa" });
assert.equal(said.current, false);
});
// A dirty tree is never equal to the clean commit it sits on.
//
// The run tested what was on disk. Naming the bare hash would claim coverage of code nobody can
// check out — and nothing else could tell, because the hash is identical either way.
test("a run taken against uncommitted work does not count as covering the commit", () => {
const said = judge(passing("2026-08-31T11:00:00Z", { "mesh-lab": "aaa+uncommitted" }), now, {
"mesh-lab": "aaa",
});
assert.equal(said.current, false, "a run against uncommitted work was read as covering the commit");
assert.match(said.lines.join("\n"), /aaa\+uncommitted, now at aaa/);
});
// headOf against a real repository, because the rule lives in headOf and not in judge.
//
// The first test written for this marked a hand-built receipt and passed with the marking removed
// — it checked how judge reads the value, never that anything produces it. A test that cannot fail
// when the behaviour is deleted is not defending the behaviour.
test("a repository with uncommitted work reports a commit that is marked as such", () => {
const repo = mkdtempSync(join(tmpdir(), "mesh-lab-headof-"));
try {
const git = (...args: string[]) =>
execFileSync("git", ["-C", repo, ...args], { stdio: ["ignore", "pipe", "ignore"] });
git("init", "-q");
git("config", "user.email", "test@example.invalid");
git("config", "user.name", "test");
writeFileSync(join(repo, "a"), "one\n");
git("add", "a");
git("commit", "-qm", "first");
const clean = headOf(repo);
assert.match(clean, /^[0-9a-f]+$/, `a clean tree was reported as ${clean}`);
writeFileSync(join(repo, "a"), "two\n");
assert.equal(headOf(repo), `${clean}+uncommitted`, "an uncommitted change was not marked");
} finally {
rmSync(repo, { recursive: true, force: true });
}
});
// A directory that is not a checkout is absent from the receipt, not guessed at.
test("a directory that is not a repository reports nothing", () => {
const plain = mkdtempSync(join(tmpdir(), "mesh-lab-plain-"));
try {
assert.equal(headOf(plain), "");
} finally {
rmSync(plain, { recursive: true, force: true });
}
});
test/rebuild.test.ts
+47
View File
@@ -0,0 +1,47 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { planned } from "../src/rebuild.ts";
import { repositories } from "../src/repos.ts";
// The control plane's image and the builder are one step, not two.
//
// Both parse manifests. On 2026-08-30 a rename was built into the image and not the binary, and
// the run that found out was a full lab raise. novox/hq 04-ISSUES/005.
test("the control plane's image and builder are always built together", () => {
const builds = planned({
MESH_LAB_MODULES: "/repo/control/examples/modules",
MESH_LAB_BUILDER: "/repo/control/build/mesh-builder",
});
const what = builds.map((b) => b.what);
assert.ok(what.includes("control plane image"), "the image was not built");
assert.ok(what.includes("builder"), "the builder was not built");
for (const build of builds) assert.equal(build.in, "/repo/control");
});
// A repository this run was not pointed at is not built, and not claimed.
test("only what this run was pointed at is built", () => {
assert.deepEqual(planned({}), []);
const hostOnly = planned({ MESH_LAB_HOST_BINARY: "/repo/host/mesh-host" });
assert.deepEqual(hostOnly.map((b) => b.what), ["host"]);
assert.equal(hostOnly[0]!.in, "/repo/host");
});
// What the receipt claims and what the run built come from one derivation.
//
// They are separate concerns that must agree: a receipt naming a repository the run did not build
// is false coverage arriving by nobody's decision — just two derivations drifting apart.
// novox/hq 04-ISSUES/005.
test("every repository the receipt claims was built by the run", () => {
const env = {
MESH_LAB_HOST_BINARY: "/repo/host/mesh-host",
MESH_LAB_MODULES: "/repo/control/examples/modules",
MESH_LAB_BUILDER: "/repo/control/build/mesh-builder",
};
const built = new Set(planned(env).map((b) => b.in));
for (const [name, directory] of Object.entries(repositories(env))) {
// mesh-lab is the exception, and it is not an omission: it is TypeScript run from source, so
// the code under test *is* the code running. There is nothing to build and nothing to go stale.
if (name === "mesh-lab") continue;
assert.ok(built.has(directory), `${name} (${directory}) is claimed but never built`);
}
});
test/suite.test.ts
+68
View File
@@ -0,0 +1,68 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { counted, reportOn } from "../src/suite.ts";
// The totals come from the runner's own summary, and from nothing else.
test("the runner's totals are read from its summary", () => {
const said = counted("✔ something (1ms)\nℹ tests 22\nℹ pass 22\nℹ fail 0\n");
assert.deepEqual(said, { passed: 22, failed: 0 });
});
// A test *named* like a total must not be mistaken for one. The summary is a line of its own, and
// the pattern says so — otherwise a test called "pass 3" would rewrite the record.
test("a test named like a total is not a total", () => {
const said = counted("✔ a machine reports pass 3 things (1ms)\nℹ pass 22\nℹ fail 0\n");
assert.equal(said.passed, 22, "a test name was read as the total");
});
// A failing run is read as a failing run.
test("failures are read", () => {
const said = counted("ℹ pass 21\nℹ fail 1\n");
assert.deepEqual(said, { passed: 21, failed: 1 });
});
// **No totals is not zero failures.** A run whose result could not be read is a run nobody can say
// anything about, and writing "0 failed" because nothing said otherwise is how a green record
// comes to mean nothing — which is the whole of 04-ISSUES/005.
test("output with no summary yields no totals rather than a clean bill", () => {
const said = counted("the runner crashed before it said anything\n");
assert.equal(said.passed, null);
assert.equal(said.failed, null);
});
// No receipt rather than a guessed one.
//
// The rule that keeps the record meaning something, and it was first written where nothing could
// check it — 04-ISSUES/005 in miniature, inside the fix for it.
test("a run whose result could not be read writes nothing", () => {
let wrote = false;
const said = reportOn({ passed: null, failed: null }, () => {
wrote = true;
return { against: {}, ran: [] };
});
assert.equal(wrote, false, "a receipt was written for a run nobody could read");
assert.match(said, /no receipt written/);
});
test("a run that was read is recorded, with what it was read against", () => {
let got: [number, number] | null = null;
const said = reportOn({ passed: 22, failed: 0 }, (p, f) => {
got = [p, f];
return { against: { "mesh-lab": "abc1234" }, ran: ["test/integration/mesh.test.ts"] };
});
assert.deepEqual(got, [22, 0]);
assert.match(said, /mesh\.test\.ts — 22 passed, 0 failed, against mesh-lab abc1234/);
});
// The runner's real output, colours and all.
//
// Captured from `node --test` writing into a pipe rather than written by hand: the first version of
// counted() passed every test and read nothing, because the fixtures were clean text and the runner
// emits escape codes. A fixture that agrees with the mistake proves the mistake.
test("the runner's totals are read from output as it actually arrives", () => {
const real = "\u001b[34m\u2139 suites 0\u001b[39m\n" +
"\u001b[34m\u2139 pass 22\u001b[39m\n" +
"\u001b[34m\u2139 fail 0\u001b[39m\n" +
"\u001b[34m\u2139 duration_ms 98.9\u001b[39m\n";
assert.deepEqual(counted(real), { passed: 22, failed: 0 });
});