A canary first: one machine, one path, three minutes

Suggested by Jochen, and it paid for itself on its first run.

A suite that takes forty minutes is a suite you hear from once a day.
Every fault found today would have shown up in the first three minutes
of it — a module pinned to an image that does not exist, a consumer
given a password and no name to present with it, a credential file
nothing could read, a search for a password that read the password as an
option. The other thirty-seven minutes proved things that were already
working.

So this runs first, on one machine, with the three images the mesh needs
for itself. It walks one path: a mesh comes up, a module lands, and a
consumer gets a credential it can actually use — the name to present,
the address, the port, and a password only the host could put there.
Deliberately not a smaller copy of the full suite: that path is where
everything went wrong, and a canary checking many things shallowly is a
canary whose failure nobody can read.

`suite` runs it and stops if it dies, saying why rather than leaving
somebody to wonder what the missing thirty-seven minutes would have
said. Skipped when the caller named its own files.

It measured 164 seconds against forty-odd minutes, and failed three
times on its first run for one reason: applying the bundle raises a
control plane but does not tell it a machine exists. I had left out
enrolment, and the long suite would have taken forty minutes to say so.
This commit is contained in:
2026-09-01 16:24:19 +02:00
parent 44d3e53bcb
commit 6591a2e040
3 changed files with 214 additions and 4 deletions
+9
View File
@@ -50,6 +50,15 @@ export interface Receipt {
*/
export const endToEnd = "test/integration/mesh.test.ts";
/**
* canary is the short run that goes first.
*
* One machine, three images, one path walked end to end. **A suite that takes forty minutes is a
* suite you hear from once a day** — and every fault found on 2026-09-01 would have shown up in
* the first three minutes of it. Running this first means a broken change costs minutes.
*/
export const canary = "test/integration/canary.test.ts";
/** Where the receipt lives: XDG state, which is for exactly this — data a tool keeps between runs. */
export function receiptPath(): string {
const state = process.env["XDG_STATE_HOME"] ?? join(homedir(), ".local", "state");
+28 -4
View File
@@ -10,7 +10,7 @@
*/
import { spawn } from "node:child_process";
import { endToEnd, record } from "./lastrun.ts";
import { canary, endToEnd, record } from "./lastrun.ts";
import { rebuild } from "./rebuild.ts";
/** counted is what the runner said, or nulls when it said nothing recognisable. */
@@ -44,6 +44,32 @@ export async function runSuite(args: string[]): Promise<number> {
if (built.length > 0) console.log(`built: ${built.join(", ")}\n`);
}
// **The canary first, and stop if it dies.** It walks one path on one machine: a mesh comes up,
// a module lands, a consumer gets a credential it can use. Everything that broke on
// 2026-09-01 broke on that path, and finding out took forty minutes each time because the long
// run had to reach it.
//
// Skipped when the caller named its own files — they asked for something specific — and when
// the canary is itself what was asked for.
if (ran.length === 0) {
const first = await runFiles([canary]);
if (first.code !== 0) {
console.log(
`\nthe canary failed, so the rest was not run. It is one machine and one path: a mesh ` +
`comes up, a module lands, a consumer gets a credential. Fix that first — the long ` +
`suite would fail on the same thing forty minutes later.`);
return first.code;
}
console.log("");
}
const { code, seen } = await runFiles(files);
console.log("\n" + reportOn(counted(seen), (p, f) => record(p, f, files)));
return code;
}
/** Run some test files, passing their output through as it arrives. */
async function runFiles(files: string[]): Promise<{ code: number; seen: string }> {
const running = spawn(
process.execPath,
["--test", "--test-concurrency=1", "--experimental-strip-types",
@@ -61,9 +87,7 @@ export async function runSuite(args: string[]): Promise<number> {
const code: number = await new Promise((resolve) => {
running.on("close", (c) => resolve(c ?? 1));
});
console.log("\n" + reportOn(counted(seen), (p, f) => record(p, f, files)));
return code;
return { code, seen };
}
/**