novox/mesh-lab
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Say what the lab is doing, while it is doing it

Browse Source 
novox/hq 04-ISSUES/024. A run stalled for thirty-five minutes and said
nothing. The cause was a link systemd was still configuring, three
layers down inside a `docker load` blocked on a socket — and every one
of those layers knew what it was waiting for. None of them said so.

Three decisions, each doing work.

**Every external command is logged, at the three places that run one.**
Ninety-seven call sites reach a hypervisor or a container runtime
through three wrappers, so instrumenting the wrappers covers all of them
and nothing has to remember to log.

**A command still running says so while it runs.** A line before and a
line after tells you nothing until the after arrives, which is exactly
the case that matters. Anything outstanding past fifteen seconds reports
itself with how long it has been going. It is reported as still running,
not as stuck — which it is is not knowable from there, and a log that
calls a slow step a hang teaches people to ignore it.

**It goes to a file, written synchronously.** Node block-buffers stdout
when redirected and a test runner buffers it again, so a console log can
sit minutes behind. `appendFileSync` cannot lag.

Two things this found in itself while being written, both the same shape
as what it exists to catch:

A question that answers no is not a fault. Half the lab's commands are
questions — does this network exist, is the agent up yet — and they fail
constantly while a scenario comes up. Logging those as faults filled a
healthy run with ✗, which is how you end up ignoring ✗ when one is real.
They are recorded quietly now, and still recorded.

And `around` skipped its own wrapper when a step's level was below the
configured one — taking the failure line and the heartbeat with it. The
two things worth having at a low level were the two that vanished at
exactly the level somebody would use. The gate belongs in `write`.

Also unsilences the four call sites that passed a callback throwing
everything away, including the one the stall sat in, and tees `raise`'s
progress into the file whether or not a caller asked to see it — the
end-to-end test passed no callback, so the one run that mattered
reported not a single step.
This commit is contained in:
jschoubben
2026-09-01 10:43:38 +02:00
parent 5137720aa7
commit bb14ecb7e0
8 changed files with 464 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lifecycle/place.ts
+16
View File
@@ -18,6 +18,7 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import { incus, incusOk, succeeds } from "../incus/client.ts";
import { around, log, shorten } from "../log.ts";
/**
* What this stage can put inside a machine.
@@ -322,6 +323,18 @@ function local(
command: string,
args: string[],
timeoutMs: number,
): Promise<{ ok: boolean; stdout: string; stderr: string }> {
// The third and last place the lab runs an external program (novox/hq 04-ISSUES/024).
// `docker save` of a large image is the slowest single thing a raise does.
return around(`${command} ${shorten(args)}`, () => runLocal(command, args, timeoutMs), {
heartbeatMs: 15_000,
});
}
function runLocal(
command: string,
args: string[],
timeoutMs: number,
): Promise<{ ok: boolean; stdout: string; stderr: string }> {
return new Promise((resolve) => {
const child = spawn(command, args, { stdio: ["ignore", "pipe", "pipe"] });
@@ -336,6 +349,9 @@ function local(
});
child.on("close", (code) => {
clearTimeout(timer);
if (code !== 0) {
log.debug(` exit ${code}: ${shorten([stderr.trim() || "(nothing on stderr)"], 400)}`);
}
resolve({ ok: code === 0, stdout, stderr });
});
});
src/lifecycle/raise.ts
+41 -14
View File
@@ -25,6 +25,7 @@ import { applyHostFirewalls } from "./firewall.ts";
import { IMAGE_PREFIX, BASE_IMAGE_ALIAS, BASE_IMAGE_HOWTO, planPlacements, applyPlacements } from "./place.ts";
import { baseImageExists, UPSTREAM_IMAGE } from "./base.ts";
import { discardStock, raiseRegistry, stockRegistry } from "./registry.ts";
import { log as record } from "../log.ts";
/** Drivers whose snapshots are copy-on-write. On `dir` a snapshot is a full copy. */
const COW_DRIVERS = ["btrfs", "zfs"];
@@ -174,7 +175,19 @@ export async function raise(
scenario: Scenario,
options: RaiseOptions = {},
): Promise<RaisedScenario> {
const log = options.onProgress ?? (() => {});
// **Progress always reaches the file, whether or not anyone asked to see it.**
//
// This used to be the caller's callback or nothing, and every function below takes its `log`
// from here — so a caller that passed none silenced the whole lifecycle. That is exactly what
// happened: the end-to-end test called `raise` with no callback, so the one run that mattered
// reported not a single step (novox/hq 04-ISSUES/024).
//
// Teeing rather than replacing: the caller still gets what it asked for, and the record is kept
// regardless. A record nobody switched on is the one you want after the thing goes wrong.
const log = (message: string): void => {
record.info(message);
options.onProgress?.(message);
};
// A scenario that places a runtime or an image needs machines built from the base image,
// because a sealed machine cannot install one (novox/hq ADR 0006). Chosen here rather than
@@ -198,19 +211,33 @@ export async function raise(
// has been created, and there is no wreckage to leave standing.
assertSupported(scenario);
let step = "choosing a storage pool";
// **The step is the log.** Setting it and recording it are one act, so a step added later
// cannot be a step that goes unrecorded — which is the drift that made a thirty-five minute
// stall untraceable (novox/hq 04-ISSUES/024). Each entry closes the previous one with its
// duration, so the log says where a raise spends its time as well as where it stopped.
let step = "";
let stepFrom = Date.now();
const enter = (next: string): string => {
if (step) record.info(` ${step} — ${((Date.now() - stepFrom) / 1000).toFixed(1)}s`);
record.info(`▶ ${next}`);
stepFrom = Date.now();
step = next;
return next;
};
enter("choosing a storage pool");
try {
const pool = await choosePool(log);
log(`instance ${instanceId} pool ${pool}`);
step = "creating segments";
enter("creating segments");
const networks: string[] = [];
for (const [segment, spec] of Object.entries(scenario.segments)) {
networks.push(await createNetwork(instanceId, segment, spec));
log(` segment ${segment}`);
}
step = "creating machines";
enter("creating machines");
const created: string[] = [];
const byMachine = new Map<string, string>();
for (const [machine, spec] of Object.entries(scenario.machines)) {
@@ -221,32 +248,32 @@ export async function raise(
log(` machine ${machine}${spec.at === "detached" ? " (detached)" : ""}`);
}
step = "starting machines";
enter("starting machines");
for (const name of created) {
await succeeds(["start", name], 60_000);
}
step = "waiting for machines to become usable";
enter("waiting for machines to become usable");
await waitUntilAllUsable(created, readyTimeout, log);
step = "applying declared addresses";
enter("applying declared addresses");
await applyAddresses(scenario, instanceId, byMachine, log);
// Transit first: a gateway's default route points at it, so it has to exist.
step = "wiring the public networks together";
enter("wiring the public networks together");
const transit = await raiseTransit(scenario, instanceId, log);
step = "raising routers";
enter("raising routers");
const routers = await raiseRouters(scenario, instanceId, planRouters(scenario, instanceId), log);
if (transit) routers.push(transit);
// Stocked on this workstation, where there is a network, and served from inside the
// scenario, where there is not (novox/hq 04-ISSUES/009).
step = "stocking the registry";
enter("stocking the registry");
const stock = await stockRegistry(scenario.images ?? [], log);
let registry: Awaited<ReturnType<typeof raiseRegistry>> = null;
try {
step = "raising the registry";
enter("raising the registry");
registry = await raiseRegistry(scenario, instanceId, stock, log);
} finally {
// Cleaning up scratch must not fail a raise that succeeded. The scenario is standing
@@ -258,17 +285,17 @@ export async function raise(
}
}
step = "routing machines through their gateways";
enter("routing machines through their gateways");
await applyDefaultRoutes(scenario, byMachine, log);
// Last: a machine that refuses inbound must still have been reachable while the lab
// was configuring it.
step = "applying host firewalls";
enter("applying host firewalls");
await applyHostFirewalls(scenario, byMachine, log);
// Last, and only once the underlay is real. Placing before the machines can reach each
// other would test the host against a network the scenario does not describe.
step = "placing";
enter("placing");
await applyPlacements(scenario, byMachine, log);
return {
src/lifecycle/registry.ts
+19 -1
View File
@@ -23,6 +23,7 @@ import { spawn } from "node:child_process";
import { incus, incusOk, succeeds } from "../incus/client.ts";
import { macFor, networkName } from "./names.ts";
import { addressLink } from "./address.ts";
import { around, log, shorten } from "../log.ts";
import { BASE_IMAGE_ALIAS, placeImage } from "./place.ts";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
@@ -171,6 +172,16 @@ async function waitForRegistry(port: number): Promise<void> {
function docker(
args: string[],
timeoutMs: number,
): Promise<{ ok: boolean; stdout: string; stderr: string }> {
// The second of the three places the lab runs an external program (novox/hq 04-ISSUES/024).
// `docker push` of a large image is minutes of legitimate silence, which is exactly when a
// heartbeat earns its keep.
return around(`docker ${shorten(args)}`, () => runDocker(args, timeoutMs), { heartbeatMs: 15_000 });
}
function runDocker(
args: string[],
timeoutMs: number,
): Promise<{ ok: boolean; stdout: string; stderr: string }> {
return new Promise((resolve) => {
const child = spawn("docker", args, { stdio: ["ignore", "pipe", "pipe"] });
@@ -185,6 +196,11 @@ function docker(
});
child.on("close", (code) => {
clearTimeout(timer);
// A docker failure is an answer here rather than an exception, so it would otherwise pass
// through the log looking exactly like a success.
if (code !== 0) {
log.debug(` exit ${code}: ${shorten([stderr.trim() || "(nothing on stderr)"], 400)}`);
}
resolve({ ok: code === 0, stdout, stderr });
});
});
@@ -321,7 +337,9 @@ export async function raiseRegistry(
// The registry's own image, placed by tag — an archive keeps a tag and cannot keep a digest,
// which is the whole reason this machine exists.
await placeImage(name, "registry", REGISTRY_IMAGE, () => {});
// Logged, not silenced. This is the step a stall sat in for thirty-five minutes while the
// caller had passed it a callback that threw everything away (novox/hq 04-ISSUES/024).
await placeImage(name, "registry", REGISTRY_IMAGE, log);
// The destination must EXIST before a recursive push, or incus copies the source's contents
// rather than the source — the data lands one directory too shallow, the registry finds
src/lifecycle/router.ts
+3 -3
View File
@@ -62,7 +62,7 @@ export async function ensureRouterImage(log: (message: string) => void = () => {
// Default profile on purpose: this is the one container that needs to reach a repository.
await incus(["launch", ROUTER_BASE, builder], 300_000);
await waitUntilUsable(builder, 120, () => {});
await waitUntilUsable(builder, 120, log);
// `exec` works before the container has an address. Usable means a command runs; it does
// not mean the network is up, and the first attempt failed on DNS because those were
@@ -279,7 +279,7 @@ export async function raiseTransit(
}
}
await succeeds(["start", name], 60_000);
await waitUntilUsable(name, 120, () => {});
await waitUntilUsable(name, 120, log);
for (const [index, segment] of publicSegments.entries()) {
const device = `eth${index}`;
@@ -343,7 +343,7 @@ export async function raiseRouters(
}
for (const name of created) {
await waitUntilUsable(name, 120, () => {});
await waitUntilUsable(name, 120, log);
}
for (const plan of plans) {