Scenario lifecycle: raise, exec, snapshot, restore, destroy

A declaration goes in and a disposable mesh comes out. Verified on a
workstation, not asserted: two machines raised and addressed in 14.6s,
snapshot 0.28s, restore-to-usable 11.6s, both families pinging with no
loss, and the workstation with no route into any of it.

The declaration layer implements the model in full — three positions a
machine can be in, keyed on forwardability; gateways carrying the address
the world sees them as; both address families; multi-homing; MTU;
inter-segment policy. It is validated hard because the failures it prevents
are silent: a private range on a public segment produces no error, the mesh
simply never forms. Public segments are refused unless they use RFC 5737 or
RFC 3849 space, and a range wider than the reserved block is refused too.
33 tests, all offline.

The runtime implements less than the model, and refuses the difference.
A scenario declaring gateways, published ports, policy, inbound deny or
place is rejected at raise with every gap named. Raising it would produce a
mesh that silently lacks what it declared, which is the fault this lab
exists to catch — 04-ISSUES/003, where a firewall key is declared in five
manifests and read by no code.

Three bugs found by review and by running it, all of one family:

The readiness check truthiness-tested incusOk's return. `exec … true`
succeeds with EMPTY output, so every machine reported unreachable while
incus exec on it worked perfectly. succeeds() now exists so the mistake is
not available, and network delete had the same bug — it counted zero
segments removed while removing them.

list() split instance from machine on the last dash, so a machine called
home-server absorbed half the instance id and destroy found nothing.
Resources are now found by the metadata they carry, never by name.

restore reported success in 0.79s while the machine's agent was still
starting, so the next command failed. Both raise and restore now wait for
usable and say how long that took — reporting the earlier number is
transport reported as effect, which is the fault the lab is being built to
find.

Two incus behaviours worth recording. Its CLI reads a YAML definition from
stdin when stdin is not a terminal, so a spawned command hangs until the
timeout kills it and arrives with empty stderr — a failure with no
explanation, on a command that works when typed. And it assigns a MAC at
runtime without recording it in device config, so MACs are derived and set
explicitly, which the guest needs anyway: it names interfaces by bus
position, and matching by name configures the wrong one on a multi-homed
machine.

No build step; Node strips the types. The lifecycle has no unit tests
because a fake hypervisor would assert that the fake behaves as expected,
which is the shape of test this project exists to stop shipping.

src/incus/client.ts

@@ -0,0 +1,216 @@
+/**
+ * A thin wrapper over the incus CLI. Thin on purpose: the lab's value is in the scenario
+ * model, not in re-describing a hypervisor.
+ *
+ * Everything here goes through incus's own channel and never over IP. A scenario is a
+ * closed address space — two scenarios raised from one declaration hold the same
+ * addresses and must never meet — so the workstation has no route into either, and
+ * reaching a machine by address would make concurrency impossible in the worst way:
+ * not with an error, but with one scenario's traffic arriving in another.
+ *
+ * See novox/hq 02-DECISIONS/0032-a-scenario-is-an-isolated-address-space.md
+ */
+
+import { spawn } from "node:child_process";
+
+/**
+ * How to invoke incus. Overridable because the socket is group-owned and a session that
+ * predates the group grant cannot reach it — which is a real thing that happens on the
+ * machine that just installed it.
+ */
+const INCUS = (process.env["MESH_LAB_INCUS"] ?? "incus").split(" ").filter(Boolean);
+
+export interface IncusResult {
+  stdout: string;
+  stderr: string;
+}
+
+export class IncusError extends Error {
+  readonly args: string[];
+  readonly stderr: string;
+  readonly exitCode: number | null;
+
+  constructor(args: string[], stderr: string, exitCode: number | null) {
+    const detail = stderr.trim() || (exitCode === null ? "timed out" : `exit ${exitCode}`);
+    super(`incus ${args.join(" ")} failed: ${detail}`);
+    this.name = "IncusError";
+    this.args = args;
+    this.stderr = stderr;
+    this.exitCode = exitCode;
+  }
+}
+
+/**
+ * Runs incus with **stdin closed**, which is not incidental.
+ *
+ * Several incus subcommands accept a YAML definition on stdin and, given a descriptor
+ * that is not a terminal, wait for one. Under a shell that never happens because stdin is
+ * a TTY; under a spawned process it hangs until the timeout kills it — and the timeout
+ * kill produces an empty stderr, so the failure arrives with no explanation at all.
+ *
+ * Found exactly that way: the first raise reported "failed: (no output)" on a command
+ * that worked perfectly when typed.
+ */
+export async function incus(args: string[], timeoutMs = 60_000): Promise<IncusResult> {
+  const [command, ...prefix] = INCUS;
+  if (!command) throw new Error("MESH_LAB_INCUS is empty");
+
+  return new Promise((resolve, reject) => {
+    const child = spawn(command, [...prefix, ...args], {
+      stdio: ["ignore", "pipe", "pipe"],
+    });
+
+    let stdout = "";
+    let stderr = "";
+    let timedOut = false;
+
+    const timer = setTimeout(() => {
+      timedOut = true;
+      child.kill("SIGKILL");
+    }, timeoutMs);
+
+    child.stdout.on("data", (chunk) => (stdout += chunk));
+    child.stderr.on("data", (chunk) => (stderr += chunk));
+
+    child.on("error", (err) => {
+      clearTimeout(timer);
+      reject(new IncusError(args, err.message, null));
+    });
+
+    child.on("close", (code) => {
+      clearTimeout(timer);
+      if (timedOut) {
+        reject(new IncusError(args, `timed out after ${timeoutMs}ms`, null));
+      } else if (code === 0) {
+        resolve({ stdout, stderr });
+      } else {
+        reject(new IncusError(args, stderr, code));
+      }
+    });
+  });
+}
+
+/**
+ * Like `incus`, but a failure is an answer rather than an exception. Returns the command's
+ * output, or `null` if it failed.
+ *
+ * **Never truthiness-test this.** Plenty of incus commands succeed with no output at all —
+ * `exec … true`, `network delete`, `start` — so an empty string means *worked and said
+ * nothing*, and `if (await incusOk(...))` reads that as failure. Use `succeeds()` when the
+ * question is whether it worked, and `!== null` when the output matters.
+ *
+ * This is the mesh's own recurring fault in miniature: absence and success made
+ * indistinguishable. It cost a raise that reported a machine unreachable while `incus exec`
+ * on that machine worked perfectly.
+ */
+export async function incusOk(args: string[], timeoutMs = 60_000): Promise<string | null> {
+  try {
+    return (await incus(args, timeoutMs)).stdout;
+  } catch {
+    return null;
+  }
+}
+
+/** Did the command work? For commands whose output is not the point. */
+export async function succeeds(args: string[], timeoutMs = 60_000): Promise<boolean> {
+  return (await incusOk(args, timeoutMs)) !== null;
+}
+
+export async function isReachable(): Promise<boolean> {
+  return succeeds(["info"], 15_000);
+}
+
+/** Storage drivers the daemon offers. It only advertises those whose tooling it found. */
+export async function supportedDrivers(): Promise<string[]> {
+  const info = (await incusOk(["info"], 15_000)) ?? "";
+  const block = info.split("storage_supported_drivers:")[1] ?? "";
+  return [...block.matchAll(/-\s+name:\s*(\w+)/g)].map((m) => m[1] ?? "");
+}
+
+export interface Pool {
+  name: string;
+  driver: string;
+}
+
+export async function pools(): Promise<Pool[]> {
+  const csv = (await incusOk(["storage", "list", "--format", "csv"])) ?? "";
+  return csv
+    .split("\n")
+    .filter(Boolean)
+    .map((line) => {
+      const [name = "", driver = ""] = line.split(",");
+      return { name, driver };
+    });
+}
+
+export async function networkExists(name: string): Promise<boolean> {
+  return succeeds(["network", "show", name], 15_000);
+}
+
+export async function instanceExists(name: string): Promise<boolean> {
+  return succeeds(["config", "show", name], 15_000);
+}
+
+export interface TaggedInstance {
+  name: string;
+  status: string;
+  instanceId: string;
+  machine: string;
+}
+
+/**
+ * Instances this lab owns, identified by the config keys set when they were created —
+ * never by parsing their names.
+ *
+ * Names were parsed at first, splitting on the last dash to separate machine from
+ * instance. That works until a machine is called something like `home-server`, at which
+ * point the instance id absorbs half the machine name and `destroy` silently finds
+ * nothing. Metadata is what the instance actually knows about itself.
+ */
+export async function taggedInstances(): Promise<TaggedInstance[]> {
+  const json = (await incusOk(["list", "--format", "json"], 30_000)) ?? "[]";
+  let parsed: unknown;
+  try {
+    parsed = JSON.parse(json);
+  } catch {
+    return [];
+  }
+  if (!Array.isArray(parsed)) return [];
+
+  const tagged: TaggedInstance[] = [];
+  for (const entry of parsed) {
+    const item = entry as { name?: string; status?: string; config?: Record<string, string> };
+    const instanceId = item.config?.["user.mesh-lab.instance"];
+    const machine = item.config?.["user.mesh-lab.machine"];
+    if (!instanceId || !machine || !item.name) continue;
+    tagged.push({ name: item.name, status: item.status ?? "", instanceId, machine });
+  }
+  return tagged;
+}
+
+export interface TaggedNetwork {
+  name: string;
+  instanceId: string;
+  segment: string;
+}
+
+export async function taggedNetworks(): Promise<TaggedNetwork[]> {
+  const json = (await incusOk(["network", "list", "--format", "json"], 30_000)) ?? "[]";
+  let parsed: unknown;
+  try {
+    parsed = JSON.parse(json);
+  } catch {
+    return [];
+  }
+  if (!Array.isArray(parsed)) return [];
+
+  const tagged: TaggedNetwork[] = [];
+  for (const entry of parsed) {
+    const item = entry as { name?: string; config?: Record<string, string> };
+    const instanceId = item.config?.["user.mesh-lab.instance"];
+    const segment = item.config?.["user.mesh-lab.segment"];
+    if (!instanceId || !segment || !item.name) continue;
+    tagged.push({ name: item.name, instanceId, segment });
+  }
+  return tagged;
+}