Scenario lifecycle: raise, exec, snapshot, restore, destroy
A declaration goes in and a disposable mesh comes out. Verified on a workstation, not asserted: two machines raised and addressed in 14.6s, snapshot 0.28s, restore-to-usable 11.6s, both families pinging with no loss, and the workstation with no route into any of it. The declaration layer implements the model in full — three positions a machine can be in, keyed on forwardability; gateways carrying the address the world sees them as; both address families; multi-homing; MTU; inter-segment policy. It is validated hard because the failures it prevents are silent: a private range on a public segment produces no error, the mesh simply never forms. Public segments are refused unless they use RFC 5737 or RFC 3849 space, and a range wider than the reserved block is refused too. 33 tests, all offline. The runtime implements less than the model, and refuses the difference. A scenario declaring gateways, published ports, policy, inbound deny or place is rejected at raise with every gap named. Raising it would produce a mesh that silently lacks what it declared, which is the fault this lab exists to catch — 04-ISSUES/003, where a firewall key is declared in five manifests and read by no code. Three bugs found by review and by running it, all of one family: The readiness check truthiness-tested incusOk's return. `exec … true` succeeds with EMPTY output, so every machine reported unreachable while incus exec on it worked perfectly. succeeds() now exists so the mistake is not available, and network delete had the same bug — it counted zero segments removed while removing them. list() split instance from machine on the last dash, so a machine called home-server absorbed half the instance id and destroy found nothing. Resources are now found by the metadata they carry, never by name. restore reported success in 0.79s while the machine's agent was still starting, so the next command failed. Both raise and restore now wait for usable and say how long that took — reporting the earlier number is transport reported as effect, which is the fault the lab is being built to find. Two incus behaviours worth recording. Its CLI reads a YAML definition from stdin when stdin is not a terminal, so a spawned command hangs until the timeout kills it and arrives with empty stderr — a failure with no explanation, on a command that works when typed. And it assigns a MAC at runtime without recording it in device config, so MACs are derived and set explicitly, which the guest needs anyway: it names interfaces by bus position, and matching by name configures the wrong one on a multi-homed machine. No build step; Node strips the types. The lifecycle has no unit tests because a fake hypervisor would assert that the fake behaves as expected, which is the shape of test this project exists to stop shipping.
This commit is contained in:
@@ -0,0 +1,100 @@
|
||||
/**
|
||||
* Put the declared addresses on the machines.
|
||||
*
|
||||
* The lab provides the underlay, and an address is underlay — it is what a hosting
|
||||
* provider or a home router would have given the machine before any of our software ran.
|
||||
* So the lab assigns it, and the mesh is left to build everything above it.
|
||||
*
|
||||
* Addresses are applied by matching on MAC rather than interface name. A guest names its
|
||||
* interfaces by bus position — `enp5s0`, not `eth0` — so the name the hypervisor uses and
|
||||
* the name the guest uses are different, and matching by name silently configures the
|
||||
* wrong interface on a multi-homed machine.
|
||||
*
|
||||
* See novox/hq 02-DECISIONS/0031-the-lab-provides-the-underlay.md
|
||||
*/
|
||||
|
||||
import type { Scenario } from "../declaration/types.ts";
|
||||
import { incus } from "../incus/client.ts";
|
||||
import { macFor } from "./names.ts";
|
||||
|
||||
export interface Wire {
|
||||
device: string;
|
||||
mac: string;
|
||||
addresses: string[];
|
||||
mtu: number | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* A systemd-networkd unit per interface. Static, because the declaration is the authority:
|
||||
* a scenario that let the hypervisor hand out addresses would be the lab supplying facts
|
||||
* the declaration is supposed to own.
|
||||
*/
|
||||
function networkUnit(wire: Wire): string {
|
||||
const lines = [
|
||||
"[Match]",
|
||||
`MACAddress=${wire.mac}`,
|
||||
"",
|
||||
"[Network]",
|
||||
...wire.addresses.map((address) => `Address=${address}`),
|
||||
// No gateway and no DNS on purpose. Routing off the segment is a gateway's job, and a
|
||||
// scenario that pre-wired it would be arranging what it is supposed to observe.
|
||||
"IPv6AcceptRA=no",
|
||||
];
|
||||
if (wire.mtu !== undefined) {
|
||||
lines.push("", "[Link]", `MTUBytes=${wire.mtu}`);
|
||||
}
|
||||
return lines.join("\n") + "\n";
|
||||
}
|
||||
|
||||
export async function applyAddresses(
|
||||
scenario: Scenario,
|
||||
instanceId: string,
|
||||
machineNames: Map<string, string>,
|
||||
log: (message: string) => void = () => {},
|
||||
): Promise<void> {
|
||||
for (const [machine, spec] of Object.entries(scenario.machines)) {
|
||||
if (spec.at === "detached") continue;
|
||||
const name = machineNames.get(machine);
|
||||
if (!name) continue;
|
||||
|
||||
const wires: Wire[] = [];
|
||||
for (const [index, attachment] of spec.at.entries()) {
|
||||
wires.push({
|
||||
device: `eth${index}`,
|
||||
mac: macFor(instanceId, machine, index),
|
||||
addresses: attachment.address.map((address) => withPrefix(scenario, attachment.segment, address)),
|
||||
mtu: scenario.segments[attachment.segment]?.mtu,
|
||||
});
|
||||
}
|
||||
|
||||
for (const [index, wire] of wires.entries()) {
|
||||
const unit = networkUnit(wire);
|
||||
await incus(
|
||||
["exec", name, "--", "sh", "-c",
|
||||
`mkdir -p /etc/systemd/network && cat > /etc/systemd/network/10-mlab-${index}.network <<'MLAB'\n${unit}MLAB`],
|
||||
30_000,
|
||||
);
|
||||
}
|
||||
|
||||
await incus(["exec", name, "--", "systemctl", "enable", "--now", "systemd-networkd"], 60_000);
|
||||
await incus(["exec", name, "--", "systemctl", "restart", "systemd-networkd"], 60_000);
|
||||
log(` addressed ${machine} (${wires.map((w) => w.addresses.join(",")).join(" | ")})`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* systemd-networkd wants a prefix length on the address. The declaration gives a bare
|
||||
* address and the segment gives the range, so the two are combined here rather than making
|
||||
* every scenario repeat the prefix on every machine.
|
||||
*/
|
||||
function withPrefix(scenario: Scenario, segment: string, address: string): string {
|
||||
const ranges = scenario.segments[segment]?.cidr ?? [];
|
||||
const wantV6 = address.includes(":");
|
||||
for (const range of ranges) {
|
||||
const slash = range.lastIndexOf("/");
|
||||
if (slash === -1) continue;
|
||||
const isV6 = range.slice(0, slash).includes(":");
|
||||
if (isV6 === wantV6) return `${address}${range.slice(slash)}`;
|
||||
}
|
||||
return address;
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
/**
|
||||
* How a scenario instance's resources are named.
|
||||
*
|
||||
* A declaration is a KIND; instances are many. Two instances of one declaration hold the
|
||||
* same addresses and must never meet, so every resource carries the instance id and
|
||||
* nothing is shared between them.
|
||||
*/
|
||||
|
||||
const PREFIX = "mlab";
|
||||
|
||||
/** Instance ids are short and sortable — the last one left standing has to be findable. */
|
||||
export function newInstanceId(scenario: string, now: Date): string {
|
||||
const stamp = now.toISOString().replace(/[-:T]/g, "").slice(2, 12);
|
||||
return `${scenario}-${stamp}`;
|
||||
}
|
||||
|
||||
/** incus network names are limited to 15 characters, so this hashes rather than truncates. */
|
||||
export function networkName(instanceId: string, segment: string): string {
|
||||
const digest = hash(`${instanceId}/${segment}`);
|
||||
return `${PREFIX}${digest}`;
|
||||
}
|
||||
|
||||
export function machineName(instanceId: string, machine: string): string {
|
||||
return `${PREFIX}-${instanceId}-${machine}`;
|
||||
}
|
||||
|
||||
export function instanceIdOf(machineName: string, machine: string): string | null {
|
||||
const suffix = `-${machine}`;
|
||||
if (!machineName.startsWith(`${PREFIX}-`) || !machineName.endsWith(suffix)) return null;
|
||||
return machineName.slice(PREFIX.length + 1, machineName.length - suffix.length);
|
||||
}
|
||||
|
||||
export function machinePrefix(instanceId: string): string {
|
||||
return `${PREFIX}-${instanceId}-`;
|
||||
}
|
||||
|
||||
/** FNV-1a, rendered base36. Short, stable, and collisions are a naming clash not a leak. */
|
||||
function hash(text: string): string {
|
||||
let h = 0x811c9dc5;
|
||||
for (let i = 0; i < text.length; i++) {
|
||||
h ^= text.charCodeAt(i);
|
||||
h = Math.imul(h, 0x01000193) >>> 0;
|
||||
}
|
||||
return h.toString(36).padStart(7, "0").slice(0, 7);
|
||||
}
|
||||
|
||||
/**
|
||||
* A deterministic MAC for a machine's Nth interface, in the locally-administered range.
|
||||
*
|
||||
* Set explicitly at creation rather than read back afterwards: incus assigns a MAC at
|
||||
* runtime and does not record it in the device config, so querying returns nothing. A
|
||||
* derived address is also stable across raises of the same instance, which makes an
|
||||
* in-guest match on it reproducible.
|
||||
*/
|
||||
export function macFor(instanceId: string, machine: string, index: number): string {
|
||||
const digest = hash(`${instanceId}/${machine}/${index}`);
|
||||
const octets: string[] = [];
|
||||
let value = 0;
|
||||
for (let i = 0; i < digest.length; i++) value = (value * 31 + digest.charCodeAt(i)) >>> 0;
|
||||
// 02 marks it locally administered, which is what a made-up address is supposed to say.
|
||||
octets.push("02");
|
||||
for (let i = 0; i < 5; i++) {
|
||||
octets.push(((value >>> (i * 5)) & 0xff).toString(16).padStart(2, "0"));
|
||||
}
|
||||
return octets.join(":");
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
/**
|
||||
* What happens to a scenario once it is raised: inspect it, run things in it, capture and
|
||||
* return it to a state, and tear it down.
|
||||
*
|
||||
* Snapshots are WHOLE-SCENARIO. Per-machine would be cheaper and wrong: the mesh keeps
|
||||
* state that spans nodes, so restoring one machine to an earlier moment while its peers
|
||||
* move on produces a mesh that has never existed and could not. Faults found there would
|
||||
* be artefacts of the lab.
|
||||
*/
|
||||
|
||||
import { incus, incusOk, succeeds, taggedInstances, taggedNetworks } from "../incus/client.ts";
|
||||
import { machineName } from "./names.ts";
|
||||
import { waitUntilAllUsable } from "./ready.ts";
|
||||
|
||||
export interface Instance {
|
||||
instanceId: string;
|
||||
machines: { name: string; machine: string; status: string }[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Every scenario instance the daemon currently holds, found by the metadata each resource
|
||||
* carries rather than by parsing names — a machine called `home-server` would otherwise
|
||||
* be split in the wrong place and its instance would appear not to exist.
|
||||
*/
|
||||
export async function list(): Promise<Instance[]> {
|
||||
const byInstance = new Map<string, Instance["machines"]>();
|
||||
for (const item of await taggedInstances()) {
|
||||
const entry = byInstance.get(item.instanceId) ?? [];
|
||||
entry.push({ name: item.name, machine: item.machine, status: item.status });
|
||||
byInstance.set(item.instanceId, entry);
|
||||
}
|
||||
return [...byInstance.entries()]
|
||||
.map(([instanceId, machines]) => ({
|
||||
instanceId,
|
||||
machines: machines.sort((a, b) => a.machine.localeCompare(b.machine)),
|
||||
}))
|
||||
.sort((a, b) => a.instanceId.localeCompare(b.instanceId));
|
||||
}
|
||||
|
||||
async function machinesOf(instanceId: string): Promise<string[]> {
|
||||
const found = (await list()).find((i) => i.instanceId === instanceId);
|
||||
if (!found) throw new Error(`no scenario instance '${instanceId}'`);
|
||||
return found.machines.map((m) => m.name);
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a command on a machine, through incus rather than over IP.
|
||||
*
|
||||
* A reachability question is therefore asked from INSIDE: *can this machine reach that
|
||||
* one* is exec on the first, testing the second. The workstation is not on the scenario's
|
||||
* network and its opinion would be a different question with a similar-looking answer.
|
||||
*/
|
||||
export async function exec(
|
||||
instanceId: string,
|
||||
machine: string,
|
||||
command: string[],
|
||||
): Promise<{ stdout: string; stderr: string }> {
|
||||
const found = (await taggedInstances()).find(
|
||||
(i) => i.instanceId === instanceId && i.machine === machine,
|
||||
);
|
||||
const name = found?.name ?? machineName(instanceId, machine);
|
||||
return incus(["exec", name, "--", ...command], 120_000);
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture the whole scenario as one state. Every machine, one name.
|
||||
*
|
||||
* Machines are snapshotted while running, so what is captured is the disk and not memory —
|
||||
* crash-consistent rather than a paused mesh. Whether a mesh restored that way is coherent
|
||||
* is an open question in the design, not something this silently assumes away.
|
||||
*/
|
||||
export async function snapshot(instanceId: string, label: string): Promise<number> {
|
||||
const machines = await machinesOf(instanceId);
|
||||
const started = Date.now();
|
||||
for (const name of machines) {
|
||||
await incus(["snapshot", "create", name, label], 300_000);
|
||||
}
|
||||
return (Date.now() - started) / 1000;
|
||||
}
|
||||
|
||||
export interface RestoreResult {
|
||||
/** How long the restore itself took. */
|
||||
restoreSeconds: number;
|
||||
/** How long until the scenario was usable again — the number that matters. */
|
||||
usableSeconds: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Return the whole scenario to a state. Restoring a subset would produce a mesh that never
|
||||
* was, so this is all-or-nothing.
|
||||
*
|
||||
* Restoring a virtual machine replaces its disk and the machine comes back up, so it
|
||||
* reports RUNNING while its agent is still starting — measured, the restore call returns
|
||||
* in 0.79s and the very next command fails. Reporting that as "restored" would be
|
||||
* transport reported as effect, so this waits for usable and returns both numbers.
|
||||
*/
|
||||
export async function restore(
|
||||
instanceId: string,
|
||||
label: string,
|
||||
readyTimeoutSeconds = 180,
|
||||
log: (message: string) => void = () => {},
|
||||
): Promise<RestoreResult> {
|
||||
const machines = await machinesOf(instanceId);
|
||||
const started = Date.now();
|
||||
for (const name of machines) {
|
||||
await incus(["snapshot", "restore", name, label], 300_000);
|
||||
}
|
||||
const restoreSeconds = (Date.now() - started) / 1000;
|
||||
|
||||
for (const name of machines) {
|
||||
await succeeds(["start", name], 60_000);
|
||||
}
|
||||
await waitUntilAllUsable(machines, readyTimeoutSeconds, log);
|
||||
|
||||
return { restoreSeconds, usableSeconds: (Date.now() - started) / 1000 };
|
||||
}
|
||||
|
||||
export async function snapshots(instanceId: string): Promise<string[]> {
|
||||
const machines = await machinesOf(instanceId);
|
||||
const first = machines[0];
|
||||
if (!first) return [];
|
||||
const csv = (await incusOk(["snapshot", "list", first, "--format", "csv"])) ?? "";
|
||||
return csv
|
||||
.split("\n")
|
||||
.filter(Boolean)
|
||||
.map((line) => line.split(",")[0] ?? "")
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
/**
|
||||
* Tear the instance down: machines first, then the links they were on.
|
||||
*
|
||||
* Networks are removed last and only if empty — a link still carrying an interface
|
||||
* cannot be deleted, and forcing it would leave the daemon with a reference to something
|
||||
* gone.
|
||||
*/
|
||||
export async function destroy(instanceId: string): Promise<{ machines: number; networks: number }> {
|
||||
const machines = (await taggedInstances()).filter((i) => i.instanceId === instanceId);
|
||||
for (const machine of machines) {
|
||||
await succeeds(["delete", "--force", machine.name], 300_000);
|
||||
}
|
||||
|
||||
// Links go last and only once nothing is attached: a network still carrying an
|
||||
// interface cannot be deleted, and forcing it would leave a dangling reference.
|
||||
let networks = 0;
|
||||
for (const network of await taggedNetworks()) {
|
||||
if (network.instanceId !== instanceId) continue;
|
||||
// `network delete` also succeeds silently — counted with succeeds(), not truthiness.
|
||||
if (await succeeds(["network", "delete", network.name], 30_000)) networks++;
|
||||
}
|
||||
|
||||
return { machines: machines.length, networks };
|
||||
}
|
||||
@@ -0,0 +1,202 @@
|
||||
/**
|
||||
* Materialise a declaration into a running scenario instance.
|
||||
*
|
||||
* Two rules from the design shape everything here.
|
||||
*
|
||||
* `raise` waits for the machines to be USABLE, not for the calls to return. Measured on a
|
||||
* workstation those are 3.4s and 14.3s apart, and reporting the earlier number would be
|
||||
* the mesh's own recurring failure — transport reported as effect.
|
||||
*
|
||||
* A failed raise LEAVES THE WRECKAGE. Tearing down on failure destroys the only evidence
|
||||
* of what went wrong, and a scenario that failed to raise is more interesting than one
|
||||
* that succeeded.
|
||||
*
|
||||
* See novox/hq 03-DESIGN/01-to-be/03-scenario-lifecycle.md
|
||||
*/
|
||||
|
||||
import type { Scenario } from "../declaration/types.ts";
|
||||
import { incus, incusOk, succeeds, pools, supportedDrivers } from "../incus/client.ts";
|
||||
import { machineName, macFor, networkName, newInstanceId } from "./names.ts";
|
||||
import { waitUntilAllUsable } from "./ready.ts";
|
||||
import { applyAddresses } from "./address.ts";
|
||||
import { assertSupported } from "./supported.ts";
|
||||
|
||||
/** Drivers whose snapshots are copy-on-write. On `dir` a snapshot is a full copy. */
|
||||
const COW_DRIVERS = ["btrfs", "zfs"];
|
||||
|
||||
export interface RaiseOptions {
|
||||
/** Base image for machines. */
|
||||
image?: string;
|
||||
/** Reuse an existing instance id rather than minting one — makes raise convergent. */
|
||||
instanceId?: string;
|
||||
/** Seconds to wait for each machine to become usable. */
|
||||
readyTimeoutSeconds?: number;
|
||||
onProgress?: (message: string) => void;
|
||||
}
|
||||
|
||||
export interface RaisedScenario {
|
||||
instanceId: string;
|
||||
scenario: string;
|
||||
machines: string[];
|
||||
networks: string[];
|
||||
pool: string;
|
||||
}
|
||||
|
||||
export class RaiseError extends Error {
|
||||
readonly instanceId: string;
|
||||
readonly step: string;
|
||||
|
||||
constructor(instanceId: string, step: string, cause: unknown) {
|
||||
const detail = cause instanceof Error ? cause.message : String(cause);
|
||||
super(
|
||||
`raise failed at '${step}': ${detail}\n` +
|
||||
`The instance '${instanceId}' has been LEFT STANDING for inspection. ` +
|
||||
`Destroy it with: mesh-lab destroy ${instanceId}`,
|
||||
);
|
||||
this.name = "RaiseError";
|
||||
this.instanceId = instanceId;
|
||||
this.step = step;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick a pool that can snapshot cheaply, and say so loudly when there is not one.
|
||||
*
|
||||
* Measured: a `dir` snapshot of a 1.5 GB machine takes 9.9s and a full 1.6 GB, with a
|
||||
* second snapshot unfinished after two minutes. On copy-on-write it is 0.13s and costs
|
||||
* the delta. Restoring is the operation the inner loop repeats most, so a `dir` pool does
|
||||
* not make the lab slow — it makes it unused.
|
||||
*/
|
||||
async function choosePool(log: (m: string) => void): Promise<string> {
|
||||
const available = await pools();
|
||||
const cow = available.find((p) => COW_DRIVERS.includes(p.driver));
|
||||
if (cow) return cow.name;
|
||||
|
||||
const drivers = await supportedDrivers();
|
||||
const possible = drivers.filter((d) => COW_DRIVERS.includes(d));
|
||||
log(
|
||||
possible.length > 0
|
||||
? `WARNING: no copy-on-write pool exists, though the daemon offers ${possible.join("/")}. ` +
|
||||
`Snapshots will be full copies — roughly 76x slower, and the inner loop unusable.`
|
||||
: `WARNING: the daemon offers no copy-on-write driver. Snapshots will be full copies — ` +
|
||||
`roughly 76x slower, and the inner loop unusable. Install btrfs tooling and restart it.`,
|
||||
);
|
||||
const fallback = available[0];
|
||||
if (!fallback) throw new Error("no storage pool exists at all");
|
||||
return fallback.name;
|
||||
}
|
||||
|
||||
/**
|
||||
* One isolated link per segment. Nothing joins them to anything outside the instance, and
|
||||
* incus is told not to hand out addresses: a scenario declares the underlay, and letting
|
||||
* a hypervisor's DHCP assign addresses would be the lab supplying facts the declaration
|
||||
* is supposed to own.
|
||||
*/
|
||||
async function createNetwork(instanceId: string, segment: string): Promise<string> {
|
||||
const name = networkName(instanceId, segment);
|
||||
if (await succeeds(["network", "show", name], 15_000)) return name;
|
||||
await incus([
|
||||
"network", "create", name,
|
||||
"ipv4.address=none",
|
||||
"ipv6.address=none",
|
||||
"ipv4.nat=false",
|
||||
"ipv6.nat=false",
|
||||
`user.mesh-lab.instance=${instanceId}`,
|
||||
`user.mesh-lab.segment=${segment}`,
|
||||
]);
|
||||
return name;
|
||||
}
|
||||
|
||||
async function createMachine(
|
||||
instanceId: string,
|
||||
machine: string,
|
||||
attachments: { segment: string }[],
|
||||
image: string,
|
||||
pool: string,
|
||||
): Promise<string> {
|
||||
const name = machineName(instanceId, machine);
|
||||
if (await succeeds(["config", "show", name], 15_000)) return name;
|
||||
|
||||
const args = [
|
||||
"init", image, name,
|
||||
"--vm",
|
||||
"-s", pool,
|
||||
// Arch images refuse to boot under secureboot with the shipped keys. Discovered by
|
||||
// the first launch failing with exactly that message.
|
||||
"-c", "security.secureboot=false",
|
||||
"-c", "limits.memory=1GiB",
|
||||
"-c", "limits.cpu=2",
|
||||
"-c", `user.mesh-lab.instance=${instanceId}`,
|
||||
"-c", `user.mesh-lab.machine=${machine}`,
|
||||
];
|
||||
await incus(args, 300_000);
|
||||
|
||||
// eth0 comes from the profile and points at the wrong network, so every attachment is
|
||||
// explicit. A machine on no segment gets no interface at all — that is what detached is.
|
||||
await succeeds(["config", "device", "remove", name, "eth0"], 15_000);
|
||||
for (const [index, attachment] of attachments.entries()) {
|
||||
await incus([
|
||||
"config", "device", "add", name, `eth${index}`, "nic",
|
||||
"nictype=bridged",
|
||||
`parent=${networkName(instanceId, attachment.segment)}`,
|
||||
// Explicit, because incus assigns one at runtime without recording it in the device
|
||||
// config — so reading it back returns nothing, and the guest has no stable handle.
|
||||
`hwaddr=${macFor(instanceId, machine, index)}`,
|
||||
]);
|
||||
}
|
||||
return name;
|
||||
}
|
||||
|
||||
export async function raise(
|
||||
scenario: Scenario,
|
||||
options: RaiseOptions = {},
|
||||
): Promise<RaisedScenario> {
|
||||
const log = options.onProgress ?? (() => {});
|
||||
const image = options.image ?? "images:archlinux/current";
|
||||
const readyTimeout = options.readyTimeoutSeconds ?? 180;
|
||||
const instanceId = options.instanceId ?? newInstanceId(scenario.scenario, new Date());
|
||||
|
||||
// Refuse before spending a minute raising something that would silently lack half of
|
||||
// what it declares. Deliberately outside the try: this is not a raise failure, nothing
|
||||
// has been created, and there is no wreckage to leave standing.
|
||||
assertSupported(scenario);
|
||||
|
||||
let step = "choosing a storage pool";
|
||||
try {
|
||||
const pool = await choosePool(log);
|
||||
log(`instance ${instanceId} pool ${pool}`);
|
||||
|
||||
step = "creating segments";
|
||||
const networks: string[] = [];
|
||||
for (const segment of Object.keys(scenario.segments)) {
|
||||
networks.push(await createNetwork(instanceId, segment));
|
||||
log(` segment ${segment}`);
|
||||
}
|
||||
|
||||
step = "creating machines";
|
||||
const created: string[] = [];
|
||||
const byMachine = new Map<string, string>();
|
||||
for (const [machine, spec] of Object.entries(scenario.machines)) {
|
||||
const attachments = spec.at === "detached" ? [] : spec.at;
|
||||
const name = await createMachine(instanceId, machine, attachments, image, pool);
|
||||
created.push(name);
|
||||
byMachine.set(machine, name);
|
||||
log(` machine ${machine}${spec.at === "detached" ? " (detached)" : ""}`);
|
||||
}
|
||||
|
||||
step = "starting machines";
|
||||
for (const name of created) {
|
||||
await succeeds(["start", name], 60_000);
|
||||
}
|
||||
|
||||
step = "waiting for machines to become usable";
|
||||
await waitUntilAllUsable(created, readyTimeout, log);
|
||||
|
||||
step = "applying declared addresses";
|
||||
await applyAddresses(scenario, instanceId, byMachine, log);
|
||||
|
||||
return { instanceId, scenario: scenario.scenario, machines: created, networks, pool };
|
||||
} catch (cause) {
|
||||
throw new RaiseError(instanceId, step, cause);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
/**
|
||||
* "Usable" means a command runs on the machine. Anything weaker is transport reported as
|
||||
* effect — the mesh's own recurring fault, and one this lab exists to catch rather than
|
||||
* commit.
|
||||
*
|
||||
* Two measurements make the case. Raising: the launch call returns in 3.4s and the machine
|
||||
* is usable at 14.3s. Restoring: the call returns in 0.79s and the machine is RUNNING
|
||||
* immediately — with its agent still starting, so the very next command fails.
|
||||
*
|
||||
* Both verbs therefore wait for the same thing, using the same code.
|
||||
*/
|
||||
|
||||
import { succeeds } from "../incus/client.ts";
|
||||
|
||||
export interface ReadyResult {
|
||||
name: string;
|
||||
seconds: number;
|
||||
}
|
||||
|
||||
export async function waitUntilUsable(
|
||||
name: string,
|
||||
timeoutSeconds: number,
|
||||
log: (message: string) => void = () => {},
|
||||
): Promise<ReadyResult> {
|
||||
const started = Date.now();
|
||||
const deadline = started + timeoutSeconds * 1000;
|
||||
|
||||
while (Date.now() < deadline) {
|
||||
// `exec … true` succeeds with EMPTY output, so this asks whether it worked rather than
|
||||
// what it said. Truthiness-testing the output reported every machine as unreachable
|
||||
// while `incus exec` on it worked perfectly.
|
||||
if (await succeeds(["exec", name, "--", "true"], 10_000)) {
|
||||
const seconds = (Date.now() - started) / 1000;
|
||||
log(` ${name} usable after ${seconds.toFixed(1)}s`);
|
||||
return { name, seconds };
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 1000));
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
`${name} did not become usable within ${timeoutSeconds}s — it may be running but ` +
|
||||
`unreachable, which is not the same as ready`,
|
||||
);
|
||||
}
|
||||
|
||||
export async function waitUntilAllUsable(
|
||||
names: string[],
|
||||
timeoutSeconds: number,
|
||||
log: (message: string) => void = () => {},
|
||||
): Promise<number> {
|
||||
const started = Date.now();
|
||||
// Concurrently: a scenario's machines boot independently, and waiting for them in turn
|
||||
// would make a four-machine scenario four boots long instead of one.
|
||||
await Promise.all(names.map((name) => waitUntilUsable(name, timeoutSeconds, log)));
|
||||
return (Date.now() - started) / 1000;
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
/**
|
||||
* What this lab can actually materialise, and a refusal for everything else.
|
||||
*
|
||||
* A declaration the runtime silently ignores is the worst thing this project could ship.
|
||||
* The mesh already has that fault catalogued: a firewall key declared in five manifests
|
||||
* and read by no code, so a manifest appears to restrict a port and restricts nothing
|
||||
* (novox/hq 04-ISSUES/003). A scenario that declares a gateway and raises without one
|
||||
* would be the same fault, in the tool built to catch it.
|
||||
*
|
||||
* So unimplemented parts of the model are refused loudly at raise time rather than
|
||||
* ignored. The declaration schema deliberately runs ahead of the runtime — it is the
|
||||
* design's shape, and validating against it is useful before any of it can be raised —
|
||||
* but the gap between the two has to be visible.
|
||||
*/
|
||||
|
||||
import type { Scenario } from "../declaration/types.ts";
|
||||
|
||||
export class UnsupportedError extends Error {
|
||||
readonly missing: string[];
|
||||
|
||||
constructor(missing: string[]) {
|
||||
super(
|
||||
`this scenario declares things the lab cannot yet materialise:\n - ${missing.join("\n - ")}\n\n` +
|
||||
`Raising it would produce a mesh that silently lacks them, which is the exact fault ` +
|
||||
`this lab exists to catch. Validation accepts them because the declaration model is ` +
|
||||
`complete; the runtime is not.`,
|
||||
);
|
||||
this.name = "UnsupportedError";
|
||||
this.missing = missing;
|
||||
}
|
||||
}
|
||||
|
||||
export function assertSupported(scenario: Scenario): void {
|
||||
const missing: string[] = [];
|
||||
|
||||
const withGateways = Object.entries(scenario.segments)
|
||||
.filter(([, segment]) => segment.gateway)
|
||||
.map(([name]) => name);
|
||||
if (withGateways.length > 0) {
|
||||
missing.push(
|
||||
`gateways (segments: ${withGateways.join(", ")}) — no router is materialised, so ` +
|
||||
`nothing routes between segments and NAT does not exist`,
|
||||
);
|
||||
}
|
||||
|
||||
const published = Object.entries(scenario.machines)
|
||||
.filter(([, machine]) => machine.published?.length)
|
||||
.map(([name]) => name);
|
||||
if (published.length > 0) {
|
||||
missing.push(
|
||||
`published ports (machines: ${published.join(", ")}) — requires a gateway to forward through`,
|
||||
);
|
||||
}
|
||||
|
||||
if (scenario.policy?.length) {
|
||||
missing.push("policy between segments — requires a gateway to enforce it");
|
||||
}
|
||||
|
||||
const inbound = Object.entries(scenario.machines)
|
||||
.filter(([, machine]) => machine.inbound === "deny")
|
||||
.map(([name]) => name);
|
||||
if (inbound.length > 0) {
|
||||
missing.push(
|
||||
`inbound: deny (machines: ${inbound.join(", ")}) — no host firewall is configured, so ` +
|
||||
`these machines would accept traffic the scenario says they refuse`,
|
||||
);
|
||||
}
|
||||
|
||||
if (scenario.place && Object.keys(scenario.place).length > 0) {
|
||||
missing.push(
|
||||
"place — nothing is placed inside the machines yet; they are raised bare",
|
||||
);
|
||||
}
|
||||
|
||||
if (missing.length > 0) throw new UnsupportedError(missing);
|
||||
}
|
||||
Reference in New Issue
Block a user