place: the host — the lab acquires a consumer
The lab raised an underlay and put nothing on it: correct, and useless, because the thing it exists to test did not exist. Tier 0 now does, so `place: [host]` works and a raised scenario finally contains something. The refusal narrows rather than disappearing. A scenario placing a host and a substrate is told which half is missing, by name — not that `place:` is unsupported when half of it now works. Placement reads back rather than assuming. A file arriving is not a host working, so the binary is run before it is trusted to answer questions, and what it reports is read from the machine (ADR 0035). The binary comes from an explicit path, because the declaration design leaves where artifacts come from open and a search would harden into the answer by accident. The integration test that matters is the one asserting the host reports the MACHINE and not the workstation that placed it. A raised VM and this workstation differ in every capability — root versus uid 1000, a clean init versus a degraded one, no docker versus docker, no wireguard versus wg0 — so a host reporting the wrong machine is obvious here and invisible anywhere else. And the placed host independently confirms ADR 0031: overlay absent on a freshly raised machine. The underlay suite already asserted that by looking for wireguard interfaces; this is a second witness rather than the same check twice. Two tests failed the moment placement worked, which is what they were for. They defended "there is nothing to place yet" while that was true; the decision changed, so they change with it rather than being deleted. Gate: 75 unit, 20 integration.
This commit is contained in:
@@ -116,16 +116,20 @@ than ignored:
|
||||
| `policy:` between segments | **works**, asymmetric |
|
||||
| `inbound: deny` | **works** — host firewall, read back after applying |
|
||||
| several public networks, routed not bridged | **works** — a transit router, never a shared bridge |
|
||||
| `place:` | **refused at raise** — the node host it would place does not exist yet |
|
||||
| `place: [host]` | **works** — tier 0 is placed and asked what the machine is |
|
||||
| `place:` anything above tier 0 | **refused, by name** — those tiers do not exist yet |
|
||||
|
||||
`raise` refuses a scenario declaring anything in the lower half, naming every gap. It does not
|
||||
raise a mesh that silently lacks what it declared — that is the fault this lab exists to catch
|
||||
(`novox/hq` 04-ISSUES/003: a firewall key declared in five manifests and read by no code, so a
|
||||
manifest appears to restrict a port and restricts nothing).
|
||||
|
||||
No scenario in `scenarios/` declares `place:` yet, so all of them raise. What they raise is
|
||||
an underlay holding empty machines — correct, and not yet useful for anything, because the
|
||||
node host that would be placed on them does not exist.
|
||||
`bootstrap-single.yml` places the host. The rest raise an underlay and put nothing on it,
|
||||
which is still correct for what they test.
|
||||
|
||||
Placing needs a built host binary — set `MESH_LAB_HOST_BINARY` to one. It is an explicit path
|
||||
rather than a search on purpose: the declaration design leaves *where `place:` gets its
|
||||
artifacts from* open, and guessing would harden into the answer by accident.
|
||||
|
||||
## Measured on a workstation
|
||||
|
||||
|
||||
@@ -14,7 +14,10 @@ machines:
|
||||
at: { segment: hosting, address: [192.0.2.10, "2001:db8:a::10"] }
|
||||
inbound: allow
|
||||
|
||||
# No `place:` yet. The node host it would place does not exist — this lab is being built to
|
||||
place:
|
||||
all: [host]
|
||||
|
||||
# The host is placed. Everything above tier 0 is still refused by name — this lab is built to
|
||||
# develop it. Declaring it anyway would make the scenario unraisable, and correctly so: the
|
||||
# lab refuses declarations it cannot materialise rather than raising a mesh that silently
|
||||
# lacks them.
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
/**
|
||||
* `place:` — putting something inside the machines.
|
||||
*
|
||||
* Until now the lab raised an underlay and put nothing on it: correct, and useless, because
|
||||
* the thing it exists to test did not exist (novox/hq 03-DESIGN/00-as-is/11-the-lab.md). Tier
|
||||
* 0 now does, so this is the seam where the lab acquires a consumer.
|
||||
*
|
||||
* Only `host` is placeable. Everything else in the placement vocabulary — the substrate, a
|
||||
* control plane, a forge — is still refused by name rather than ignored, because a scenario
|
||||
* that declares something and raises without it is the fault this lab was built to catch
|
||||
* (novox/hq 04-ISSUES/003).
|
||||
*/
|
||||
|
||||
import type { Scenario } from "../declaration/types.ts";
|
||||
import { incus, incusOk } from "../incus/client.ts";
|
||||
|
||||
/** What this stage can put inside a machine. */
|
||||
export const PLACEABLE = ["host"] as const;
|
||||
export type Placeable = (typeof PLACEABLE)[number];
|
||||
|
||||
/** Where the host binary lives on a machine once placed. */
|
||||
export const HOST_PATH = "/usr/local/bin/mesh-host";
|
||||
|
||||
export interface Placement {
|
||||
machine: string;
|
||||
artifacts: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve `place:` to one list per machine.
|
||||
*
|
||||
* `all:` applies to every machine; a per-machine entry **overrides** it rather than adding to
|
||||
* it, which is what the declaration design says and is worth being exact about — a scenario
|
||||
* naming one artifact for one machine gets that artifact and not that artifact plus the rest.
|
||||
*/
|
||||
export function planPlacements(scenario: Scenario): Placement[] {
|
||||
const place = scenario.place;
|
||||
if (!place) return [];
|
||||
|
||||
const all = place.all ?? [];
|
||||
return Object.keys(scenario.machines).map((machine) => {
|
||||
const own = place[machine];
|
||||
return { machine, artifacts: own ?? all };
|
||||
}).filter((p) => p.artifacts.length > 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* The host binary to place, from the environment.
|
||||
*
|
||||
* Deliberately an explicit path rather than a search. The declaration design leaves *where
|
||||
* `place:` gets its artifacts from* open — before the mesh is self-hosting they come from
|
||||
* outside, afterwards from the mesh — and it suggests a named source rather than a path. This
|
||||
* is neither: it is the smallest thing that works while that stays undecided, and it refuses
|
||||
* loudly rather than guessing, so nothing here hardens into the answer by accident.
|
||||
*/
|
||||
export function hostBinaryPath(): string | null {
|
||||
return process.env["MESH_LAB_HOST_BINARY"] ?? null;
|
||||
}
|
||||
|
||||
export class PlacementError extends Error {
|
||||
readonly machine: string;
|
||||
|
||||
constructor(machine: string, message: string) {
|
||||
super(message);
|
||||
this.name = "PlacementError";
|
||||
this.machine = machine;
|
||||
}
|
||||
}
|
||||
|
||||
export interface PlacedHost {
|
||||
machine: string;
|
||||
/** What the host reported about the machine, read back from it. */
|
||||
profile: unknown;
|
||||
version: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Put the host on a machine and ask it what the machine is.
|
||||
*
|
||||
* The result is read back from the running binary, never assumed from the fact that the copy
|
||||
* succeeded (novox/hq ADR 0035). A file arriving is not a host working, which is the same
|
||||
* distinction the host itself makes about installed packages.
|
||||
*/
|
||||
export async function placeHost(
|
||||
instanceName: string,
|
||||
machine: string,
|
||||
binary: string,
|
||||
log: (message: string) => void = () => {},
|
||||
): Promise<PlacedHost> {
|
||||
await incus(["file", "push", binary, `${instanceName}${HOST_PATH}`, "--mode", "0755"], 180_000);
|
||||
|
||||
// Read back that it is there and executable before trusting it to answer questions.
|
||||
const version = (await incusOk(["exec", instanceName, "--", HOST_PATH, "version"], 60_000))?.trim();
|
||||
if (!version) {
|
||||
throw new PlacementError(
|
||||
machine,
|
||||
`the host binary was copied to ${machine} but does not run there. A file arriving is ` +
|
||||
`not a host working.`,
|
||||
);
|
||||
}
|
||||
|
||||
const reported = await incusOk(
|
||||
["exec", instanceName, "--", HOST_PATH, "profile", "--json"],
|
||||
120_000,
|
||||
);
|
||||
if (!reported) {
|
||||
throw new PlacementError(
|
||||
machine,
|
||||
`the host runs on ${machine} (${version}) but reported no profile. A host that cannot ` +
|
||||
`say what a machine is cannot be asked to change it.`,
|
||||
);
|
||||
}
|
||||
|
||||
let profile: unknown;
|
||||
try {
|
||||
profile = JSON.parse(reported);
|
||||
} catch (err) {
|
||||
throw new PlacementError(
|
||||
machine,
|
||||
`the host on ${machine} reported something that is not a profile: ${(err as Error).message}`,
|
||||
);
|
||||
}
|
||||
|
||||
log(` placed the host on ${machine} (${version})`);
|
||||
return { machine, profile, version };
|
||||
}
|
||||
|
||||
/** Place everything a scenario declares. */
|
||||
export async function applyPlacements(
|
||||
scenario: Scenario,
|
||||
machineNames: Map<string, string>,
|
||||
log: (message: string) => void = () => {},
|
||||
): Promise<PlacedHost[]> {
|
||||
const placements = planPlacements(scenario);
|
||||
if (placements.length === 0) return [];
|
||||
|
||||
const binary = hostBinaryPath();
|
||||
if (!binary) {
|
||||
throw new Error(
|
||||
`this scenario places the host, and no host binary was given. Set ` +
|
||||
`MESH_LAB_HOST_BINARY to a built mesh-host. Guessing at a path would place ` +
|
||||
`whatever happened to be there.`,
|
||||
);
|
||||
}
|
||||
|
||||
const placed: PlacedHost[] = [];
|
||||
for (const { machine, artifacts } of placements) {
|
||||
const name = machineNames.get(machine);
|
||||
if (!name) continue;
|
||||
for (const artifact of artifacts) {
|
||||
if (artifact !== "host") continue; // refused earlier; belt and braces
|
||||
placed.push(await placeHost(name, machine, binary, log));
|
||||
}
|
||||
}
|
||||
return placed;
|
||||
}
|
||||
@@ -22,6 +22,7 @@ import { applyAddresses, applyDefaultRoutes } from "./address.ts";
|
||||
import { assertSupported } from "./supported.ts";
|
||||
import { planRouters, raiseRouters, raiseTransit } from "./router.ts";
|
||||
import { applyHostFirewalls } from "./firewall.ts";
|
||||
import { applyPlacements } from "./place.ts";
|
||||
|
||||
/** Drivers whose snapshots are copy-on-write. On `dir` a snapshot is a full copy. */
|
||||
const COW_DRIVERS = ["btrfs", "zfs"];
|
||||
@@ -224,6 +225,11 @@ export async function raise(
|
||||
step = "applying host firewalls";
|
||||
await applyHostFirewalls(scenario, byMachine, log);
|
||||
|
||||
// Last, and only once the underlay is real. Placing before the machines can reach each
|
||||
// other would test the host against a network the scenario does not describe.
|
||||
step = "placing";
|
||||
await applyPlacements(scenario, byMachine, log);
|
||||
|
||||
return {
|
||||
instanceId,
|
||||
scenario: scenario.scenario,
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
*/
|
||||
|
||||
import type { Scenario } from "../declaration/types.ts";
|
||||
import { PLACEABLE, planPlacements } from "./place.ts";
|
||||
|
||||
export class UnsupportedError extends Error {
|
||||
readonly missing: string[];
|
||||
@@ -33,9 +34,19 @@ export class UnsupportedError extends Error {
|
||||
export function assertSupported(scenario: Scenario): void {
|
||||
const missing: string[] = [];
|
||||
|
||||
if (scenario.place && Object.keys(scenario.place).length > 0) {
|
||||
// `place: [host]` works. Everything else in the vocabulary is named individually rather
|
||||
// than refused as a whole, so a scenario that places a host and a substrate is told exactly
|
||||
// which half the lab cannot do.
|
||||
const unplaceable = new Set<string>();
|
||||
for (const { artifacts } of planPlacements(scenario)) {
|
||||
for (const artifact of artifacts) {
|
||||
if (!(PLACEABLE as readonly string[]).includes(artifact)) unplaceable.add(artifact);
|
||||
}
|
||||
}
|
||||
for (const artifact of [...unplaceable].sort()) {
|
||||
missing.push(
|
||||
"place — nothing is placed inside the machines yet; they are raised bare",
|
||||
`place: ${artifact} — only ${PLACEABLE.join(", ")} can be placed; the tiers above ` +
|
||||
`tier 0 do not exist yet`,
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
/**
|
||||
* The lab, placing tier 0 inside a machine it raised.
|
||||
*
|
||||
* This is the seam that ends the lab being infrastructure with no consumer
|
||||
* (novox/hq 03-DESIGN/00-as-is/11-the-lab.md). It needs a built host binary; without one it
|
||||
* skips with a reason rather than passing having checked nothing.
|
||||
*/
|
||||
|
||||
import { test, after } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { existsSync } from "node:fs";
|
||||
import { loadScenario } from "../../src/declaration/parse.ts";
|
||||
import { raise } from "../../src/lifecycle/raise.ts";
|
||||
import { destroy, exec } from "../../src/lifecycle/operate.ts";
|
||||
import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts";
|
||||
import { labIsUsable, destroyAll } from "./harness.ts";
|
||||
|
||||
const capability = await labIsUsable();
|
||||
const binary = hostBinaryPath();
|
||||
|
||||
const skip = !capability.usable
|
||||
? `lab not usable: ${capability.why}`
|
||||
: !binary
|
||||
? "MESH_LAB_HOST_BINARY is not set — build novox/mesh-host and point at it"
|
||||
: !existsSync(binary)
|
||||
? `MESH_LAB_HOST_BINARY points at ${binary}, which does not exist`
|
||||
: false;
|
||||
|
||||
let instanceId = "";
|
||||
|
||||
after(async () => {
|
||||
if (instanceId) await destroy(instanceId);
|
||||
await destroyAll("bootstrap-single-");
|
||||
}, { timeout: 400_000 });
|
||||
|
||||
test("a raised machine contains the host", { skip, timeout: 900_000 }, async () => {
|
||||
const raised = await raise(loadScenario("scenarios/bootstrap-single.yml"), {});
|
||||
instanceId = raised.instanceId;
|
||||
|
||||
const { stdout } = await exec(instanceId, "anchor", [HOST_PATH, "version"]);
|
||||
assert.ok(stdout.trim().length > 0, "the host is on the machine but does not run there");
|
||||
});
|
||||
|
||||
test("the host reports the MACHINE, not the workstation that placed it", { skip, timeout: 120_000 }, async () => {
|
||||
// The check that proves detection detects rather than reporting a constant. A raised VM and
|
||||
// the workstation differ in every capability, so a host that reported the workstation's
|
||||
// answers would be obvious here and invisible anywhere else.
|
||||
const { stdout } = await exec(instanceId, "anchor", [HOST_PATH, "profile", "--json"]);
|
||||
const profile = JSON.parse(stdout) as {
|
||||
capabilities: { name: string; present: boolean; detail: string }[];
|
||||
};
|
||||
const by = new Map(profile.capabilities.map((c) => [c.name, c]));
|
||||
|
||||
// A machine raised by the lab is root and has a clean init. The workstation session is
|
||||
// neither, so these are the two that would flip if the wrong machine were being read.
|
||||
assert.equal(by.get("privileged")?.present, true, "a raised machine should be root");
|
||||
assert.equal(by.get("service-manager")?.present, true, "a raised machine should have an init");
|
||||
|
||||
for (const [name, verdict] of by) {
|
||||
assert.ok(verdict.detail.trim().length > 0, `${name} was reported with no reason`);
|
||||
}
|
||||
});
|
||||
|
||||
test("ADR 0031 — the host confirms the machine carries no overlay", { skip, timeout: 120_000 }, async () => {
|
||||
// The lab provides the underlay and NOTHING of the overlay. Asserted elsewhere by looking
|
||||
// for wireguard interfaces; here the placed host reports it independently, which is a
|
||||
// second witness rather than the same check twice.
|
||||
const { stdout } = await exec(instanceId, "anchor", [HOST_PATH, "profile", "--json"]);
|
||||
const profile = JSON.parse(stdout) as { capabilities: { name: string; present: boolean }[] };
|
||||
const overlay = profile.capabilities.find((c) => c.name === "overlay");
|
||||
|
||||
assert.equal(overlay?.present, false, "a freshly raised machine already had an overlay");
|
||||
});
|
||||
|
||||
test("the host's inventory is of the raised machine", { skip, timeout: 120_000 }, async () => {
|
||||
const { stdout } = await exec(instanceId, "anchor", [HOST_PATH, "inventory", "--json"]);
|
||||
const inv = JSON.parse(stdout) as {
|
||||
machine: string; cpus: number; memory_kb: number; observed_at: string; unreadable?: string[];
|
||||
};
|
||||
|
||||
assert.ok(inv.machine.length > 0, "the machine did not report a name");
|
||||
assert.ok(inv.cpus > 0 && inv.memory_kb > 0, "the machine reported no cpus or no memory");
|
||||
assert.deepEqual(inv.unreadable ?? [], [], "something could not be read on a machine we raised");
|
||||
|
||||
// The scenario gives each machine 1GiB and 2 cpus. A host reporting the workstation's
|
||||
// 24 cpus would pass every check above.
|
||||
assert.ok(inv.cpus <= 4, `reported ${inv.cpus} cpus — that is not the raised machine`);
|
||||
});
|
||||
@@ -0,0 +1,89 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { parseScenario } from "../src/declaration/parse.ts";
|
||||
import { planPlacements, PLACEABLE } from "../src/lifecycle/place.ts";
|
||||
import { assertSupported, UnsupportedError } from "../src/lifecycle/supported.ts";
|
||||
|
||||
/**
|
||||
* `place:` is the seam where the lab stops being infrastructure with no consumer. Each test
|
||||
* names what it defends, per novox/hq ADR 0034.
|
||||
*/
|
||||
|
||||
function scenario(place: string): ReturnType<typeof parseScenario> {
|
||||
return parseScenario(`
|
||||
scenario: placing
|
||||
segments:
|
||||
hosting:
|
||||
kind: public
|
||||
cidr: [192.0.2.0/24]
|
||||
machines:
|
||||
anchor:
|
||||
at: { segment: hosting, address: [192.0.2.10] }
|
||||
peer:
|
||||
at: { segment: hosting, address: [192.0.2.20] }
|
||||
${place}
|
||||
`);
|
||||
}
|
||||
|
||||
test("`all:` reaches every machine", () => {
|
||||
const placements = planPlacements(scenario("place:\n all: [host]"));
|
||||
assert.deepEqual(
|
||||
placements.map((p) => p.machine).sort(),
|
||||
["anchor", "peer"],
|
||||
);
|
||||
for (const p of placements) assert.deepEqual(p.artifacts, ["host"]);
|
||||
});
|
||||
|
||||
test("a per-machine entry OVERRIDES `all:`, it does not add to it", () => {
|
||||
// Worth being exact about: a scenario naming one artifact for one machine gets that
|
||||
// artifact, not that artifact plus everything in `all:`. The opposite reading would place
|
||||
// things nobody asked for, which is the shape of fault this lab exists to catch.
|
||||
const placements = planPlacements(scenario("place:\n all: [host]\n anchor: [substrate]"));
|
||||
const byMachine = new Map(placements.map((p) => [p.machine, p.artifacts]));
|
||||
|
||||
assert.deepEqual(byMachine.get("anchor"), ["substrate"], "anchor should have ONLY substrate");
|
||||
assert.deepEqual(byMachine.get("peer"), ["host"]);
|
||||
});
|
||||
|
||||
test("a machine placed with nothing is not a placement", () => {
|
||||
const placements = planPlacements(scenario("place:\n all: [host]\n anchor: []"));
|
||||
assert.deepEqual(placements.map((p) => p.machine), ["peer"]);
|
||||
});
|
||||
|
||||
test("no `place:` at all is no placements, not an error", () => {
|
||||
assert.deepEqual(planPlacements(scenario("")), []);
|
||||
});
|
||||
|
||||
test("placing the host is supported", () => {
|
||||
// The whole point of stage 1: this used to be refused.
|
||||
assert.doesNotThrow(() => assertSupported(scenario("place:\n all: [host]")));
|
||||
});
|
||||
|
||||
test("a tier that does not exist is refused BY NAME", () => {
|
||||
// Named individually rather than refused as a whole, so a scenario placing a host and a
|
||||
// substrate is told exactly which half the lab cannot do — rather than being told `place:`
|
||||
// is unsupported when half of it now works.
|
||||
try {
|
||||
assertSupported(scenario("place:\n all: [host, substrate]\n peer: [control]"));
|
||||
assert.fail("expected a refusal");
|
||||
} catch (err) {
|
||||
assert.ok(err instanceof UnsupportedError);
|
||||
const missing = err.missing.join("\n");
|
||||
assert.match(missing, /substrate/, "the substrate was not named");
|
||||
assert.match(missing, /control/, "the control plane was not named");
|
||||
assert.doesNotMatch(missing, /place: host/, "the host is placeable and was refused anyway");
|
||||
}
|
||||
});
|
||||
|
||||
test("the refusal says what CAN be placed", () => {
|
||||
// A refusal that does not say what is possible sends someone to the source to find out.
|
||||
try {
|
||||
assertSupported(scenario("place:\n all: [forge]"));
|
||||
assert.fail("expected a refusal");
|
||||
} catch (err) {
|
||||
assert.ok(err instanceof UnsupportedError);
|
||||
for (const placeable of PLACEABLE) {
|
||||
assert.match(err.missing.join("\n"), new RegExp(placeable));
|
||||
}
|
||||
}
|
||||
});
|
||||
+13
-5
@@ -48,24 +48,32 @@ machines: { a: { at: { segment: net, address: [192.0.2.1] }, inbound: deny } }`)
|
||||
assert.doesNotThrow(() => assertSupported(scenario));
|
||||
});
|
||||
|
||||
test("place is still refused — there is nothing to place yet", () => {
|
||||
test("the host is placeable — it used to be refused, and tier 0 now exists", () => {
|
||||
// These two tests failed the moment placement worked, which is what they were for. They
|
||||
// defended "there is nothing to place yet" while that was true; the decision changed, so
|
||||
// they change with it rather than being deleted (novox/hq ADR 0034).
|
||||
const scenario = parseScenario(`scenario: x
|
||||
segments: { net: { kind: public, cidr: [192.0.2.0/24] } }
|
||||
machines: { a: { at: { segment: net, address: [192.0.2.1] } } }
|
||||
place: { all: [host] }`);
|
||||
assert.throws(() => assertSupported(scenario), UnsupportedError);
|
||||
assert.doesNotThrow(() => assertSupported(scenario));
|
||||
});
|
||||
|
||||
test("the refusal explains what would silently be missing", () => {
|
||||
test("a tier above 0 is still refused, and named", () => {
|
||||
// The refusal narrowed rather than disappearing. A scenario placing a host AND a substrate
|
||||
// must be told which half is missing — not that `place:` is unsupported, when half of it
|
||||
// now works.
|
||||
const scenario = parseScenario(`scenario: x
|
||||
segments: { net: { kind: public, cidr: [192.0.2.0/24] } }
|
||||
machines: { a: { at: { segment: net, address: [192.0.2.1] } } }
|
||||
place: { all: [host] }`);
|
||||
place: { all: [host, substrate] }`);
|
||||
try {
|
||||
assertSupported(scenario);
|
||||
assert.fail("should have refused");
|
||||
} catch (err) {
|
||||
assert.equal((err as UnsupportedError).missing.length, 1);
|
||||
assert.ok(err instanceof UnsupportedError);
|
||||
assert.equal(err.missing.length, 1, `expected only the substrate: ${err.missing.join(", ")}`);
|
||||
assert.match(err.missing[0] ?? "", /substrate/);
|
||||
assert.match(err instanceof Error ? err.message : "", /silently lacks them/);
|
||||
}
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user