novox/mesh-lab
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Raise machines that are somebody, and a bed that hands over nothing

Browse Source 
Every machine in a bed is a clone of one base image, so all of them booted with
the same /etc/machine-id. systemd's DHCP client derives its client identifier
from that file and dnsmasq keys leases on the identifier rather than the MAC, so
four machines with four distinct MACs were handed one address and the host kept
one ARP entry for it. Whichever machine last answered an ARP request received
everybody's replies.

This is the fault behind every run lost to "flaky lab DNS": resolution that works
two times in three, pulls that succeed on a retry, and one machine out of four
being fine while the rest have no path at all. It survived an earlier diagnosis
that blamed resolver ordering, because reordering resolvers on a machine that has
just won the ARP race looks exactly like a fix.

Each machine is now given its own machine-id before the uplink lease is asked
for, and a check after addresses are applied refuses to go on if two machines
took the same one — the positive control this never had, since the fault is
invisible where it happens and unrecognisable where it surfaces.

The egress check also now demands five consecutive lookups rather than one. A
single answer is what let a machine resolving one query in three pass and then
die twenty minutes later inside a pull.

And fresh-mesh: whole-mesh-full's topology with genesis-single's honesty. The
four-machine bed loads thirty-four of the mesh's own images onto its machines
from the workstation because it does not build them, which is a shape no real
installation has and the same fiction the lab removed when it deleted its own
registry. This scenario names no images at all. The machines pull what is public,
the installer builds the control plane, and the mesh builds the rest — including,
last and deliberately, a module on a machine that did not build it.

Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
This commit is contained in:
jschoubben
2026-09-14 20:41:55 +02:00
parent 7fa1f1f0bb
commit babf08b9f8
4 changed files with 600 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/integration/fresh-mesh.test.ts
+355
View File
@@ -0,0 +1,355 @@
/**
* FOUR FRESH MACHINES, AND NOTHING HANDED TO THEM.
*
* The four-machine bed (`whole-mesh-full`) proves the mesh converges. It does so by loading
* thirty-four of the mesh's own images onto its machines from the workstation, because it does not
* build them — something beside the bed built them and copied them in. No real installation looks
* like that, and the lab has been burned by exactly this shape before: it used to raise a registry
* inside the scenario, and a bootstrap that only worked against that registry went green here and
* would have failed on any bare machine.
*
* This bed hands over nothing. The scenario has no `images:` list at all. What the machines get is
* a container runtime and the host binary — prerequisites of a machine, not parts of a mesh — and
* from there:
*
* 1. novox is raised into a mesh of one by the installer, which BUILDS the control plane.
* 2. ace, shanks and g14 JOIN it, across a household NAT, with a token and nothing else.
* 3. The mesh builds the shared base from source, with its own builder.
* 4. The mesh builds a real module standing on that base.
* 5. The anchor runs it, pinned to a digest the mesh's own registry assigned.
* 6. A JOINED machine runs it — which means pulling from a registry that asks who it is.
*
* Steps 1 and 2 are proven elsewhere and are here because the later ones need them. **Steps 3
* through 6 are what this bed exists for**, and 6 is the one nothing has ever checked: genesis
* puts the builder, the registry and everything they produce on ONE machine, so every earlier
* proof of a mesh-built module running is a proof about the machine that built it. A second
* machine has to fetch, and fetching needs an account nothing yet grants (novox/hq issue 042).
*
* Each step is recorded separately rather than allowed to throw, so a gap at 6 reports as a gap at
* 6 instead of erasing the evidence for 3, 4 and 5.
*
* MESH_LAB_INCUS='sudo -n incus'
* MESH_LAB_HOST_BINARY=.../mesh-host/mesh-host
* MESH_LAB_BOOTSTRAP_BINARY=.../mesh-host/mesh-bootstrap
* MESH_LAB_BUNDLE=.../mesh-host/examples/substrate-first-node.lock
* MESH_LAB_CATALOG=.../mesh-catalog/modules
* MESH_LAB_SOURCE=<forge>/mesh-control.git MESH_LAB_SOURCE_REF=<commit>
* MESH_LAB_KEEP=1 to leave it standing afterwards
*/
import { test, before, after } from "node:test";
import assert from "node:assert/strict";
import { existsSync } from "node:fs";
import { resolve } from "node:path";
import { loadScenario } from "../../src/declaration/parse.ts";
import { raise } from "../../src/lifecycle/raise.ts";
import { destroy, exec, push } from "../../src/lifecycle/operate.ts";
import { bootstrapBinaryPath, hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts";
import { labIsUsable, destroyAll, substrateBundle } from "./harness.ts";
import { genesis, type GenesisResult } from "./genesis.ts";
const SCENARIO = "fresh-mesh";
const CONTROL = "novox";
const HOME_NODES = ["ace", "shanks", "g14"];
/** The joined machine asked to run a mesh-built module. Any of the three would do. */
const SECOND = "ace";
/** The anchor's public address — what every other machine dials, and what its own token must name. */
const ANCHOR = "192.0.2.20";
/** Where this mesh's registry answers, on the anchor's public address so a joined node can reach it. */
const REGISTRY = `${ANCHOR}:5000`;
/**
* The module built on top of the base, and the base it stands on.
*
* `amqp-ping` is deliberately small and deliberately REAL: its own TypeScript, compiled by the
* shared toolchain, running on the shared runtime, talking to the broker. A module whose artifact
* is a mirrored public image would pass every assertion below while skipping the whole of what is
* under test (novox/hq SELF-UPGRADE-PLAN, rule 1).
*/
const BASE = { module: "mesh-tools", repo: "mesh-tools", path: "" };
const MODULE = { module: "amqp-ping", repo: "mesh-catalog", path: "modules/amqp-ping" };
const capability = await labIsUsable();
const binary = hostBinaryPath();
const installer = bootstrapBinaryPath();
const bundle = process.env["MESH_LAB_BUNDLE"] ?? "";
const catalogDir = process.env["MESH_LAB_CATALOG"] ?? "";
const source = process.env["MESH_LAB_SOURCE"] ?? "";
const sourceRef = process.env["MESH_LAB_SOURCE_REF"] ?? "";
const KEEP = !!process.env["MESH_LAB_KEEP"];
const FIXED_ID = process.env["MESH_LAB_INSTANCE_ID"] ?? (KEEP ? "fresh-mesh-live" : undefined);
/**
* Where a repository other than the control plane's lives.
*
* Derived from `MESH_LAB_SOURCE` by swapping the last path segment, because every one of these
* repositories sits beside the others under the same owner on the same forge. Overridable, so a
* forge that is arranged differently does not need this bed edited.
*/
function forgeUrl(repo: string): string {
const override = process.env[`MESH_LAB_SOURCE_${repo.toUpperCase().replaceAll("-", "_")}`];
if (override) return override;
return source.replace(/[^/]+\.git$/, `${repo}.git`);
}
/** A branch is acceptable for an ordinary build; only genesis insists on a commit (ADR 0071). */
const buildRef = process.env["MESH_LAB_BUILD_REF"] ?? "main";
/**
* The shared base's manifest, on this workstation.
*
* The base is a repository with a manifest at its root (novox/hq ADR 0069), so unlike the
* catalogue's modules it is not under `MESH_LAB_CATALOG`. Derived from that path on the convention
* that the checkouts sit beside each other, and overridable for a layout where they do not.
*/
const baseManifest = process.env["MESH_LAB_BASE_MANIFEST"] ??
resolve(catalogDir, "..", "..", BASE.repo, "module.json");
const skip =
!capability.usable ? capability.why :
!binary ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" :
!installer ? "MESH_LAB_BOOTSTRAP_BINARY is not set to a built mesh-bootstrap" :
!source ? "MESH_LAB_SOURCE is not set to the repository the control plane is built from" :
!sourceRef ? "MESH_LAB_SOURCE_REF is not set to the commit to build" :
!bundle || !existsSync(bundle) ? "MESH_LAB_BUNDLE is not set to a substrate template" :
!catalogDir || !existsSync(catalogDir) ? "MESH_LAB_CATALOG is not set to mesh-catalog/modules" :
false;
let instanceId = "";
let raised: GenesisResult;
// ---- talking to the machines ------------------------------------------------------------------
function quote(s: string): string {
return `'${s.replaceAll("'", `'\\''`)}'`;
}
async function on(machine: string, command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> {
const { stdout } = await exec(instanceId, machine, [
"sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`,
], timeoutMs);
const marker = stdout.lastIndexOf("__exit=");
if (marker < 0) return { out: stdout, ok: false };
return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" };
}
async function must(machine: string, command: string, timeoutMs?: number): Promise<string> {
const { out, ok } = await on(machine, command, timeoutMs);
if (!ok) throw new Error(`${machine}: ${command}\n${out}`);
return out;
}
async function mesh(command: string, timeoutMs?: number): Promise<string> {
return must(CONTROL, `docker exec mesh-control /mesh-control ${command}`, timeoutMs);
}
function tokenFrom(said: string): string {
const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" "));
assert.ok(found, `no token in:\n${said}`);
return found;
}
// ---- steps, recorded rather than thrown --------------------------------------------------------
interface Step { ok: boolean; why: string; said: string }
const steps = new Map<string, Step>();
const order: string[] = [];
/** Run a step, remember what it said, and never throw. A step whose predecessor failed is skipped. */
async function step(name: string, after_: string | null, fn: () => Promise<string>): Promise<void> {
order.push(name);
if (after_ && !steps.get(after_)?.ok) {
steps.set(name, { ok: false, why: `not attempted — "${after_}" did not succeed`, said: "" });
console.log(`SKIPPED ${name}`);
return;
}
console.log(`\n======== ${name} ========`);
try {
const said = await fn();
steps.set(name, { ok: true, why: "", said });
console.log(`OK ${name}`);
} catch (err) {
const why = (err as Error).message;
steps.set(name, { ok: false, why, said: "" });
console.log(`FAILED ${name}\n${why.split("\n").slice(0, 25).join("\n")}`);
}
}
function report(name: string): string {
const s = steps.get(name);
if (!s) return `${name}: never ran`;
const lines = order.map((n) => {
const it = steps.get(n);
return ` ${it?.ok ? "PASS" : "FAIL"} ${n}`;
});
return `${s.why}\n\nWhere this bed got to:\n${lines.join("\n")}`;
}
before(async () => {
if (skip) return;
const bed = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), {
onProgress: (m) => console.log(`raise: ${m}`),
...(FIXED_ID ? { instanceId: FIXED_ID } : {}),
});
instanceId = bed.instanceId;
console.log(`INSTANCE ${instanceId}${KEEP ? " (KEEP — will be left standing)" : ""}`);
console.log(`NOTHING WAS LOADED: this scenario names no images. Every image on every machine ` +
`below was pulled from the internet or built by the mesh.`);
// ---- 1. GENESIS -----------------------------------------------------------------------------
//
// The shared description, the same one `genesis-single` calls. The bundle is the TEMPLATE with
// nothing held: no image is pre-resolved, because none is here to resolve to.
await step("novox becomes a mesh of one, raised by the installer", null, async () => {
try {
raised = await genesis({
instanceId,
node: CONTROL,
installer: installer as string,
catalogDir,
// The broker's advertised address, corrected.
//
// The template hardcodes 192.0.2.10:5671 — the address of the anchor in the single-machine
// bed. A token carries this verbatim as the endpoint an enrolling node dials, so on a mesh
// whose anchor is somewhere else every node, including this one, would enrol against an
// address nothing answers on. The installer refuses to guess it and says so, which is
// right: it does not know what this machine is called from outside.
bundleTemplate: substrateBundle(bundle, []).replaceAll("192.0.2.10:5671", `${ANCHOR}:5671`),
registry: REGISTRY,
source,
sourceRef,
log: (m) => console.log(m),
});
} catch (err) {
throw new Error(`the installer never ran: ${(err as Error).message}`);
}
if (!raised.ok) throw new Error(`${raised.step || "no step named"}: ${raised.why}\n\n${raised.report.join("\n")}`);
return raised.report.join("\n");
});
// ---- 2. JOINING -----------------------------------------------------------------------------
//
// Host binary and a token. novox is NOT in this loop — the installer enrolled it, and enrolling
// it again would offer the mesh a second identity for a node it already knows.
await step("three machines join it across the household gateway",
"novox becomes a mesh of one, raised by the installer", async () => {
const said: string[] = [];
for (const machine of HOME_NODES) {
await mesh(`node add ${machine}`);
const token = tokenFrom(await mesh(`token issue --node ${machine}`));
const out = await must(machine, `${HOST_PATH} enrol --token ${quote(token)}`, 180_000);
assert.match(out, new RegExp(`enrolled as ${machine}`), out);
await must(machine, `nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`);
said.push(` ${machine} enrolled and running`);
}
const nodes = await mesh("node list");
said.push(nodes.trim());
return said.join("\n");
});
// ---- 3. THE MESH BUILDS THE SHARED BASE -----------------------------------------------------
//
// The toolchain and runtime every module with code of its own stands on. It is a module, and it
// is built like one — cloned from the forge by the builder installing put here, compiled on the
// machine, published into the mesh's own registry.
await step("the mesh builds the shared base from source",
"three machines join it across the household gateway", async () => {
assert.ok(existsSync(baseManifest),
`no manifest for the shared base at ${baseManifest} — set MESH_LAB_BASE_MANIFEST`);
await push(instanceId, CONTROL, baseManifest, `/tmp/${BASE.module}.json`);
// Registered from the manifest the builder will also read, so what the mesh holds and what it
// builds are the same description of the same module.
await mesh(`module add /tmp/${BASE.module}.json`).catch(() => {});
const built = await mesh(
`build ${forgeUrl(BASE.repo)} --ref ${buildRef} --wait 1200s`, 1_500_000);
assert.doesNotMatch(built, /failed/i, built);
return built;
});
// ---- 4. AND A MODULE STANDING ON IT ---------------------------------------------------------
await step("the mesh builds a module standing on that base",
"the mesh builds the shared base from source", async () => {
const manifest = resolve(catalogDir, MODULE.module, "module.json");
assert.ok(existsSync(manifest), `no manifest at ${manifest}`);
await push(instanceId, CONTROL, manifest, `/tmp/${MODULE.module}.json`);
await mesh(`module add /tmp/${MODULE.module}.json`);
const built = await mesh(
`build ${forgeUrl(MODULE.repo)} --path ${MODULE.path} --ref ${buildRef} --wait 1200s`,
1_500_000);
assert.doesNotMatch(built, /failed/i, built);
// The point of the whole step: what came out is named by a digest this mesh's registry
// assigned, not by a placeholder and not by a tag.
const builds = await mesh(`builds ${MODULE.module}`);
assert.match(builds, /sha256:[0-9a-f]{12}/,
`the build recorded no digest — the module is not pinned to anything this registry serves:\n${builds}`);
return `${built}\n${builds}`;
});
// ---- 5. THE ANCHOR RUNS IT ------------------------------------------------------------------
//
// The machine that built it. This is the case every earlier proof covered, and it is here as the
// control for step 6: if this fails, step 6's failure says nothing about fetching.
await step("the anchor runs the module the mesh built",
"the mesh builds a module standing on that base", async () => {
await mesh(`module issue ${MODULE.module} --node ${CONTROL}`).catch(() => {});
await mesh(`assign ${CONTROL} ${MODULE.module}`);
await mesh(`push ${CONTROL}`, 600_000);
for (let i = 0; i < 40; i++) {
const ps = (await on(CONTROL, `docker ps --format '{{.Names}}\t{{.Status}}'`)).out;
if (/amqp-ping/.test(ps) && /Up /.test(ps.split("\n").find((l) => l.includes("amqp-ping")) ?? "")) {
return ps;
}
await new Promise((r) => setTimeout(r, 5_000));
}
throw new Error(`amqp-ping never came up on ${CONTROL}:\n` +
(await on(CONTROL, `docker ps -a --format '{{.Names}}\t{{.Status}}'`)).out);
});
// ---- 6. AND A MACHINE THAT DID NOT BUILD IT -------------------------------------------------
//
// **The thing nothing has ever checked.** ace did not build this image and has never seen it. To
// run it, it must fetch it from the mesh's registry — and a joined node has no account there.
// The mesh grants a consumer a credential for a database; it does not yet do so for the store
// its own images live in (novox/hq issue 042).
//
// Asked anyway, and asked LAST, so that when it fails the five steps above still stand as
// evidence of what does work.
await step(`a joined machine runs the module the mesh built`,
"the anchor runs the module the mesh built", async () => {
await mesh(`module issue ${MODULE.module} --node ${SECOND}`).catch(() => {});
await mesh(`assign ${SECOND} ${MODULE.module}`);
await mesh(`push ${SECOND}`, 600_000);
for (let i = 0; i < 40; i++) {
const ps = (await on(SECOND, `docker ps --format '{{.Names}}\t{{.Status}}'`)).out;
const line = ps.split("\n").find((l) => l.includes("amqp-ping"));
if (line && /Up /.test(line)) return ps;
await new Promise((r) => setTimeout(r, 5_000));
}
const state = (await on(SECOND, `docker ps -a --format '{{.Names}}\t{{.Status}}'`)).out;
const log = (await on(SECOND, `tail -40 /var/log/mesh-host.log`)).out;
throw new Error(
`amqp-ping never came up on ${SECOND} — the machine that did NOT build it.\n\n` +
`containers:\n${state}\n\nwhat the host said:\n${log}`);
});
console.log(`\n================ WHAT THIS MESH DID FOR ITSELF ================`);
for (const n of order) console.log(` ${steps.get(n)?.ok ? "PASS" : "FAIL"} ${n}`);
}, { timeout: 7_200_000 });
after(async () => {
if (KEEP) {
console.log(`\nLEFT STANDING: ${instanceId} — not destroyed (MESH_LAB_KEEP).`);
return;
}
if (instanceId) await destroy(instanceId);
await destroyAll(`${SCENARIO}-`);
}, { timeout: 900_000 });
for (const name of [
"novox becomes a mesh of one, raised by the installer",
"three machines join it across the household gateway",
"the mesh builds the shared base from source",
"the mesh builds a module standing on that base",
"the anchor runs the module the mesh built",
"a joined machine runs the module the mesh built",
]) {
test(name, { skip, timeout: 60_000 }, () => {
assert.ok(steps.get(name)?.ok, report(name));
});
}