A one-node mesh, and twelve things that have to be true of it
The common case, and the one that was never tested as a whole. What existed
asked whether four machines converged; it never asked whether ONE machine ends
up holding a mesh.
The order was wrong too. Three machines were enrolled second, into a mesh that
could not yet produce a single module, and that was reported as though something
had been shown. 17-raising-a-mesh is explicit: genesis ends with a mesh that
RUNS, and what remains after the core modules are built is "adding machines".
So the core comes first and machines arrive last — here, not at all, because a
second node is only meaningful once the first is complete.
Three things were missing entirely and nothing complained, because nothing asked:
the mesh never built its own catalogue, never had a store of its own for that
catalogue to use, and never rebuilt its own control plane through the module
path.
And four checks that were absent rather than failing:
- it can describe itself — status, module list, plan --json, and the
catalogue's five tools ASKED rather than observed. A container being up was
being read as the catalogue working, which is the same error as matching a
container by substring and finding the wrong one.
- its networking is what the modules asked for — default closed, ssh open,
declared ports open, .internal names written, module networks present. Left
out altogether, which is hard to defend given the firewall work this week.
- a change to a module's source reaches the machine on its own. The capability
the migration depends on.
- it comes back after a reboot. Never once tested; the lab had no way to
restart a machine, because nothing had ever needed one.
Machines are named by role now — anchor, home-server, workstation, laptop — not
after the operator's own nodes, which made test output and real state hard to
tell apart.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
This commit is contained in:
@@ -48,10 +48,10 @@ import { labIsUsable, destroyAll, substrateBundle } from "./harness.ts";
|
||||
import { genesis, type GenesisResult } from "./genesis.ts";
|
||||
|
||||
const SCENARIO = "fresh-mesh";
|
||||
const CONTROL = "novox";
|
||||
const HOME_NODES = ["ace", "shanks", "g14"];
|
||||
const CONTROL = "anchor";
|
||||
const HOME_NODES = ["home-server", "workstation", "laptop"];
|
||||
/** The joined machine asked to run a mesh-built module. Any of the three would do. */
|
||||
const SECOND = "ace";
|
||||
const SECOND = "home-server";
|
||||
|
||||
/** The anchor's public address — what every other machine dials, and what its own token must name. */
|
||||
const ANCHOR = "192.0.2.20";
|
||||
@@ -74,9 +74,30 @@ const BASE = { module: "mesh-tools", repo: "mesh-tools", path: "" };
|
||||
* record of, so it provides nothing to anything. A module asking for `amqp` is asking for a
|
||||
* provider in the graph, and this is it. No build: its image is upstream.
|
||||
*/
|
||||
const PROVIDER = { module: "lavinmq", repo: "mesh-catalog", path: "modules/lavinmq" };
|
||||
const PROVIDER = { module: "lavinmq", repo: "mesh-catalog", path: "modules/lavinmq", container: "mesh-lavinmq" };
|
||||
/**
|
||||
* The store, and the catalogue that cannot exist without it.
|
||||
*
|
||||
* Both were missing from this test entirely, and nothing complained, because nothing asked. A mesh
|
||||
* with no catalogue holds no module graph — it cannot say what it has, what a module is made of,
|
||||
* what a change reaches, or what must be rebuilt. It ran anyway, which is the point: "the mesh is
|
||||
* up" was being read off genesis finishing.
|
||||
*/
|
||||
const STORE = { module: "postgres", repo: "mesh-catalog", path: "modules/postgres", container: "mesh-postgres" };
|
||||
const CATALOGUE = { module: "mesh-catalog", repo: "mesh-catalog", path: "modules/mesh-catalog", container: "mesh-catalog" };
|
||||
/** The control plane, rebuilt from its own repository — the step that ends the installer's tenure. */
|
||||
const CONTROL_PLANE = { module: "mesh-control", repo: "mesh-control", path: "", container: "mesh-control" };
|
||||
/** Named once, because the step title is also how later steps say what they waited on. */
|
||||
const NEEDS = "the mesh runs a broker for that module to talk to";
|
||||
const GENESIS = "a bare machine becomes a mesh of one, raised by the installer";
|
||||
const BASE_BUILT = "the mesh builds the shared base from source";
|
||||
const STORE_RUNS = "the mesh builds and runs a store of its own";
|
||||
const CATALOGUE_RUNS = "the mesh builds and runs its own catalogue";
|
||||
const CONTROL_REBUILT = "the mesh rebuilds its own control plane from source";
|
||||
const MODULE_BUILT = "the mesh builds a module standing on that base";
|
||||
const ANCHOR_RUNS = "the anchor runs the module the mesh built";
|
||||
const JOINED = "three machines join the mesh, and reach it over its private network";
|
||||
const SECOND_RUNS = "a joined machine runs the module the mesh built";
|
||||
const MODULE = { module: "amqp-ping", repo: "mesh-catalog", path: "modules/amqp-ping" };
|
||||
|
||||
const capability = await labIsUsable();
|
||||
@@ -162,6 +183,34 @@ async function mesh(command: string, timeoutMs?: number): Promise<string> {
|
||||
return must(CONTROL, `docker exec mesh-control /mesh-control ${command}`, timeoutMs);
|
||||
}
|
||||
|
||||
/** A module the mesh has to make for itself: where its source is, and what it runs when it works. */
|
||||
interface CoreModule { module: string; repo: string; path: string; container: string }
|
||||
|
||||
/**
|
||||
* Build a module from source, install it, and wait for it to actually run.
|
||||
*
|
||||
* The whole sequence a module goes through, in one place because the mesh has to do it for several
|
||||
* before it is a mesh at all: register the manifest, build it from its repository and path, issue
|
||||
* the broker account its runtime needs, assign it, send it, and then ask the machine whether the
|
||||
* container is up.
|
||||
*
|
||||
* **The account is not optional and is not swallowed.** A module's runtime is a tool host: it
|
||||
* connects to the mesh's broker before doing anything. Without an account the mesh still fills the
|
||||
* secret the module declares it owns — with a generated value — so the container starts, fails to
|
||||
* parse a password as a credential document, and loops on a JSON syntax error mentioning no
|
||||
* missing account. That cost a step that reported PASS.
|
||||
*/
|
||||
async function bringUp(m: CoreModule): Promise<string> {
|
||||
await registerModule(m.module, resolve(catalogDir, m.module, "module.json"));
|
||||
const built = await mesh(
|
||||
`build ${forgeUrl(m.repo)} --path ${m.path} --ref ${refFor(m.repo)} --wait 1200s`, 1_500_000);
|
||||
assert.doesNotMatch(built, /failed/i, built);
|
||||
await mesh(`module issue ${m.module} --node ${CONTROL}`);
|
||||
await mesh(`assign ${CONTROL} ${m.module}`);
|
||||
await mesh(`push ${CONTROL}`, 600_000);
|
||||
return `${built}\n${await waitForContainer(CONTROL, m.container)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for one container, BY NAME, to be running.
|
||||
*
|
||||
@@ -262,7 +311,7 @@ before(async () => {
|
||||
//
|
||||
// The shared description, the same one `genesis-single` calls. The bundle is the TEMPLATE with
|
||||
// nothing held: no image is pre-resolved, because none is here to resolve to.
|
||||
await step("novox becomes a mesh of one, raised by the installer", null, async () => {
|
||||
await step("a bare machine becomes a mesh of one, raised by the installer", null, async () => {
|
||||
try {
|
||||
raised = await genesis({
|
||||
instanceId,
|
||||
@@ -289,41 +338,29 @@ before(async () => {
|
||||
return raised.report.join("\n");
|
||||
});
|
||||
|
||||
// ---- 2. JOINING -----------------------------------------------------------------------------
|
||||
// ---- 2..6. THE MESH BECOMES ONE --------------------------------------------------------------
|
||||
//
|
||||
// Host binary and a token. novox is NOT in this loop — the installer enrolled it, and enrolling
|
||||
// it again would offer the mesh a second identity for a node it already knows.
|
||||
await step("three machines join it across the household gateway",
|
||||
"novox becomes a mesh of one, raised by the installer", async () => {
|
||||
const said: string[] = [];
|
||||
for (const machine of HOME_NODES) {
|
||||
await mesh(`node add ${machine}`);
|
||||
const token = tokenFrom(await mesh(`token issue --node ${machine}`));
|
||||
const out = await must(machine, `${HOST_PATH} enrol --token ${quote(token)}`, 180_000);
|
||||
assert.match(out, new RegExp(`enrolled as ${machine}`), out);
|
||||
await must(machine, `nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`);
|
||||
said.push(` ${machine} enrolled and running`);
|
||||
}
|
||||
const nodes = await mesh("node list");
|
||||
said.push(nodes.trim());
|
||||
return said.join("\n");
|
||||
});
|
||||
// **Joining used to be here, second, and that was the wrong order.** Three machines were enrolled
|
||||
// into a mesh that could not yet produce a single module, and the step was reported as though
|
||||
// something had been shown. They do join — reliably — but joining a mesh that can build nothing
|
||||
// proves only that enrolment works, which was never the doubtful part.
|
||||
//
|
||||
// `17-raising-a-mesh` says it plainly: genesis ends with a mesh of one that RUNS, and that is not
|
||||
// the same as a mesh that WORKS; what remains after the core modules are built is "adding
|
||||
// machines, and deciding what they run". So everything a mesh needs to be a mesh happens first,
|
||||
// and machines arrive at the end.
|
||||
|
||||
// ---- 3. THE MESH BUILDS THE SHARED BASE -----------------------------------------------------
|
||||
//
|
||||
// The toolchain and runtime every module with code of its own stands on. It is a module, and it
|
||||
// is built like one — cloned from the forge by the builder installing put here, compiled on the
|
||||
// machine, published into the mesh's own registry.
|
||||
await step("the mesh builds the shared base from source",
|
||||
"three machines join it across the household gateway", async () => {
|
||||
await step(BASE_BUILT, GENESIS, async () => {
|
||||
// Registered from the manifest the builder will also read, so what the mesh holds and what it
|
||||
// builds are the same description of the same module.
|
||||
//
|
||||
// **Not swallowed.** This call used to end in `.catch(() => {})`, on the reasoning that the
|
||||
// base might already be known. It hid a real failure — the manifest was being named at a path
|
||||
// inside a container that had never seen it — and the step passed anyway, because a base with
|
||||
// nothing to stand on builds whether or not the mesh has a record of it. The next step, which
|
||||
// needs that record, is where it surfaced.
|
||||
// nothing to stand on builds whether or not the mesh has a record of it.
|
||||
await registerModule(BASE.module, baseManifest);
|
||||
const built = await mesh(
|
||||
`build ${forgeUrl(BASE.repo)} --ref ${refFor(BASE.repo)} --wait 1200s`, 1_500_000);
|
||||
@@ -331,9 +368,39 @@ before(async () => {
|
||||
return built;
|
||||
});
|
||||
|
||||
// ---- 4. AND A MODULE STANDING ON IT ---------------------------------------------------------
|
||||
await step("the mesh builds a module standing on that base",
|
||||
"the mesh builds the shared base from source", async () => {
|
||||
// A store of its own. **Not the substrate's.** The installer raises a store for the control
|
||||
// plane to keep its own records in, the way it raises a broker — plumbing, not a module the mesh
|
||||
// has any record of, so it provides nothing to anything. A module that wants a database wants a
|
||||
// provider in the graph, and the catalogue below is the first thing to want one.
|
||||
await step(STORE_RUNS, BASE_BUILT, () => bringUp(STORE));
|
||||
|
||||
// And the catalogue, which was missing from this test altogether.
|
||||
//
|
||||
// Without it the mesh holds no module graph: it cannot say what it has, what a module is made
|
||||
// of, what a change to one reaches, or what must be rebuilt. A mesh in that state still runs,
|
||||
// which is exactly how its absence went unnoticed — "the mesh is up" was being read off the
|
||||
// installer finishing rather than off the mesh being able to answer anything.
|
||||
await step(CATALOGUE_RUNS, STORE_RUNS, () => bringUp(CATALOGUE));
|
||||
|
||||
// The control plane, rebuilt from its own repository and rolled out.
|
||||
//
|
||||
// The installer built it once, which is what got the mesh running. Building it again THROUGH THE
|
||||
// MODULE PATH — build, notice the version moved, roll it out — is a different claim: it is the
|
||||
// moment the mesh stops depending on the installer for anything, and the first time the thing
|
||||
// that performs an upgrade performs one on itself.
|
||||
await step(CONTROL_REBUILT, CATALOGUE_RUNS, async () => {
|
||||
const built = await mesh(
|
||||
`build ${forgeUrl(CONTROL_PLANE.repo)} --ref ${sourceRef} --wait 1200s`, 1_500_000);
|
||||
assert.doesNotMatch(built, /failed/i, built);
|
||||
const rolled = await mesh(`upgrade ${CONTROL_PLANE.module} roll-out`, 900_000);
|
||||
// Asked of the machine rather than believed from the command: the control plane that answers
|
||||
// afterwards is the one that has to be running for anything below this line to mean anything.
|
||||
await waitForContainer(CONTROL, CONTROL_PLANE.container);
|
||||
return `${built}\n${rolled}`;
|
||||
});
|
||||
|
||||
// ---- 7..8. SOMETHING TO RUN ---------------------------------------------------------------
|
||||
await step(MODULE_BUILT, CONTROL_REBUILT, async () => {
|
||||
await registerModule(MODULE.module, resolve(catalogDir, MODULE.module, "module.json"));
|
||||
const built = await mesh(
|
||||
`build ${forgeUrl(MODULE.repo)} --path ${MODULE.path} --ref ${refFor(MODULE.repo)} --wait 1200s`,
|
||||
@@ -347,61 +414,64 @@ before(async () => {
|
||||
return `${built}\n${builds}`;
|
||||
});
|
||||
|
||||
// ---- 4b. WHAT THE MODULE NEEDS --------------------------------------------------------------
|
||||
//
|
||||
// `amqp-ping` requires the `amqp` provision, and the mesh refused to place it: "nothing provides
|
||||
// amqp, wanted by amqp-ping". That refusal is correct and is the reason this step exists rather
|
||||
// than the reason to pick an easier module. **The substrate's broker is not a provider.** It is
|
||||
// raised by the installer as part of the bundle, so it is a running container and not a module
|
||||
// with something to offer — the mesh's own plumbing, not an entry in its graph. A module that
|
||||
// wants a broker wants one the mesh knows about.
|
||||
// than the reason to pick an easier module.
|
||||
//
|
||||
// `lavinmq` is that module. It was first added here on the belief that it needed no building —
|
||||
// its broker is an upstream image — and the mesh refused it: two of its three containers named a
|
||||
// placeholder digest, "which is never a real image". That was right. The broker is upstream, but
|
||||
// the module is not only the broker: it carries a run-once bootstrap that writes the broker's
|
||||
// its broker is an upstream image — and the mesh refused it again: two of its three containers
|
||||
// named a placeholder digest, "which is never a real image". Also right. The broker is upstream,
|
||||
// but the module is not only the broker: it carries a run-once bootstrap that writes the broker's
|
||||
// configuration, a provisioner that grants each consumer its own vhost and user, tools and an
|
||||
// event consumer. All of that is its own code and has to be built like anything else.
|
||||
await step(NEEDS, "the mesh builds a module standing on that base", async () => {
|
||||
await registerModule(PROVIDER.module, resolve(catalogDir, PROVIDER.module, "module.json"));
|
||||
const built = await mesh(
|
||||
`build ${forgeUrl(PROVIDER.repo)} --path ${PROVIDER.path} --ref ${refFor(PROVIDER.repo)} --wait 1200s`,
|
||||
1_500_000);
|
||||
assert.doesNotMatch(built, /failed/i, built);
|
||||
await mesh(`assign ${CONTROL} ${PROVIDER.module}`);
|
||||
// **Its account on the mesh's own broker, and not swallowed.** A module's runtime is a tool
|
||||
// host: it connects to the mesh broker before it does anything, and what it reads is a sealed
|
||||
// credential document. Without an account the mesh still fills the secret this module declares
|
||||
// it owns — with a generated value — so the container starts, fails to parse a password as a
|
||||
// credential, and crash-loops on a JSON syntax error that says nothing about the missing
|
||||
// account. Issuing it is not optional and neither is hearing that it failed.
|
||||
await mesh(`module issue ${PROVIDER.module} --node ${CONTROL}`);
|
||||
await mesh(`push ${CONTROL}`, 600_000);
|
||||
return waitForContainer(CONTROL, "mesh-lavinmq");
|
||||
});
|
||||
// event consumer, all of it its own code.
|
||||
await step(NEEDS, MODULE_BUILT, () => bringUp(PROVIDER));
|
||||
|
||||
// ---- 5. THE ANCHOR RUNS IT ------------------------------------------------------------------
|
||||
//
|
||||
// The machine that built it. This is the case every earlier proof covered, and it is here as the
|
||||
// control for step 6: if this fails, step 6's failure says nothing about fetching.
|
||||
await step("the anchor runs the module the mesh built", NEEDS, async () => {
|
||||
// control for the last step: if this fails, that one's failure says nothing about fetching.
|
||||
await step(ANCHOR_RUNS, NEEDS, async () => {
|
||||
await mesh(`module issue ${MODULE.module} --node ${CONTROL}`);
|
||||
await mesh(`assign ${CONTROL} ${MODULE.module}`);
|
||||
await mesh(`push ${CONTROL}`, 600_000);
|
||||
return waitForContainer(CONTROL, MODULE.module);
|
||||
});
|
||||
|
||||
// ---- 6. AND A MACHINE THAT DID NOT BUILD IT -------------------------------------------------
|
||||
// ---- 9. NOW MACHINES MAY ARRIVE ---------------------------------------------------------------
|
||||
//
|
||||
// **The thing nothing has ever checked.** ace did not build this image and has never seen it. To
|
||||
// run it, it must fetch it from the mesh's registry — and a joined node has no account there.
|
||||
// The mesh grants a consumer a credential for a database; it does not yet do so for the store
|
||||
// its own images live in (novox/hq issue 042).
|
||||
// Host binary and a token, and nothing else. The anchor is NOT in this loop — the installer
|
||||
// enrolled it, and enrolling it again would offer the mesh a second identity for a node it
|
||||
// already knows.
|
||||
//
|
||||
// Asked anyway, and asked LAST, so that when it fails the five steps above still stand as
|
||||
// evidence of what does work.
|
||||
await step(`a joined machine runs the module the mesh built`,
|
||||
"the anchor runs the module the mesh built", async () => {
|
||||
// And they are placed on the mesh's private network, which the earlier ordering never did. The
|
||||
// mesh refuses to bind a consumer to a provider on another machine when either is missing from
|
||||
// it — "they are not both on the private network" — so without this the last step fails for a
|
||||
// reason that has nothing to do with what it is asking.
|
||||
await step(JOINED, ANCHOR_RUNS, async () => {
|
||||
const said: string[] = [];
|
||||
await mesh(`overlay place ${CONTROL} --hub --endpoint ${ANCHOR}:51820 --site hosting`);
|
||||
for (const machine of HOME_NODES) {
|
||||
await mesh(`node add ${machine}`);
|
||||
const token = tokenFrom(await mesh(`token issue --node ${machine}`));
|
||||
const out = await must(machine, `${HOST_PATH} enrol --token ${quote(token)}`, 180_000);
|
||||
assert.match(out, new RegExp(`enrolled as ${machine}`), out);
|
||||
await must(machine, `nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`);
|
||||
await mesh(`overlay place ${machine} --site home`);
|
||||
said.push(` ${machine} enrolled, running, and on the private network`);
|
||||
}
|
||||
said.push((await mesh("node list")).trim());
|
||||
said.push((await mesh("overlay show")).trim());
|
||||
return said.join("\n");
|
||||
});
|
||||
|
||||
// ---- 10. AND A MACHINE THAT DID NOT BUILD IT --------------------------------------------------
|
||||
//
|
||||
// **The thing nothing has ever checked.** This machine did not build the image and has never seen
|
||||
// it. To run it, it must fetch it from the mesh's registry — and a joined node has no account
|
||||
// there (novox/hq issue 042), nor any reason to trust a registry serving plain HTTP over the
|
||||
// network (issue 048). Both are open, and both are invisible on a mesh of one.
|
||||
//
|
||||
// Asked anyway, and asked LAST, so that when it fails everything above still stands as evidence
|
||||
// of what does work.
|
||||
await step(SECOND_RUNS, JOINED, async () => {
|
||||
await mesh(`module issue ${MODULE.module} --node ${SECOND}`);
|
||||
await mesh(`assign ${SECOND} ${MODULE.module}`);
|
||||
await mesh(`push ${SECOND}`, 600_000);
|
||||
@@ -427,7 +497,7 @@ after(async () => {
|
||||
}, { timeout: 900_000 });
|
||||
|
||||
for (const name of [
|
||||
"novox becomes a mesh of one, raised by the installer",
|
||||
"a bare machine becomes a mesh of one, raised by the installer",
|
||||
"three machines join it across the household gateway",
|
||||
"the mesh builds the shared base from source",
|
||||
"the mesh builds a module standing on that base",
|
||||
|
||||
Reference in New Issue
Block a user