jschoubben
555401a787
ADR 0067's own acceptance check said the lab must raise its anchor by running the program a bare machine runs. It did not: whole-mesh-full applied the substrate bundle by hand and then looped enrolment over all four machines as one continuous operation. That gets the order right by accident and models the wrong shape — and an install procedure that exists only as a test fixture is exercised by whoever writes tests and never by whoever installs, which is why every bootstrap fault this year was found late. Two acts now, and the first gates the second. GENESIS is novox running mesh-bootstrap: the installer is built from source before the raise (make bootstrap, carrying the control-plane image built in the same run), placed beside the host binary, given the two manifests it reads, and run. The bed then asserts a WORKING MESH OF ONE — the control plane answers, the registry replies on /v2/, the container called mesh-control is running from a registry-pinned digest rather than an image id, the registry agrees it serves it, temp-mesh-control is gone, and the mesh has heard from its node. The image-id check is ADR 0067's "the pivot completed" verbatim: if it is still an id, nothing was published and this mesh can never roll out its own upgrades. JOINING is ace, shanks and g14: host binary, token, enrol, run. novox is NOT enrolled again — the installer already did it, and a second identity is one the mesh does not know. If genesis stops, the bed prints which of the installer's ten steps it stopped at and goes no further. A second machine joining a mesh that is not ready is a different failure, and running it would bury this one underneath it. The anchor is no longer handed mesh-control:development. Its absence is the point: the installer carries that image inside itself, and handing it over as well would make the load say "already held" and leave the carrying untested — the same class of fiction the lab's own registry used to hide. A unit test asserts the scenario keeps it out. The registry is reached at 127.0.0.1:5000, which is a finding rather than a shortcut: a runtime refuses a plain-HTTP registry at any address but a loopback one, so the digest the control-plane module is pinned to is one only the anchor can pull. Enough here, because only the anchor runs a control plane. Written down in the bed. Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
1036 lines
55 KiB
TypeScript
1036 lines
55 KiB
TypeScript
/**
|
|
* The FULL mesh in its REAL production shape: two segments, one access point, one overlay — and the
|
|
* first multi-segment whole-mesh bed. It rewrites the flat three-node whole-mesh-full (separate
|
|
* anchor, everything on one public segment) into what production actually is:
|
|
*
|
|
* hosting (public) home (private, behind a NAT access point)
|
|
* novox 192.0.2.20 — the ANCHOR: ace 10.99.1.10 the home server, media/IoT set
|
|
* substrate (store/broker/ shanks 10.99.1.20 workstation (light: portainer only)
|
|
* control) + the whole novox g14 10.99.1.30 workstation (light: portainer only)
|
|
* set + overlay hub + ingress
|
|
*
|
|
* There is NO separate anchor: novox IS the anchor. The substrate runs on novox, and novox also
|
|
* enrols as a node and receives its own service set — the substrate host and a service node at once.
|
|
*
|
|
* TWO ACTS, AND THE BED NOW DISTINGUISHES THEM (novox/hq ADR 0067).
|
|
*
|
|
* GENESIS — novox is brought into existence by `mesh-bootstrap`, the installer, run on the
|
|
* machine exactly as a person would run it on a bare one: preflight, load the carried
|
|
* control-plane image, write the bundle, apply, verify, enrol itself, install the registry
|
|
* module, push the control-plane image into it, reinstall the control plane as an ordinary
|
|
* module pinned to the digest that push produced, retire the temporary one. Afterwards novox
|
|
* is a WORKING MESH OF ONE, and this bed asserts exactly that before going any further.
|
|
*
|
|
* JOINING — ace, shanks and g14 then join a mesh that already exists: host binary, token,
|
|
* `enrol`, run the agent. No bootstrap, no substrate, no registry. novox is NOT enrolled again.
|
|
*
|
|
* The bed used to do neither. It applied the substrate bundle itself and looped enrolment over all
|
|
* four machines as one continuous operation — which got the order right by accident and modelled
|
|
* the wrong shape, and is why ADR 0067's own acceptance check ("the bed bootstraps through the
|
|
* installer rather than around it") went unmet. Genesis GATES joining: if it stops, the bed says
|
|
* which of the installer's ten steps it stopped at and goes no further, because a second machine
|
|
* joining a mesh that is not ready is a different failure and must not be mistaken for this one.
|
|
*
|
|
* THE THING THIS BED EXISTS TO PROVE (the flat beds never could): does the WireGuard overlay tunnel
|
|
* FORM across the access point? A home node (ace/shanks/g14) dials novox's PUBLIC hub endpoint
|
|
* 192.0.2.20:51820/udp OUT through the household gateway's masquerade; the handshake has to complete
|
|
* through that NAT and the keepalive has to hold the hole open. Phase A drives exactly this and
|
|
* verifies it — WireGuard handshake state AND a ping over the overlay from a home node to novox —
|
|
* BEFORE any heavy module lands, so the cross-segment-overlay verdict survives whatever the module
|
|
* convergence then does. Phase B converges the full node sets and reports per node.
|
|
*
|
|
* SUBSTRATE-ON-NOVOX PORT COLLISIONS (a real consequence of collapsing the anchor onto novox that the
|
|
* separate-anchor beds never hit): the substrate store binds 127.0.0.1:5432 and novox's postgres
|
|
* provider publishes 5432; the substrate broker binds 5671 + 127.0.0.1:5672 and novox's lavinmq
|
|
* provider publishes 5672. The two provider host publishes are REMAPPED off the substrate's ports
|
|
* (REMAP below); consumers reach the providers over the mesh network on the container port, so the
|
|
* host side is free to move. Reported as a topology finding.
|
|
*
|
|
* PERSISTENT RAISE. With MESH_LAB_KEEP set the instance is raised under a fixed id
|
|
* (whole-mesh-full-live) and NOT torn down — it is left standing and browsable. Without it the bed
|
|
* behaves like every other: raise in before(), destroy in after().
|
|
*
|
|
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock
|
|
* MESH_LAB_BOOTSTRAP_BINARY=.../mesh-bootstrap MESH_LAB_CATALOG=.../mesh-catalog/modules
|
|
*/
|
|
|
|
import { test, before, after } from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import { existsSync, readFileSync, writeFileSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import { dirname, join, resolve } from "node:path";
|
|
import { loadScenario } from "../../src/declaration/parse.ts";
|
|
import { raise } from "../../src/lifecycle/raise.ts";
|
|
import { destroy, exec, instanceNameOf, push } from "../../src/lifecycle/operate.ts";
|
|
import {
|
|
bootstrapBinaryPath, hostBinaryPath, placeBootstrap, BOOTSTRAP_PATH, HOST_PATH,
|
|
} from "../../src/lifecycle/place.ts";
|
|
import { labIsUsable, destroyAll, substrateBundle, onTheMachine } from "./harness.ts";
|
|
import type { HeldImage } from "../../src/pinning.ts";
|
|
|
|
const capability = await labIsUsable();
|
|
const binary = hostBinaryPath();
|
|
const installer = bootstrapBinaryPath();
|
|
const bundle = process.env["MESH_LAB_BUNDLE"] ?? "";
|
|
const modulesEnv = process.env["MESH_LAB_MODULES"] ?? "";
|
|
|
|
const skip = !capability.usable
|
|
? `lab not usable: ${capability.why}`
|
|
: !binary || !existsSync(binary)
|
|
? "MESH_LAB_HOST_BINARY is not set to a built mesh-host"
|
|
: !bundle || !existsSync(bundle)
|
|
? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)"
|
|
: !installer || !existsSync(installer)
|
|
? "MESH_LAB_BOOTSTRAP_BINARY is not set to a built mesh-bootstrap (mesh-host `make " +
|
|
"bootstrap IMAGE=mesh-control:development`). The anchor is raised BY the installer now, " +
|
|
"so a run without one would be testing the procedure this bed exists to stop testing"
|
|
: false;
|
|
|
|
const SCENARIO = "whole-mesh-full";
|
|
/** novox hosts the substrate and the control plane; it is where `mesh` commands run. */
|
|
const CONTROL = "novox";
|
|
/** Every node in the mesh. novox is on hosting; the rest are behind the home gateway. */
|
|
const NODES = ["novox", "ace", "shanks", "g14"];
|
|
/**
|
|
* The machines that JOIN. novox is not one of them, and that is the distinction this bed was
|
|
* restructured to make: novox is brought into existence by the installer, which enrols it as part
|
|
* of genesis. Enrolling it again here would be a second identity the mesh does not know.
|
|
*/
|
|
const HOME_NODES = ["ace", "shanks", "g14"];
|
|
|
|
/** A checkout of the mesh's catalogue, on the anchor, for the installer to read manifests from. */
|
|
const CATALOGUE_ON_MACHINE = "/opt/mesh-catalog";
|
|
|
|
/**
|
|
* The manifests `mesh-bootstrap` reads out of that checkout — and only those.
|
|
*
|
|
* Named here rather than pushing the whole repository because the whole repository is a hundred
|
|
* megabytes of `node_modules` and the installer opens exactly two files: mesh-host's
|
|
* `internal/bootstrap` RegistryModule and ControlPlaneModule. If it ever opens a third, this list
|
|
* is where the bed finds out, by the installer saying which manifest it could not read.
|
|
*/
|
|
const CATALOGUE_MODULES = ["registry", "mesh-control"];
|
|
|
|
/**
|
|
* Where this mesh keeps its own images, as the anchor reaches it.
|
|
*
|
|
* **Loopback, and that is a finding rather than a shortcut.** The mesh's registry is plain HTTP on
|
|
* purpose — it is reached over the mesh's own network, which is already the encrypted and
|
|
* authenticated thing — and a container runtime refuses a plain-HTTP registry at any address
|
|
* EXCEPT a loopback one unless it has been told to allow it. So genesis can push to 127.0.0.1:5000
|
|
* with no configuration, and the reference the control-plane module is then pinned to is one only
|
|
* the anchor can pull. On this bed that is enough, because only the anchor runs the control plane.
|
|
* A mesh where a second machine had to pull it would need the runtimes told about the registry
|
|
* first, and nothing in the design says who does that.
|
|
*/
|
|
const MESH_REGISTRY = "127.0.0.1:5000";
|
|
|
|
/**
|
|
* ADR 0066 — the domain each public-facing node composes its routed names under.
|
|
*
|
|
* **The bed had none, so the ADR was untested by construction.** A module now contributes a `label`
|
|
* to `route` and nothing else; the mesh joins it to the node's public domain and the join is the
|
|
* whole feature. On a node with no public domain a labelled contribution composes to nothing — no
|
|
* host, no route — so every routed module on this bed was silently unreachable and the bed still
|
|
* went green. Two nodes face outward here; the workstations do not and get none, which is also part
|
|
* of the design being exercised.
|
|
*
|
|
* `.incus` rather than the real domains: this repository's beds name nothing routable.
|
|
*/
|
|
const PUBLIC_DOMAIN: Record<string, string> = { novox: "novox.incus", ace: "zurag.incus" };
|
|
|
|
/** Keep the instance standing and browsable rather than tearing it down. */
|
|
const KEEP = !!process.env["MESH_LAB_KEEP"];
|
|
const FIXED_ID = process.env["MESH_LAB_INSTANCE_ID"] ?? (KEEP ? "whole-mesh-full-live" : undefined);
|
|
|
|
const catalogDir = process.env["MESH_LAB_CATALOG"]
|
|
?? (modulesEnv ? resolve(dirname(dirname(dirname(modulesEnv))), "mesh-catalog", "modules") : "")
|
|
?? resolve(process.cwd(), "..", "mesh-catalog", "modules");
|
|
|
|
const MEDIA_DIRS = [
|
|
"/services/media/series", "/services/media/anime", "/services/media/movies",
|
|
"/services/media/music", "/services/media/audiobooks", "/services/media/downloads",
|
|
"/services/media/books",
|
|
];
|
|
|
|
type Mod = { name: string; containers: string[]; node?: boolean; runOnce?: string[] };
|
|
|
|
/**
|
|
* The novox set (feat/novox-conversions @ 431310f). The slug fix means only-office/de-spiegel/
|
|
* amqp-email-forwarder now resolve (their minted login was over the 20-char cap before), so they
|
|
* are INCLUDED. CORE gates; the rest are reported gaps (documented in the whole-mesh-novox bed):
|
|
* umami (provisioner url/admin unset), mailu (nox-schema gaps), only-office/de-spiegel (new plain
|
|
* apps, boot secondary), amqp-email-forwarder (hard-coded AMQP vhost authz), firewall/fail2ban
|
|
* (offline lab cannot fetch the package).
|
|
*/
|
|
const NOVOX: Mod[] = [
|
|
{ name: "postgres", containers: ["postgres", "mesh-postgres"] },
|
|
{ name: "redis", containers: ["redis", "mesh-redis"] },
|
|
{ name: "minio", containers: ["minio", "mesh-minio"] },
|
|
{ name: "mongodb", containers: ["mongo", "mesh-mongodb"] },
|
|
{ name: "mssql", containers: ["mssql", "mesh-mssql"] },
|
|
{ name: "lavinmq", containers: ["lavinmq", "mesh-lavinmq"] },
|
|
// ADR 0066: the proxy now REQUIRES an `acme-ca`, so the bed must assign a provider of one or
|
|
// route-proxy is unresolvable and takes every routed module down with it. step-ca is that
|
|
// provider, on the anchor, at mesh scope.
|
|
{ name: "step-ca", containers: ["step-ca"] },
|
|
{ name: "route-proxy", containers: ["route-proxy"] },
|
|
{ name: "keycloak", containers: ["keycloak", "mesh-keycloak"] },
|
|
{ name: "gitea", containers: ["gitea", "mesh-gitea"] },
|
|
{ name: "nextcloud", containers: ["nextcloud", "mesh-nextcloud"] },
|
|
{ name: "umami", containers: ["umami", "mesh-umami"] },
|
|
{ name: "photos", containers: ["photos-server", "photos-admin-client", "photos-client-eef", "photos-client-filip"] },
|
|
{ name: "invoicing", containers: ["invoicing-app", "invoicing-api"] },
|
|
{ name: "novox.be", containers: ["novox-be"] },
|
|
{ name: "only-office", containers: ["office-novox-be"] },
|
|
{ name: "de-spiegel", containers: ["de-spiegel-novox-be"] },
|
|
{ name: "amqp-email-forwarder", containers: ["amqp-email-forwarder"] },
|
|
{ name: "portainer", containers: ["portainer", "mesh-portainer"] },
|
|
{ name: "verdaccio", containers: ["verdaccio", "mesh-verdaccio"] },
|
|
// Already installed, assigned and running — genesis needed it to publish the control plane's
|
|
// image. Left in the plan on purpose: registering the same manifest and assigning it again is
|
|
// what an operator's `module add` + `assign` would do on a mesh that already has it, and a
|
|
// module the installer put there had better survive being asked for a second time. It also keeps
|
|
// mesh-registry in the convergence report, where a reader expects to see it.
|
|
{ name: "registry", containers: ["mesh-registry"] },
|
|
{
|
|
name: "mailu",
|
|
containers: [
|
|
"mailu-resolver", "mailu-redis", "mailu-admin", "mailu-imap", "mailu-smtp",
|
|
"mailu-antispam", "mailu-antivirus", "mailu-webmail", "mailu-webdav", "mailu-fetchmail",
|
|
"mailu-front", "mesh-mailu",
|
|
],
|
|
},
|
|
{ name: "firewall", containers: [], node: true },
|
|
{ name: "fail2ban", containers: [], node: true },
|
|
];
|
|
const CORE_NOVOX = new Set([
|
|
"postgres", "redis", "minio", "mongodb", "mssql", "lavinmq",
|
|
"route-proxy", "keycloak", "gitea", "nextcloud", "invoicing", "photos", "novox.be",
|
|
"portainer", "verdaccio", "registry",
|
|
]);
|
|
const GAPS_NOVOX = new Set([
|
|
"umami", "mailu", "firewall", "fail2ban", "only-office", "de-spiegel", "amqp-email-forwarder",
|
|
// step-ca is reported, not gated: the internal-CA ISSUANCE path is still being fixed in
|
|
// mesh-control, and this bed is not the place to discover that a fix has not landed yet. What is
|
|
// gated is the half that is decided and cheap — see the ADR 0066 section at the end.
|
|
"step-ca",
|
|
]);
|
|
|
|
/** The ace media/home set. */
|
|
const ACE: Mod[] = [
|
|
{ name: "postgres", containers: ["postgres", "mesh-postgres"] },
|
|
{ name: "redis", containers: ["redis", "mesh-redis"] },
|
|
{ name: "mssql", containers: ["mssql", "mesh-mssql"] },
|
|
{ name: "sonarr", containers: ["sonarr", "mesh-sonarr"] },
|
|
{ name: "radarr", containers: ["radarr", "mesh-radarr"] },
|
|
{ name: "lidarr", containers: ["lidarr", "mesh-lidarr"] },
|
|
{ name: "plex", containers: ["plex", "mesh-plex"] },
|
|
{ name: "bazarr", containers: ["bazarr", "mesh-bazarr"] },
|
|
{ name: "nzbget", containers: ["nzbget", "mesh-nzbget"] },
|
|
{ name: "qbittorrent", containers: ["qbittorrent", "mesh-qbittorrent"] },
|
|
{ name: "jackett", containers: ["jackett", "mesh-jackett"] },
|
|
{ name: "ombi", containers: ["ombi", "mesh-ombi"] },
|
|
{ name: "tautulli", containers: ["tautulli", "mesh-tautulli"] },
|
|
{ name: "bookshelf", containers: ["bookshelf", "mesh-bookshelf"] },
|
|
{ name: "home-assistant", containers: ["home-assistant", "mesh-home-assistant"] },
|
|
{ name: "mosquitto", containers: ["mosquitto", "mesh-mosquitto"], runOnce: ["mosquitto-bootstrap"] },
|
|
{ name: "influxdb", containers: ["influxdb", "mesh-influxdb"] },
|
|
{ name: "grafana", containers: ["grafana", "mesh-grafana"] },
|
|
{ name: "baserow", containers: ["baserow", "mesh-baserow"] },
|
|
{ name: "letta", containers: ["letta", "mesh-letta"] },
|
|
{ name: "nodered", containers: ["nodered", "mesh-nodered"] },
|
|
{ name: "searxng", containers: ["valkey", "searxng", "mesh-searxng"] },
|
|
{ name: "unifi", containers: ["unifi-controller", "mesh-unifi"] },
|
|
{ name: "portainer", containers: ["portainer", "mesh-portainer"] },
|
|
];
|
|
const CORE_ACE = new Set([
|
|
"postgres", "redis", "mssql",
|
|
"sonarr", "radarr", "lidarr", "jackett", "tautulli", "bookshelf",
|
|
"mosquitto", "influxdb", "grafana", "baserow", "nodered", "searxng", "unifi", "portainer",
|
|
]);
|
|
const GAPS_ACE = new Set(["plex", "bazarr", "nzbget", "qbittorrent", "ombi", "home-assistant", "letta"]);
|
|
|
|
/** The two workstations run one light module each, to prove a real module converges and joins the overlay. */
|
|
const LIGHT: Mod[] = [{ name: "portainer", containers: ["portainer", "mesh-portainer"] }];
|
|
const CORE_LIGHT = new Set(["portainer"]);
|
|
const GAPS_LIGHT = new Set<string>();
|
|
|
|
const PLAN: { node: string; mods: Mod[]; core: Set<string>; gaps: Set<string> }[] = [
|
|
{ node: "novox", mods: NOVOX, core: CORE_NOVOX, gaps: GAPS_NOVOX },
|
|
{ node: "ace", mods: ACE, core: CORE_ACE, gaps: GAPS_ACE },
|
|
{ node: "shanks", mods: LIGHT, core: CORE_LIGHT, gaps: GAPS_LIGHT },
|
|
{ node: "g14", mods: LIGHT, core: CORE_LIGHT, gaps: GAPS_LIGHT },
|
|
];
|
|
|
|
/**
|
|
* Host-port remaps (per module; host ports are per-VM so novox's and ace's never clash across nodes).
|
|
* The two SUBSTRATE collisions are the new ones: postgres 5432 and lavinmq 5672 are moved off the
|
|
* substrate store/broker's host ports, which only exist on novox because that is where the substrate
|
|
* runs. The rest break the novox web/app host-port collisions (route-proxy fronts 80/443).
|
|
*/
|
|
const REMAP: Record<string, Record<string, string>> = {
|
|
postgres: { "5432": "127.0.0.1:15432:5432" },
|
|
lavinmq: { "5672": "127.0.0.1:15673:5672" },
|
|
nextcloud: { "80": "8090:80" },
|
|
umami: { "3000": "3090:3000" },
|
|
invoicing: { "80": "8091:80", "9000": "9091:9000" },
|
|
qbittorrent: { "8080": "8090:8080" },
|
|
searxng: { "8080": "8092:8080" },
|
|
nzbget: { "6789": "6790:6789" },
|
|
};
|
|
|
|
/** Operator-provided app credentials, delivered as fake values through the real `secret accept` path. */
|
|
const CREDENTIALS: { node: string; module: string; name: string; crash: string }[] = [
|
|
{ node: "ace", module: "plex", name: "token", crash: "no Plex token" },
|
|
{ node: "ace", module: "bazarr", name: "api-key", crash: "no Bazarr API key" },
|
|
{ node: "ace", module: "ombi", name: "api-key", crash: "no Ombi API key" },
|
|
{ node: "ace", module: "home-assistant", name: "token", crash: "no Home Assistant token" },
|
|
{ node: "ace", module: "nzbget", name: "password", crash: "NZBGet not configured" },
|
|
{ node: "ace", module: "qbittorrent", name: "password", crash: "qBittorrent not configured" },
|
|
{ node: "novox", module: "umami", name: "admin", crash: "admin password is not set" },
|
|
];
|
|
|
|
/** Operator secrets for the credential modules that own-secret their whole app (mailu, de-spiegel). */
|
|
const OPERATOR_SECRETS: { node: string; module: string; name: string; value: string }[] = [
|
|
{ node: "novox", module: "mailu", name: "secret-key", value: "0123456789abcdef0123456789abcdef" },
|
|
{ node: "novox", module: "mailu", name: "admin", value: "MailuAdminFakePass123" },
|
|
{ node: "novox", module: "mailu", name: "api-token", value: "mailuapitokenfake0123456789abcd" },
|
|
{ node: "novox", module: "de-spiegel", name: "smtp-user", value: "despiegel-smtp-fake" },
|
|
{ node: "novox", module: "de-spiegel", name: "smtp-pass", value: "despiegel-pass-fake" },
|
|
{ node: "novox", module: "amqp-email-forwarder", name: "smtp-user", value: "eef-smtp-fake" },
|
|
{ node: "novox", module: "amqp-email-forwarder", name: "smtp-pass", value: "eef-pass-fake" },
|
|
];
|
|
|
|
let instanceId = "";
|
|
let held: HeldImage[] = [];
|
|
|
|
function quote(s: string): string {
|
|
return `'${s.replaceAll("'", `'\\''`)}'`;
|
|
}
|
|
|
|
async function on(machine: string, command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> {
|
|
const { stdout } = await exec(instanceId, machine, [
|
|
"sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`,
|
|
], timeoutMs);
|
|
const marker = stdout.lastIndexOf("__exit=");
|
|
if (marker < 0) return { out: stdout, ok: false };
|
|
return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" };
|
|
}
|
|
|
|
async function must(machine: string, command: string, timeoutMs?: number): Promise<string> {
|
|
const { out, ok } = await on(machine, command, timeoutMs);
|
|
if (!ok) throw new Error(`${machine}: ${command}\n${out}`);
|
|
return out;
|
|
}
|
|
|
|
/** The control plane, a container on novox (the anchor). */
|
|
async function mesh(command: string, timeoutMs?: number): Promise<string> {
|
|
return must(CONTROL, `docker exec mesh-control /mesh-control ${command}`, timeoutMs);
|
|
}
|
|
|
|
/** What a manifest's image reference becomes on the machine — ours by ID, everything else as written. */
|
|
function pinned(reference: string): string {
|
|
return onTheMachine(reference, held);
|
|
}
|
|
|
|
function bundleFor(images: HeldImage[]): string {
|
|
return substrateBundle(bundle, images);
|
|
}
|
|
|
|
function loadManifest(name: string): { manifest: string; broker: boolean } {
|
|
const path = resolve(catalogDir, name, "module.json");
|
|
const m = JSON.parse(readFileSync(path, "utf8")) as {
|
|
resources?: { type: string; image?: string; ports?: string[] }[];
|
|
};
|
|
const remap = REMAP[name] ?? {};
|
|
for (const r of m.resources ?? []) {
|
|
if (r.type !== "container") continue;
|
|
if (typeof r.image === "string") r.image = pinned(r.image);
|
|
if (Array.isArray(r.ports)) r.ports = r.ports.map((p) => remap[p] ?? p);
|
|
}
|
|
const manifest = JSON.stringify(m);
|
|
return { manifest, broker: manifest.includes("MESH_BROKER_FILE") };
|
|
}
|
|
|
|
function tokenFrom(said: string): string {
|
|
const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" "));
|
|
assert.ok(found, `no token in:\n${said}`);
|
|
return found;
|
|
}
|
|
|
|
interface NodeState {
|
|
reached: boolean;
|
|
applied: boolean;
|
|
current: boolean;
|
|
waiting: boolean;
|
|
wrong?: { outcome: string; refused?: string | undefined; failed?: { id: string; error: string }[] | undefined } | undefined;
|
|
raw: string;
|
|
}
|
|
|
|
async function nodeState(node: string): Promise<NodeState> {
|
|
const asked = await on(CONTROL, `docker exec mesh-control /mesh-control status --json`);
|
|
if (!asked.ok) return { reached: false, applied: false, current: false, waiting: false, raw: asked.out };
|
|
let state: {
|
|
wrong: { node: string; outcome: string; refused?: string; failed?: { id: string; error: string }[] }[];
|
|
waiting: { node: string }[];
|
|
reported: { node: string; outcome: string; current: boolean }[];
|
|
};
|
|
try {
|
|
state = JSON.parse(asked.out);
|
|
} catch {
|
|
return { reached: false, applied: false, current: false, waiting: false, raw: asked.out };
|
|
}
|
|
const word = state.reported.find((r) => r.node === node);
|
|
const bad = state.wrong.find((w) => w.node === node);
|
|
return {
|
|
reached: true,
|
|
applied: word?.outcome === "applied",
|
|
current: !!word?.current,
|
|
waiting: state.waiting.some((w) => w.node === node),
|
|
wrong: bad ? { outcome: bad.outcome, refused: bad.refused, failed: bad.failed } : undefined,
|
|
raw: asked.out,
|
|
};
|
|
}
|
|
|
|
async function psMapOf(node: string): Promise<Map<string, string>> {
|
|
const out = (await on(node, `docker ps -a --format '{{.Names}}\t{{.Status}}'`)).out;
|
|
const map = new Map<string, string>();
|
|
for (const line of out.split("\n")) {
|
|
const [n, ...rest] = line.split("\t");
|
|
if (n) map.set(n.trim(), rest.join("\t").trim());
|
|
}
|
|
return map;
|
|
}
|
|
|
|
/**
|
|
* ADR 0066: the internal CA is initialised FROM AN OPERATOR'S ROOT — it does not mint its own.
|
|
*
|
|
* So the bed has to be an operator. The material is made on the anchor with openssl and handed to
|
|
* the mesh through the real `secret accept` path, exactly as a person would: the mesh cannot invent
|
|
* a PEM, and the random 32 bytes it makes for an own-secret nobody supplied would leave step-ca
|
|
* crash-looping on a root key that is not a key.
|
|
*/
|
|
async function deliverCaRoot(): Promise<boolean> {
|
|
const made = await on(CONTROL, [
|
|
"set -e",
|
|
"mkdir -p /tmp/ca && cd /tmp/ca",
|
|
// No trailing newline on a password file: step-ca reads the file as the password itself.
|
|
"openssl rand -hex 16 | tr -d '\\n' > key-password",
|
|
"openssl ecparam -genkey -name prime256v1 -out root.unenc",
|
|
"openssl ec -in root.unenc -aes256 -passout file:key-password -out root.key",
|
|
"rm -f root.unenc",
|
|
"openssl req -x509 -new -key root.key -passin file:key-password -sha256 -days 3650" +
|
|
` -out root.crt -subj "/CN=Mesh Internal CA/O=Novox Mesh Lab"`,
|
|
// Readable by the control plane, which is not root. Its image is FROM scratch and runs as
|
|
// 65534, and `docker cp` keeps the ownership and mode a file had outside — openssl writes a
|
|
// private key 0600 root-owned, so the copy landed unreadable and `secret accept` failed with
|
|
// `open /ca-root-key: permission denied`. The CA then crash-looped on a root it never got.
|
|
// Chowning it inside the container is not available: there is no shell in there to do it with.
|
|
//
|
|
// Safe here and nowhere else: these three exist for the seconds between being written and
|
|
// being sealed to the machine, on a lab node, for a CA thrown away with the scenario.
|
|
"chmod 0644 /tmp/ca/root.crt /tmp/ca/root.key /tmp/ca/key-password",
|
|
"docker cp /tmp/ca/root.crt mesh-control:/ca-root-cert",
|
|
"docker cp /tmp/ca/root.key mesh-control:/ca-root-key",
|
|
"docker cp /tmp/ca/key-password mesh-control:/ca-root-key-password",
|
|
].join("\n"), 180_000);
|
|
if (!made.ok) {
|
|
console.log(`CA ROOT NOT MADE on ${CONTROL}:\n${made.out.split("\n").slice(-8).join("\n")}`);
|
|
return false;
|
|
}
|
|
for (const [name, file] of [
|
|
["root-cert", "/ca-root-cert"],
|
|
["root-key", "/ca-root-key"],
|
|
["root-key-password", "/ca-root-key-password"],
|
|
] as const) {
|
|
try {
|
|
await mesh(`secret accept ${CONTROL} step-ca ${name} --from ${file}`);
|
|
} catch (err) {
|
|
console.log(`CA ROOT ACCEPT FAILED (${name}): ${(err as Error).message.split("\n").slice(0, 3).join(" | ")}`);
|
|
return false;
|
|
}
|
|
}
|
|
return true;
|
|
}
|
|
|
|
/** What a module's manifest says its route label is, or "" if it contributes no route. */
|
|
function routeLabelOf(name: string): string {
|
|
const path = resolve(catalogDir, name, "module.json");
|
|
const m = JSON.parse(readFileSync(path, "utf8")) as {
|
|
contributes?: { route?: { label?: string } };
|
|
};
|
|
return m.contributes?.route?.label ?? "";
|
|
}
|
|
|
|
/** A node's overlay (mesh0) address, or "" if it has none yet. */
|
|
async function overlayAddr(node: string): Promise<string> {
|
|
const out = (await on(node, `ip -4 -o addr show mesh0 2>/dev/null | awk '{print $4}' | cut -d/ -f1`)).out;
|
|
return out.split("\n").map((l) => l.trim()).find(Boolean) ?? "";
|
|
}
|
|
|
|
// ==================================================================================================
|
|
// PHASE 1 — GENESIS. novox is brought into existence by the installer.
|
|
// ==================================================================================================
|
|
|
|
/** What genesis did, or where it stopped. */
|
|
interface GenesisResult {
|
|
ok: boolean;
|
|
/** `step 7 of 10, registry` — the installer's own words, so the bed reports the cause. */
|
|
step: string;
|
|
why: string;
|
|
report: string[];
|
|
}
|
|
|
|
/** The step a failed `mesh-bootstrap` names, as it prints it, or "" if it named none. */
|
|
function stepIn(said: string): string {
|
|
return said.match(/^mesh-bootstrap: (step \d+ of \d+, [a-z-]+):/m)?.[1] ?? "";
|
|
}
|
|
|
|
/**
|
|
* Put on the anchor what the installer needs to read, and run it.
|
|
*
|
|
* **This is the whole of what changed, and it is not a refactor.** The bed used to apply the
|
|
* substrate bundle itself, by hand, and then enrol four machines in one loop. It got the order
|
|
* right by accident and it modelled the wrong shape: an install procedure that exists only as a
|
|
* test fixture is exercised by whoever writes tests and never by whoever installs, which is why
|
|
* every bootstrap fault this year was found late (novox/hq ADR 0067). The anchor is now raised by
|
|
* running the same program a bare machine runs, and the bed only reads the result.
|
|
*
|
|
* It is run up to three times. Not to paper over a failure — every attempt's failing step is
|
|
* printed — but because `mesh-bootstrap` is idempotent by design and says so, and because the one
|
|
* thing here that fails for a reason which goes away by itself is a pull: the store, the broker and
|
|
* the registry come from the internet, through a household gateway's masquerade, and Docker Hub
|
|
* rate-limiting an anonymous pull is not this mesh's fault. A re-run is the retry, and it is the
|
|
* retry the installer's own documentation names.
|
|
*/
|
|
async function genesis(images: HeldImage[]): Promise<GenesisResult> {
|
|
const report: string[] = ["================ GENESIS: novox becomes a mesh of one ================"];
|
|
const stop = (step: string, why: string): GenesisResult => {
|
|
report.push(`\nSTOPPED at ${step || "(no step named)"}: ${why}`);
|
|
return { ok: false, step, why, report };
|
|
};
|
|
|
|
// The installer, beside the host binary. Everything else on this machine was placed by `raise`;
|
|
// this one is placed here because only the anchor is bootstrapped.
|
|
const name = await instanceNameOf(instanceId, CONTROL);
|
|
const version = await placeBootstrap(name, CONTROL, installer as string,
|
|
(m) => console.log(`genesis:${m}`));
|
|
report.push(` installer ${version} at ${BOOTSTRAP_PATH}`);
|
|
|
|
// The catalogue. `mesh-bootstrap --catalog` reads manifests from a CHECKOUT on the machine,
|
|
// because at this moment the mesh has no forge, no build machine and — until step 7 finishes —
|
|
// no registry. A manifest is a file, and somebody has to have put it there.
|
|
await must(CONTROL, `mkdir -p ${CATALOGUE_MODULES.map((m) => `${CATALOGUE_ON_MACHINE}/modules/${m}`).join(" ")}`);
|
|
for (const module of CATALOGUE_MODULES) {
|
|
const from = resolve(catalogDir, module, "module.json");
|
|
assert.ok(existsSync(from), `the catalogue has no ${module}/module.json at ${from}`);
|
|
await push(instanceId, CONTROL, from, `${CATALOGUE_ON_MACHINE}/modules/${module}/module.json`);
|
|
}
|
|
report.push(` catalogue ${CATALOGUE_MODULES.join(", ")} at ${CATALOGUE_ON_MACHINE}`);
|
|
|
|
// The substrate TEMPLATE — not the bundle. The installer produces the bundle from it: it replaces
|
|
// the control plane's image with the id of the image it carries, renames that container
|
|
// `temp-mesh-control`, and writes the result where a person can read it.
|
|
//
|
|
// Two substitutions still happen here, and both belong to the bed rather than to the installer.
|
|
// The example names three images at a registry the lab no longer raises: the store and the broker
|
|
// become the upstream references mesh-catalog pins (harness), and the machine pulls them over its
|
|
// uplink like any first node. The third, mesh-control, is deliberately LEFT naming that dead
|
|
// registry — the installer overwrites it, and leaving it proves that it does.
|
|
//
|
|
// And the broker's advertised address. The template hardcodes 192.0.2.10:5671, the old
|
|
// separate-anchor address; a token carries MESH_BROKER_ADDRESS verbatim as the endpoint an
|
|
// enrolling node dials, so with the substrate on novox it must be novox's own public address or
|
|
// every node would enrol against a dead one. The installer refuses to guess this and says so
|
|
// loudly, which is right — it does not know what this machine is called from outside.
|
|
const template = bundleFor(images).replaceAll("192.0.2.10:5671", "192.0.2.20:5671");
|
|
const local = join(tmpdir(), `mesh-lab-substrate-${process.pid}.lock`);
|
|
writeFileSync(local, template);
|
|
await push(instanceId, CONTROL, local, "/tmp/substrate-template.lock");
|
|
|
|
const command = [
|
|
BOOTSTRAP_PATH,
|
|
`--bundle /tmp/substrate-template.lock`,
|
|
`--catalog ${CATALOGUE_ON_MACHINE}`,
|
|
`--node ${CONTROL}`,
|
|
`--registry ${MESH_REGISTRY}`,
|
|
`--host ${HOST_PATH}`,
|
|
// The lab has no unit to supervise the host with, and the installer refuses to invent one — a
|
|
// unit file is a packaging decision. This is the arrangement it offers instead, and it is loud
|
|
// about what it is: a host started this way does not survive a reboot.
|
|
`--host-in-background`,
|
|
].join(" ");
|
|
|
|
let said = "";
|
|
let step = "";
|
|
for (let attempt = 1; attempt <= 3; attempt++) {
|
|
const ran = await on(CONTROL, command, 2_400_000);
|
|
said = ran.out;
|
|
console.log(`\n---- mesh-bootstrap on ${CONTROL} (attempt ${attempt}) ----\n${said}`);
|
|
if (ran.ok) {
|
|
step = "";
|
|
break;
|
|
}
|
|
step = stepIn(said);
|
|
if (attempt < 3) {
|
|
console.log(`genesis attempt ${attempt} stopped at ${step || "an unnamed step"}; ` +
|
|
`re-running in 30s — every step it already did will say so`);
|
|
await new Promise((r) => setTimeout(r, 30_000));
|
|
}
|
|
}
|
|
if (step) {
|
|
return stop(step, said.split("\n").filter(Boolean).slice(-6).join("\n"));
|
|
}
|
|
|
|
// ------------------------------------------------------------------------------------------
|
|
// And now the only thing that matters: is novox a WORKING MESH OF ONE? Asked of the machine,
|
|
// never inferred from the installer exiting zero (novox/hq ADR 0018).
|
|
// ------------------------------------------------------------------------------------------
|
|
|
|
// 1. The control plane answers. Asked of the PERMANENT container by name — `status` opens all
|
|
// three stores, so a reply proves the connections it was given are the substrate's own.
|
|
const answered = await on(CONTROL, `docker exec mesh-control /mesh-control status`, 60_000);
|
|
report.push(` control plane ${answered.ok ? answered.out.split("\n")[0] : "NO ANSWER"}`);
|
|
if (!answered.ok) return stop("after step 10", `mesh-control does not answer:\n${answered.out}`);
|
|
|
|
// 2. The registry replies on /v2/ — the registry API's own "yes, I am one and I am ready". A
|
|
// container that is up is not a registry that serves.
|
|
const v2 = await on(CONTROL,
|
|
`curl -s -o /dev/null -w '%{http_code}' --max-time 10 http://${MESH_REGISTRY}/v2/`);
|
|
const v2Code = v2.out.trim();
|
|
report.push(` registry /v2/ ${v2Code || "no answer"}`);
|
|
if (v2Code !== "200") return stop("after step 10", `the mesh's own registry answered ${v2Code || "nothing"}`);
|
|
|
|
// 3. THE PIVOT COMPLETED. ADR 0067 states this check in as many words: after installing, the
|
|
// running control plane's image is pinned by a digest THE MESH'S OWN REGISTRY ASSIGNED — not
|
|
// by an image id. If it is still an image id, the substrate's container is what is running,
|
|
// nothing was published, and this mesh can never roll out its own upgrades.
|
|
const pinnedTo = (await on(CONTROL, `docker inspect --format '{{.Config.Image}}' mesh-control`))
|
|
.out.trim();
|
|
report.push(` pinned to ${pinnedTo || "(nothing)"}`);
|
|
if (/^sha256:[0-9a-f]{64}$/.test(pinnedTo)) {
|
|
return stop("after step 10",
|
|
`mesh-control is running from ${pinnedTo}, which is an IMAGE ID — the digest of the image's ` +
|
|
`own configuration, which no registry ever served. The pivot did not happen: what is ` +
|
|
`running is the image the installer carried, not one this mesh published, so this mesh ` +
|
|
`cannot upgrade itself (novox/hq ADR 0067, "the pivot completed").`);
|
|
}
|
|
if (!new RegExp(`^${MESH_REGISTRY.replaceAll(".", "\\.")}/mesh-control@sha256:[0-9a-f]{64}$`)
|
|
.test(pinnedTo)) {
|
|
return stop("after step 10",
|
|
`mesh-control is running from ${pinnedTo || "nothing this bed could read"}, which is not a ` +
|
|
`digest assigned by ${MESH_REGISTRY}.`);
|
|
}
|
|
|
|
// 3b. And the registry really serves it, asked of the registry rather than of the container. A
|
|
// reference is a claim; a tag list is the registry agreeing.
|
|
const tags = await on(CONTROL,
|
|
`curl -s --max-time 10 http://${MESH_REGISTRY}/v2/mesh-control/tags/list`);
|
|
report.push(` registry holds ${tags.out.trim() || "nothing"}`);
|
|
if (!tags.out.includes("genesis")) {
|
|
return stop("after step 10",
|
|
`${MESH_REGISTRY} does not serve mesh-control, so the digest the container is pinned to ` +
|
|
`names an image nothing can pull: ${tags.out.trim()}`);
|
|
}
|
|
|
|
// 4. The temporary control plane is GONE. Two control planes is the half-finished state, and the
|
|
// name is the audit: a machine running mesh-control and not temp-mesh-control has pivoted.
|
|
const temp = await on(CONTROL, `docker inspect --format '{{.State.Status}}' temp-mesh-control`);
|
|
report.push(` temp-mesh-control ${temp.ok ? `STILL HERE (${temp.out.trim()})` : "gone"}`);
|
|
if (temp.ok) {
|
|
return stop("after step 10",
|
|
`temp-mesh-control is still ${temp.out.trim()}. Two control planes are consuming this ` +
|
|
`mesh's broker queues; neither is wrong and the pivot is not finished.`);
|
|
}
|
|
|
|
// 5. And the mesh has heard from its one node. Everything the join phase does next depends on it.
|
|
const nodes = await on(CONTROL, `docker exec mesh-control /mesh-control node list`);
|
|
report.push(` node list ${nodes.out.trim().split("\n").join(" | ")}`);
|
|
const line = nodes.out.split("\n").map((l) => l.trim()).find((l) => l.startsWith(`${CONTROL} `));
|
|
if (!line || !/^\S+\s+here\b/.test(line)) {
|
|
return stop("after step 10",
|
|
`the mesh has not heard from ${CONTROL}: ${line ?? "it has no record of it at all"}`);
|
|
}
|
|
|
|
report.push(`\nVERDICT: ${CONTROL} is a working mesh of one, bootstrapped through the installer.`);
|
|
return { ok: true, step: "", why: "", report };
|
|
}
|
|
|
|
// ==================================================================================================
|
|
// PHASE 2 — JOINING. Everything else is a machine joining a mesh that already exists.
|
|
// ==================================================================================================
|
|
|
|
/**
|
|
* ace, shanks and g14 join. Host binary plus a token — no bootstrap, no substrate, no registry.
|
|
*
|
|
* **novox is not in this loop.** It was enrolled by the installer, as part of becoming a mesh, and
|
|
* enrolling it again would present the mesh with a second identity for a node it already knows —
|
|
* which `mesh-host enrol` refuses, and rightly.
|
|
*
|
|
* The home nodes reach novox's public 192.0.2.20:5671 by dialling OUT through the household
|
|
* gateway, so the enrol itself is the first proof that outbound home→public works.
|
|
*/
|
|
async function joinTheMesh(): Promise<void> {
|
|
// ADR 0066: said as soon as the record exists, because everything routed is composed from it. A
|
|
// node that faces the outside has one; the workstations do not, and are given none. The anchor's
|
|
// is set here rather than in genesis because it is a fact about the mesh, not part of raising
|
|
// one — and the installer has an opinion about neither.
|
|
const anchorDomain = PUBLIC_DOMAIN[CONTROL];
|
|
if (anchorDomain) await mesh(`node public-domain ${CONTROL} ${anchorDomain}`);
|
|
|
|
for (const machine of HOME_NODES) {
|
|
await mesh(`node add ${machine}`);
|
|
const domain = PUBLIC_DOMAIN[machine];
|
|
if (domain) await mesh(`node public-domain ${machine} ${domain}`);
|
|
const token = tokenFrom(await mesh(`token issue --node ${machine}`));
|
|
const said = await must(machine, `${HOST_PATH} enrol --token ${quote(token)}`, 180_000);
|
|
assert.match(said, new RegExp(`enrolled as ${machine}`), said);
|
|
await must(machine, `nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`);
|
|
}
|
|
}
|
|
|
|
before(async () => {
|
|
if (skip) return;
|
|
assert.ok(existsSync(catalogDir), `mesh-catalog modules not found at ${catalogDir}`);
|
|
|
|
const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), {
|
|
onProgress: (m) => console.log(`raise: ${m}`),
|
|
...(FIXED_ID ? { instanceId: FIXED_ID } : {}),
|
|
});
|
|
instanceId = raised.instanceId;
|
|
held = raised.images;
|
|
console.log(`INSTANCE ${instanceId}${KEEP ? " (KEEP — will be left standing)" : ""}`);
|
|
|
|
// ---- PHASE 1, and it GATES phase 2 ---------------------------------------------------------
|
|
//
|
|
// Caught rather than allowed to propagate, so that a failure BEFORE the installer ran — a
|
|
// manifest that is not where the bed thought, a binary that would not copy — is reported in the
|
|
// same shape as one the installer itself named, instead of as a bare stack trace from a helper.
|
|
let genesisResult: GenesisResult;
|
|
try {
|
|
genesisResult = await genesis(raised.images);
|
|
} catch (err) {
|
|
genesisResult = {
|
|
ok: false,
|
|
step: "getting the anchor ready to be bootstrapped — the installer never ran",
|
|
why: (err as Error).message,
|
|
report: ["================ GENESIS: novox becomes a mesh of one ================"],
|
|
};
|
|
}
|
|
console.log(genesisResult.report.join("\n"));
|
|
assert.ok(genesisResult.ok,
|
|
`GENESIS FAILED — ${genesisResult.step || "no step named"}.\n\n${genesisResult.why}\n\n` +
|
|
`No other machine was asked to join. A second machine joining a mesh that is not ready is a ` +
|
|
`different failure with a different cause, and running it now would bury this one under it.\n\n` +
|
|
genesisResult.report.join("\n"));
|
|
|
|
// ---- PHASE 2 --------------------------------------------------------------------------------
|
|
await joinTheMesh();
|
|
|
|
// The operator provides ace's media library (ADR 0051 accesses confirm the paths, create nothing).
|
|
await must("ace", `mkdir -p ${MEDIA_DIRS.join(" ")}`);
|
|
}, { timeout: 5_400_000 });
|
|
|
|
after(async () => {
|
|
if (KEEP) {
|
|
console.log(`\nLEFT STANDING: ${instanceId} — not destroyed (MESH_LAB_KEEP).`);
|
|
return;
|
|
}
|
|
if (instanceId) await destroy(instanceId);
|
|
await destroyAll(`${SCENARIO}-`);
|
|
}, { timeout: 900_000 });
|
|
|
|
test("the full mesh forms across the access point and both server sets converge", {
|
|
skip, timeout: 5_400_000,
|
|
}, async () => {
|
|
// ================================================================================================
|
|
// PHASE A — THE HEADLINE. Place the overlay (hub on novox at its public endpoint; the home nodes
|
|
// dial out, no endpoint of their own), assign networking to every node, push, and VERIFY the tunnel
|
|
// forms ACROSS the gateway. This runs BEFORE any heavy module, so the cross-segment-overlay verdict
|
|
// is captured whatever the module convergence then does.
|
|
// ================================================================================================
|
|
await mesh("overlay place novox --hub --endpoint 192.0.2.20:51820 --site hosting");
|
|
for (const node of HOME_NODES) await mesh(`overlay place ${node} --site home`);
|
|
for (const node of NODES) await mesh(`assign ${node} networking`);
|
|
for (const node of NODES) {
|
|
try {
|
|
await mesh(`push ${node}`, 180_000);
|
|
} catch (err) {
|
|
console.log(`networking push rejected (${node}): ${(err as Error).message.split("\n").slice(0, 4).join(" | ")}`);
|
|
}
|
|
}
|
|
|
|
// Give the home nodes time to dial the hub and complete a handshake through the NAT.
|
|
const overlay: Record<string, string> = {};
|
|
const deadline = Date.now() + 300_000;
|
|
while (Date.now() < deadline) {
|
|
for (const node of NODES) if (!overlay[node]) overlay[node] = await overlayAddr(node);
|
|
if (NODES.every((n) => overlay[n])) break;
|
|
await new Promise((r) => setTimeout(r, 8000));
|
|
}
|
|
// A little longer for handshakes to settle (keepalive interval).
|
|
await new Promise((r) => setTimeout(r, 30000));
|
|
|
|
const overlayReport: string[] = ["================ CROSS-SEGMENT OVERLAY (the headline) ================"];
|
|
for (const node of NODES) overlayReport.push(` ${node.padEnd(8)} mesh0 = ${overlay[node] || "NONE"}`);
|
|
|
|
// The hub's WireGuard peers and their handshakes, from novox.
|
|
const hubWg = (await on("novox", `wg show 2>&1 || echo 'wg tool absent'`)).out;
|
|
overlayReport.push(`\n---- novox (hub) wg show ----\n${hubWg}`);
|
|
|
|
// From each home node: its wg peer state (endpoint should be 192.0.2.20:51820, with a recent
|
|
// handshake) AND a ping to novox's overlay address — the functional proof the tunnel carries
|
|
// traffic across the gateway.
|
|
const overlayFormed: Record<string, boolean> = {};
|
|
const novoxOverlay = overlay["novox"] ?? "";
|
|
for (const node of HOME_NODES) {
|
|
const wg = (await on(node, `wg show 2>&1 || echo 'wg tool absent'`)).out;
|
|
const handshake = (await on(node, `wg show all latest-handshakes 2>/dev/null | awk '{print $2}' | sort -rn | head -1`)).out.trim();
|
|
const ping = novoxOverlay
|
|
? await on(node, `ping -c 3 -W 2 ${novoxOverlay} 2>&1 | tail -3`)
|
|
: { out: "novox has no overlay address to ping", ok: false };
|
|
const handshakeSecs = Number(handshake) || 0;
|
|
// Formed = we can reach novox over the overlay from this home node (traffic across the NAT).
|
|
overlayFormed[node] = ping.ok;
|
|
overlayReport.push(`\n---- ${node} (home) ----`);
|
|
overlayReport.push(wg.split("\n").map((l) => ` ${l}`).join("\n"));
|
|
overlayReport.push(` latest-handshake epoch: ${handshake || "none"}${handshakeSecs ? "" : " (no handshake recorded)"}`);
|
|
overlayReport.push(` ping novox(${novoxOverlay}) over overlay: ${ping.ok ? "REPLIES" : "NO REPLY"}`);
|
|
overlayReport.push(ping.out.split("\n").map((l) => ` ${l}`).join("\n"));
|
|
}
|
|
const anyHomeFormed = HOME_NODES.some((n) => overlayFormed[n]);
|
|
const allHomeFormed = HOME_NODES.every((n) => overlayFormed[n]);
|
|
overlayReport.push(`\nVERDICT: overlay across the access point ${allHomeFormed ? "FORMED for all home nodes" : anyHomeFormed ? "FORMED for some home nodes" : "DID NOT FORM"}.`);
|
|
const overlaySummary = overlayReport.join("\n");
|
|
console.log(overlaySummary);
|
|
|
|
// ================================================================================================
|
|
// PHASE B — converge the full node sets on top of the overlay.
|
|
// ================================================================================================
|
|
const added = new Map<string, boolean>();
|
|
async function ensureAdded(name: string): Promise<boolean> {
|
|
const known = added.get(name);
|
|
if (known !== undefined) return known;
|
|
const { manifest, broker } = loadManifest(name);
|
|
await must(CONTROL, `printf %s ${quote(manifest)} > /tmp/${name}.json && docker cp /tmp/${name}.json mesh-control:/${name}.json`);
|
|
await mesh(`module add /${name}.json`);
|
|
added.set(name, broker);
|
|
return broker;
|
|
}
|
|
|
|
const assigned: Record<string, Set<string>> = { novox: new Set(), ace: new Set(), shanks: new Set(), g14: new Set() };
|
|
const refused: Record<string, { name: string; why: string }[]> = { novox: [], ace: [], shanks: [], g14: [] };
|
|
for (const { node, mods } of PLAN) {
|
|
for (const { name } of mods) {
|
|
try {
|
|
const broker = await ensureAdded(name);
|
|
if (broker) await mesh(`module issue ${name} --node ${node}`);
|
|
await mesh(`assign ${node} ${name}`);
|
|
assigned[node]!.add(name);
|
|
} catch (err) {
|
|
const why = (err as Error).message.split("\n").map((l) => l.trim()).filter(Boolean).slice(1, 5).join(" | ");
|
|
refused[node]!.push({ name, why });
|
|
console.log(`NOT ASSIGNED ${node}/${name}: ${why}`);
|
|
}
|
|
}
|
|
}
|
|
|
|
// Operator-provided app credentials (own-secrets), delivered as fake values through `secret accept`.
|
|
const credentialDelivered = new Map<string, boolean>();
|
|
for (const name of new Set(CREDENTIALS.map((c) => c.name))) {
|
|
await must(CONTROL, `printf %s ${quote(`fake-${name}-value`)} > /tmp/fake-${name} && docker cp /tmp/fake-${name} mesh-control:/fake-${name}`);
|
|
}
|
|
for (const c of CREDENTIALS) {
|
|
if (!assigned[c.node]!.has(c.module)) {
|
|
credentialDelivered.set(`${c.node}/${c.module}`, false);
|
|
continue;
|
|
}
|
|
try {
|
|
await mesh(`secret accept ${c.node} ${c.module} ${c.name} --from /fake-${c.name}`);
|
|
credentialDelivered.set(`${c.node}/${c.module}`, true);
|
|
} catch (err) {
|
|
credentialDelivered.set(`${c.node}/${c.module}`, false);
|
|
console.log(`CREDENTIAL ACCEPT FAILED ${c.node}/${c.module}: ${(err as Error).message.split("\n").slice(0, 3).join(" | ")}`);
|
|
}
|
|
}
|
|
// Whole-app own-secrets (mailu/de-spiegel/amqp-email-forwarder).
|
|
for (const s of OPERATOR_SECRETS) {
|
|
if (!assigned[s.node]!.has(s.module)) continue;
|
|
try {
|
|
const inControl = `/secret-${s.module}-${s.name}`;
|
|
await must(CONTROL, `printf %s ${quote(s.value)} > /tmp${inControl} && docker cp /tmp${inControl} mesh-control:${inControl}`);
|
|
await mesh(`secret accept ${s.node} ${s.module} ${s.name} --from ${inControl}`);
|
|
} catch (err) {
|
|
console.log(`OPERATOR SECRET FAILED ${s.node}/${s.module}/${s.name}: ${(err as Error).message.split("\n").slice(0, 2).join(" | ")}`);
|
|
}
|
|
}
|
|
|
|
// ADR 0066: the CA's root, before the push that would otherwise deliver a random 32 bytes for it.
|
|
const caRootDelivered = assigned["novox"]!.has("step-ca") ? await deliverCaRoot() : false;
|
|
if (!caRootDelivered) console.log("ADR 0066: no operator root delivered; step-ca cannot initialise.");
|
|
|
|
// ONE push per node (workstations first — cheap — then the heavy service nodes).
|
|
const pushError: Record<string, string> = {};
|
|
for (const node of ["shanks", "g14", "novox", "ace"]) {
|
|
try {
|
|
await mesh(`push ${node}`, 300_000);
|
|
} catch (err) {
|
|
pushError[node] = (err as Error).message;
|
|
console.log(`PUSH REJECTED (${node}):\n${pushError[node]!.split("\n").slice(0, 6).join("\n")}`);
|
|
}
|
|
}
|
|
|
|
// Wait for each node's CORE containers to come up (all nodes pull concurrently from the one registry).
|
|
const psMaps: Record<string, Map<string, string>> = { novox: new Map(), ace: new Map(), shanks: new Map(), g14: new Map() };
|
|
for (const { node, mods, core } of PLAN) {
|
|
if (pushError[node]) continue;
|
|
const coreContainers = mods.filter((m) => core.has(m.name) && assigned[node]!.has(m.name)).flatMap((m) => m.containers);
|
|
const until = Date.now() + 3_000_000;
|
|
while (Date.now() < until) {
|
|
psMaps[node] = await psMapOf(node);
|
|
if (coreContainers.every((c) => (psMaps[node]!.get(c) ?? "").startsWith("Up"))) break;
|
|
await new Promise((r) => setTimeout(r, 10000));
|
|
}
|
|
}
|
|
await new Promise((r) => setTimeout(r, 20000));
|
|
|
|
// ================================================================================================
|
|
// Per-node convergence report.
|
|
// ================================================================================================
|
|
const users = (await on("novox", `docker exec mesh-broker lavinmqctl list_users 2>&1`)).out;
|
|
const allProblems: string[] = [];
|
|
const report: string[] = ["================ FULL MESH CONVERGENCE ================"];
|
|
|
|
for (const { node, mods, core, gaps } of PLAN) {
|
|
const psMap = psMaps[node] = await psMapOf(node);
|
|
const st = await nodeState(node);
|
|
const running = (name: string): boolean => (psMap.get(name) ?? "").startsWith("Up");
|
|
const ranOnce = (name: string): boolean => !psMap.has(name) || /^(Up|Exited \(0\))/.test(psMap.get(name) ?? "");
|
|
const failedResources = st.wrong?.failed ?? [];
|
|
|
|
report.push(`\n---- node ${node}: reached=${st.reached} applied=${st.applied} current=${st.current} waiting=${st.waiting} ----`);
|
|
if (pushError[node]) report.push(` PUSH REJECTED: ${pushError[node]!.split("\n").slice(0, 6).join("\n ")}`);
|
|
if (st.wrong) {
|
|
report.push(` NODE WRONG: outcome=${st.wrong.outcome}`);
|
|
for (const f of failedResources) report.push(` failed ${f.id}: ${f.error}`);
|
|
}
|
|
for (const r of refused[node]!) report.push(` REFUSED ${r.name}: ${r.why}`);
|
|
|
|
const coreFailures: string[] = [];
|
|
for (const mod of mods) {
|
|
if (!assigned[node]!.has(mod.name)) continue;
|
|
if (mod.node) {
|
|
report.push(` ${core.has(mod.name) ? "*" : " "} ${mod.name.padEnd(20)} ${core.has(mod.name) ? "CORE" : "gap "} node-service`);
|
|
continue;
|
|
}
|
|
const states = mod.containers.map((c) => `${c}:${running(c) ? "UP" : (psMap.get(c) ?? "MISSING")}`);
|
|
const ok = mod.containers.every(running) && (mod.runOnce ?? []).every(ranOnce);
|
|
const tag = core.has(mod.name) ? (ok ? "OK " : "FAIL") : (ok ? "ok " : "GAP ");
|
|
report.push(` ${core.has(mod.name) ? "*" : " "} ${mod.name.padEnd(20)} ${tag} ${states.join(" ")}`);
|
|
if (core.has(mod.name) && !ok) coreFailures.push(mod.name);
|
|
}
|
|
const issuedHere = mods.filter((m) => new RegExp(`${node}-${m.name}\\b`).test(users)).length;
|
|
report.push(` broker accounts: ${issuedHere} present for ${node}`);
|
|
|
|
const gapOwnerOf = (f: { id: string; error: string }): string => {
|
|
const m = f.error.match(/applying "([^".]+)\./);
|
|
return m?.[1] ?? (f.id.split(".")[0] ?? "");
|
|
};
|
|
if (pushError[node]) allProblems.push(`${node}: push rejected`);
|
|
if (coreFailures.length) allProblems.push(`${node}: CORE not converged: ${coreFailures.join(", ")}`);
|
|
const nonGapFailed = failedResources.filter((f) => !gaps.has(gapOwnerOf(f)));
|
|
if (nonGapFailed.length) allProblems.push(`${node}: non-gap resource failed: ${nonGapFailed.map((f) => `${f.id} (${f.error.slice(0, 60)})`).join(", ")}`);
|
|
}
|
|
|
|
const summary = report.join("\n");
|
|
console.log(summary);
|
|
|
|
// Diagnostics for any CORE container that did not come up.
|
|
for (const { node, mods, core } of PLAN) {
|
|
const psMap = psMaps[node]!;
|
|
for (const mod of mods) {
|
|
if (!core.has(mod.name) || !assigned[node]!.has(mod.name)) continue;
|
|
for (const c of mod.containers) {
|
|
if (psMap.has(c) && !(psMap.get(c) ?? "").startsWith("Up")) {
|
|
console.log(`\n---- ${node} logs: ${c} (${psMap.get(c)}) ----\n${(await on(node, `docker logs ${c} 2>&1 | tail -25`)).out}`);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// ================================================================================================
|
|
// ADR 0066 — ROUTE NAMES AND THE INTERNAL CA. Additive, and deliberately only the cheap half.
|
|
//
|
|
// What is checked here is the part that is DECIDED and costs one file read: a module contributes a
|
|
// LABEL, the node carries a PUBLIC DOMAIN, and the mesh joins them — `<label>.<public-domain>`,
|
|
// with `@` composing to the bare domain. That join is what makes a routed module reachable at all,
|
|
// and before this bed set a public domain it composed to nothing on every node, silently.
|
|
//
|
|
// What is NOT checked here is issuance: whether route-proxy actually obtains a certificate from
|
|
// step-ca over ACME. That path is being fixed in mesh-control as this is written, and a bed that
|
|
// gated on it would be reporting somebody else's in-flight work as this bed's failure.
|
|
// ================================================================================================
|
|
const adr: string[] = ["================ ADR 0066: LABELLED ROUTES ================"];
|
|
|
|
const wanted: { node: string; module: string; label: string; name: string }[] = [];
|
|
for (const { node, mods } of PLAN) {
|
|
const domain = PUBLIC_DOMAIN[node];
|
|
if (!domain) continue;
|
|
for (const { name } of mods) {
|
|
if (!assigned[node]!.has(name)) continue;
|
|
const label = routeLabelOf(name);
|
|
if (!label) continue;
|
|
wanted.push({ node, module: name, label, name: label === "@" ? domain : `${label}.${domain}` });
|
|
}
|
|
}
|
|
|
|
// The composed names as the MESH wrote them, read from the proxy's own received-routes file —
|
|
// the mesh's answer, on the machine, rather than this test's arithmetic checked against itself.
|
|
const routesFile = (await on("novox", `cat /var/lib/route-proxy/routes/mesh.json 2>&1`)).out;
|
|
const missing: string[] = [];
|
|
const composed: typeof wanted = [];
|
|
for (const w of wanted.filter((w) => w.node === "novox")) {
|
|
const present = routesFile.includes(`"${w.name}"`);
|
|
adr.push(` ${w.module.padEnd(20)} label ${w.label.padEnd(10)} -> ${w.name.padEnd(28)} ${present ? "COMPOSED" : "MISSING"}`);
|
|
if (present) composed.push(w);
|
|
else missing.push(`${w.module} (${w.label} -> ${w.name})`);
|
|
}
|
|
|
|
// And that the proxy answers for one of them. Over HTTP, on the anchor: a certificate is the
|
|
// issuance question, and this one is only whether the name reaches the proxy at all.
|
|
const probe = composed[0];
|
|
const servedCode = probe
|
|
? (await on("novox", `curl -s -o /dev/null -w '%{http_code}' --max-time 10 -H 'Host: ${probe.name}' http://127.0.0.1/`)).out.trim()
|
|
: "";
|
|
adr.push(`\n proxy answers for ${probe?.name ?? "(nothing composed)"}: ${servedCode || "no answer"}`);
|
|
adr.push(` operator root delivered to step-ca: ${caRootDelivered}`);
|
|
adr.push(` step-ca container: ${psMaps["novox"]?.get("step-ca") ?? "MISSING"}`);
|
|
console.log(adr.join("\n"));
|
|
|
|
// ================================================================================================
|
|
// GATING. The headline gates: the cross-segment overlay must FORM for at least one home node
|
|
// (that is the thing this bed exists to prove). Convergence gates on each node's CORE and no
|
|
// non-gap resource failing. The KEEP run is about leaving a browsable instance, so its convergence
|
|
// is reported but not hard-gated; a normal run gates fully.
|
|
// ================================================================================================
|
|
assert.ok(anyHomeFormed,
|
|
`the overlay did NOT form across the access point — no home node could reach novox over the overlay:\n${overlaySummary}`);
|
|
|
|
if (!KEEP) {
|
|
// ADR 0066, the cheap half. Reported on a KEEP run like everything else there.
|
|
assert.deepEqual(missing, [],
|
|
`these routed modules composed no name — a label with no public domain to join it to is a ` +
|
|
`module nothing can reach, and it fails silently:\n${adr.join("\n")}\n\nroutes file:\n${routesFile}`);
|
|
assert.ok(probe && servedCode !== "" && servedCode !== "000",
|
|
`the proxy did not answer for ${probe?.name ?? "any composed name"} (got "${servedCode}"). ` +
|
|
`The name composes, so this is the proxy, not the join:\n${adr.join("\n")}`);
|
|
}
|
|
|
|
if (!KEEP) {
|
|
assert.deepEqual(allProblems, [], `the full mesh did not converge:\n ${allProblems.join("\n ")}\n\n${summary}`);
|
|
} else if (allProblems.length) {
|
|
console.log(`\nCONVERGENCE PROBLEMS (reported, not gated on a KEEP run):\n ${allProblems.join("\n ")}`);
|
|
}
|
|
});
|