jschoubben
cf6fcdff3c
The control plane's manifest comes out of the build the installer runs (novox/hq ADR 0069, 04-ISSUES/072); the catalogue no longer holds a copy, and a bed that demanded one would stop a genesis that is about to succeed.
1091 lines
59 KiB
TypeScript
1091 lines
59 KiB
TypeScript
/**
|
|
* The FULL mesh in its REAL production shape: two segments, one access point, one overlay — and the
|
|
* first multi-segment whole-mesh bed. It rewrites the flat three-node whole-mesh-full (separate
|
|
* anchor, everything on one public segment) into what production actually is:
|
|
*
|
|
* hosting (public) home (private, behind a NAT access point)
|
|
* novox 192.0.2.20 — the ANCHOR: ace 10.99.1.10 the home server, media/IoT set
|
|
* foundation (store/broker/ shanks 10.99.1.20 workstation (light: portainer only)
|
|
* control) + the whole novox g14 10.99.1.30 workstation (light: portainer only)
|
|
* set + overlay hub + ingress
|
|
*
|
|
* There is NO separate anchor: novox IS the anchor. The foundation runs on novox, and novox also
|
|
* enrols as a node and receives its own service set — the foundation host and a service node at once.
|
|
*
|
|
* TWO ACTS, AND THE BED NOW DISTINGUISHES THEM (novox/hq ADR 0067).
|
|
*
|
|
* GENESIS — novox is brought into existence by `mesh-bootstrap`, the installer, run on the
|
|
* machine exactly as a person would run it on a bare one: preflight, load the carried
|
|
* control-plane image, write the bundle, apply, verify, enrol itself, install the registry
|
|
* module, push the control-plane image into it, reinstall the control plane as an ordinary
|
|
* module pinned to the digest that push produced, retire the temporary one. Afterwards novox
|
|
* is a WORKING MESH OF ONE, and this bed asserts exactly that before going any further.
|
|
*
|
|
* JOINING — ace, shanks and g14 then join a mesh that already exists: host binary, token,
|
|
* `enrol`, run the agent. No bootstrap, no foundation, no registry. novox is NOT enrolled again.
|
|
*
|
|
* The bed used to do neither. It applied the foundation bundle itself and looped enrolment over all
|
|
* four machines as one continuous operation — which got the order right by accident and modelled
|
|
* the wrong shape, and is why ADR 0067's own acceptance check ("the bed bootstraps through the
|
|
* installer rather than around it") went unmet. Genesis GATES joining: if it stops, the bed says
|
|
* which of the installer's ten steps it stopped at and goes no further, because a second machine
|
|
* joining a mesh that is not ready is a different failure and must not be mistaken for this one.
|
|
*
|
|
* THE THING THIS BED EXISTS TO PROVE (the flat beds never could): does the WireGuard overlay tunnel
|
|
* FORM across the access point? A home node (ace/shanks/g14) dials novox's PUBLIC hub endpoint
|
|
* 192.0.2.20:51820/udp OUT through the household gateway's masquerade; the handshake has to complete
|
|
* through that NAT and the keepalive has to hold the hole open. Phase A drives exactly this and
|
|
* verifies it — WireGuard handshake state AND a ping over the overlay from a home node to novox —
|
|
* BEFORE any heavy module lands, so the cross-segment-overlay verdict survives whatever the module
|
|
* convergence then does. Phase B converges the full node sets and reports per node.
|
|
*
|
|
* FOUNDATION-ON-NOVOX PORT COLLISIONS (a real consequence of collapsing the anchor onto novox that the
|
|
* separate-anchor beds never hit): the foundation store binds 127.0.0.1:5432 and novox's postgres
|
|
* provider publishes 5432; the foundation broker binds 5671 + 127.0.0.1:5672 and novox's lavinmq
|
|
* provider publishes 5672. The two provider host publishes are REMAPPED off the foundation's ports
|
|
* (REMAP below); consumers reach the providers over the mesh network on the container port, so the
|
|
* host side is free to move. Reported as a topology finding.
|
|
*
|
|
* PERSISTENT RAISE. With MESH_LAB_KEEP set the instance is raised under a fixed id
|
|
* (whole-mesh-full-live) and NOT torn down — it is left standing and browsable. Without it the bed
|
|
* behaves like every other: raise in before(), destroy in after().
|
|
*
|
|
* MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock
|
|
* MESH_LAB_BOOTSTRAP_BINARY=.../mesh-bootstrap MESH_LAB_CATALOG=.../mesh-catalog/modules
|
|
*/
|
|
|
|
import { test, before, after } from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import { existsSync, readFileSync, writeFileSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import { dirname, join, resolve } from "node:path";
|
|
import { loadScenario } from "../../src/declaration/parse.ts";
|
|
import { raise } from "../../src/lifecycle/raise.ts";
|
|
import { destroy, exec, instanceNameOf, push } from "../../src/lifecycle/operate.ts";
|
|
import {
|
|
bootstrapBinaryPath, hostBinaryPath, placeBootstrap, BOOTSTRAP_PATH, HOST_PATH,
|
|
} from "../../src/lifecycle/place.ts";
|
|
import { labIsUsable, destroyAll, foundationBundle, onTheMachine } from "./harness.ts";
|
|
import { referenceFor, type HeldImage } from "../../src/pinning.ts";
|
|
|
|
const capability = await labIsUsable();
|
|
const binary = hostBinaryPath();
|
|
const installer = bootstrapBinaryPath();
|
|
const bundle = process.env["MESH_LAB_BUNDLE"] ?? "";
|
|
const modulesEnv = process.env["MESH_LAB_MODULES"] ?? "";
|
|
// What the installer is told to build. It carries a builder rather than a finished control plane
|
|
// (novox/hq ADR 0073), so genesis is given a repository and a commit — and a commit rather than a
|
|
// branch, because what is cloned is the trust anchor for everything this mesh will ever run.
|
|
const source = process.env["MESH_LAB_SOURCE"] ?? "";
|
|
const sourceRef = process.env["MESH_LAB_SOURCE_REF"] ?? "";
|
|
|
|
const skip = !capability.usable
|
|
? `lab not usable: ${capability.why}`
|
|
: !binary || !existsSync(binary)
|
|
? "MESH_LAB_HOST_BINARY is not set to a built mesh-host"
|
|
: !bundle || !existsSync(bundle)
|
|
? "MESH_LAB_BUNDLE is not set to a foundation bundle (mesh-host examples/)"
|
|
: !installer || !existsSync(installer)
|
|
? "MESH_LAB_BOOTSTRAP_BINARY is not set to a built mesh-bootstrap (mesh-host `make " +
|
|
"bootstrap IMAGE=mesh-builder:development`). The anchor is raised BY the installer now, " +
|
|
"so a run without one would be testing the procedure this bed exists to stop testing"
|
|
: !source
|
|
? "MESH_LAB_SOURCE is not set to the repository the control plane is built from"
|
|
: !sourceRef
|
|
? "MESH_LAB_SOURCE_REF is not set to the commit to build"
|
|
: false;
|
|
|
|
const SCENARIO = "whole-mesh-full";
|
|
/** novox hosts the foundation and the control plane; it is where `mesh` commands run. */
|
|
const CONTROL = "novox";
|
|
/** Every node in the mesh. novox is on hosting; the rest are behind the home gateway. */
|
|
const NODES = ["novox", "ace", "shanks", "g14"];
|
|
/**
|
|
* The machines that JOIN. novox is not one of them, and that is the distinction this bed was
|
|
* restructured to make: novox is brought into existence by the installer, which enrols it as part
|
|
* of genesis. Enrolling it again here would be a second identity the mesh does not know.
|
|
*/
|
|
const HOME_NODES = ["ace", "shanks", "g14"];
|
|
|
|
/** A checkout of the mesh's catalogue, on the anchor, for the installer to read manifests from. */
|
|
const CATALOGUE_ON_MACHINE = "/opt/mesh-catalog";
|
|
|
|
/**
|
|
* The manifests `mesh-bootstrap` reads out of that checkout — and only those.
|
|
*
|
|
* Named here rather than pushing the whole repository because the whole repository is a hundred
|
|
* megabytes of `node_modules` and the installer opens exactly two files: mesh-host's
|
|
* `internal/bootstrap` RegistryModule and BuilderModule. (Not the control plane's: that manifest
|
|
* comes out of the build the installer runs — novox/hq ADR 0069, 04-ISSUES/072.) If it ever opens
|
|
* a third, this list is where the bed finds out, by the installer saying which manifest it could
|
|
* not read.
|
|
*/
|
|
const CATALOGUE_MODULES = ["distribution", "builder"];
|
|
|
|
/**
|
|
* Where this mesh keeps its own images, as the anchor reaches it.
|
|
*
|
|
* **Loopback, and that is a finding rather than a shortcut.** The mesh's registry is plain HTTP on
|
|
* purpose — it is reached over the mesh's own network, which is already the encrypted and
|
|
* authenticated thing — and a container runtime refuses a plain-HTTP registry at any address
|
|
* EXCEPT a loopback one unless it has been told to allow it. So genesis can push to 127.0.0.1:5000
|
|
* with no configuration, and the reference the control-plane module is then pinned to is one only
|
|
* the anchor can pull. On this bed that is enough, because only the anchor runs the control plane.
|
|
* A mesh where a second machine had to pull it would need the runtimes told about the registry
|
|
* first, and nothing in the design says who does that.
|
|
*/
|
|
const MESH_REGISTRY = "127.0.0.1:5000";
|
|
|
|
/**
|
|
* ADR 0066 — the domain each public-facing node composes its routed names under.
|
|
*
|
|
* **The bed had none, so the ADR was untested by construction.** A module now contributes a `label`
|
|
* to `route` and nothing else; the mesh joins it to the node's public domain and the join is the
|
|
* whole feature. On a node with no public domain a labelled contribution composes to nothing — no
|
|
* host, no route — so every routed module on this bed was silently unreachable and the bed still
|
|
* went green. Two nodes face outward here; the workstations do not and get none, which is also part
|
|
* of the design being exercised.
|
|
*
|
|
* `.incus` rather than the real domains: this repository's beds name nothing routable.
|
|
*/
|
|
const PUBLIC_DOMAIN: Record<string, string> = { novox: "novox.incus", ace: "zurag.incus" };
|
|
|
|
/** Keep the instance standing and browsable rather than tearing it down. */
|
|
const KEEP = !!process.env["MESH_LAB_KEEP"];
|
|
const FIXED_ID = process.env["MESH_LAB_INSTANCE_ID"] ?? (KEEP ? "whole-mesh-full-live" : undefined);
|
|
|
|
const catalogDir = process.env["MESH_LAB_CATALOG"]
|
|
?? (modulesEnv ? resolve(dirname(dirname(dirname(modulesEnv))), "mesh-catalog", "modules") : "")
|
|
?? resolve(process.cwd(), "..", "mesh-catalog", "modules");
|
|
|
|
const MEDIA_DIRS = [
|
|
"/services/media/series", "/services/media/anime", "/services/media/movies",
|
|
"/services/media/music", "/services/media/audiobooks", "/services/media/downloads",
|
|
"/services/media/books",
|
|
];
|
|
|
|
type Mod = { name: string; containers: string[]; node?: boolean; runOnce?: string[] };
|
|
|
|
/**
|
|
* The novox set (feat/novox-conversions @ 431310f). The slug fix means only-office/de-spiegel/
|
|
* amqp-email-forwarder now resolve (their minted login was over the 20-char cap before), so they
|
|
* are INCLUDED. CORE gates; the rest are reported gaps (documented in the whole-mesh-novox bed):
|
|
* umami (provisioner url/admin unset), mailu (nox-schema gaps), only-office/de-spiegel (new plain
|
|
* apps, boot secondary), amqp-email-forwarder (hard-coded AMQP vhost authz), firewall/fail2ban
|
|
* (offline lab cannot fetch the package).
|
|
*/
|
|
const NOVOX: Mod[] = [
|
|
{ name: "postgres", containers: ["postgres", "mesh-postgres"] },
|
|
{ name: "redis", containers: ["redis", "mesh-redis"] },
|
|
{ name: "minio", containers: ["minio", "mesh-minio"] },
|
|
{ name: "mongodb", containers: ["mongo", "mesh-mongodb"] },
|
|
{ name: "mssql", containers: ["mssql", "mesh-mssql"] },
|
|
{ name: "lavinmq", containers: ["lavinmq", "mesh-lavinmq"] },
|
|
// ADR 0066: the proxy now REQUIRES an `acme-ca`, so the bed must assign a provider of one or
|
|
// route-proxy is unresolvable and takes every routed module down with it. step-ca is that
|
|
// provider, on the anchor, at mesh scope.
|
|
{ name: "step-ca", containers: ["step-ca"] },
|
|
{ name: "route-proxy", containers: ["route-proxy"] },
|
|
{ name: "keycloak", containers: ["keycloak", "mesh-keycloak"] },
|
|
{ name: "gitea", containers: ["gitea", "mesh-gitea"] },
|
|
{ name: "nextcloud", containers: ["nextcloud", "mesh-nextcloud"] },
|
|
{ name: "umami", containers: ["umami", "mesh-umami"] },
|
|
{ name: "photos", containers: ["photos-server", "photos-admin-client", "photos-client-eef", "photos-client-filip"] },
|
|
{ name: "invoicing", containers: ["invoicing-app", "invoicing-api"] },
|
|
{ name: "novox.be", containers: ["novox-be"] },
|
|
{ name: "only-office", containers: ["office-novox-be"] },
|
|
{ name: "de-spiegel", containers: ["de-spiegel-novox-be"] },
|
|
{ name: "amqp-email-forwarder", containers: ["amqp-email-forwarder"] },
|
|
{ name: "portainer", containers: ["portainer", "mesh-portainer"] },
|
|
{ name: "verdaccio", containers: ["verdaccio", "mesh-verdaccio"] },
|
|
// Already installed, assigned and running — genesis needed it to publish the control plane's
|
|
// image. Left in the plan on purpose: registering the same manifest and assigning it again is
|
|
// what an operator's `module add` + `assign` would do on a mesh that already has it, and a
|
|
// module the installer put there had better survive being asked for a second time. It also keeps
|
|
// mesh-registry in the convergence report, where a reader expects to see it.
|
|
{ name: "registry", containers: ["mesh-registry"] },
|
|
{
|
|
name: "mailu",
|
|
containers: [
|
|
"mailu-resolver", "mailu-redis", "mailu-admin", "mailu-imap", "mailu-smtp",
|
|
"mailu-antispam", "mailu-antivirus", "mailu-webmail", "mailu-webdav", "mailu-fetchmail",
|
|
"mailu-front", "mesh-mailu",
|
|
],
|
|
},
|
|
{ name: "firewall", containers: [], node: true },
|
|
{ name: "fail2ban", containers: [], node: true },
|
|
];
|
|
const CORE_NOVOX = new Set([
|
|
"postgres", "redis", "minio", "mongodb", "mssql", "lavinmq",
|
|
"route-proxy", "keycloak", "gitea", "nextcloud", "invoicing", "photos", "novox.be",
|
|
"portainer", "verdaccio", "registry",
|
|
]);
|
|
const GAPS_NOVOX = new Set([
|
|
"umami", "mailu", "firewall", "fail2ban", "only-office", "de-spiegel", "amqp-email-forwarder",
|
|
// step-ca is reported, not gated: the internal-CA ISSUANCE path is still being fixed in
|
|
// mesh-controller, and this bed is not the place to discover that a fix has not landed yet. What is
|
|
// gated is the half that is decided and cheap — see the ADR 0066 section at the end.
|
|
"step-ca",
|
|
]);
|
|
|
|
/** The ace media/home set. */
|
|
const ACE: Mod[] = [
|
|
{ name: "postgres", containers: ["postgres", "mesh-postgres"] },
|
|
{ name: "redis", containers: ["redis", "mesh-redis"] },
|
|
{ name: "mssql", containers: ["mssql", "mesh-mssql"] },
|
|
{ name: "sonarr", containers: ["sonarr", "mesh-sonarr"] },
|
|
{ name: "radarr", containers: ["radarr", "mesh-radarr"] },
|
|
{ name: "lidarr", containers: ["lidarr", "mesh-lidarr"] },
|
|
{ name: "plex", containers: ["plex", "mesh-plex"] },
|
|
{ name: "bazarr", containers: ["bazarr", "mesh-bazarr"] },
|
|
{ name: "nzbget", containers: ["nzbget", "mesh-nzbget"] },
|
|
{ name: "qbittorrent", containers: ["qbittorrent", "mesh-qbittorrent"] },
|
|
{ name: "jackett", containers: ["jackett", "mesh-jackett"] },
|
|
{ name: "ombi", containers: ["ombi", "mesh-ombi"] },
|
|
{ name: "tautulli", containers: ["tautulli", "mesh-tautulli"] },
|
|
{ name: "bookshelf", containers: ["bookshelf", "mesh-bookshelf"] },
|
|
{ name: "home-assistant", containers: ["home-assistant", "mesh-home-assistant"] },
|
|
{ name: "mosquitto", containers: ["mosquitto", "mesh-mosquitto"], runOnce: ["mosquitto-bootstrap"] },
|
|
{ name: "influxdb", containers: ["influxdb", "mesh-influxdb"] },
|
|
{ name: "grafana", containers: ["grafana", "mesh-grafana"] },
|
|
{ name: "baserow", containers: ["baserow", "mesh-baserow"] },
|
|
{ name: "letta", containers: ["letta", "mesh-letta"] },
|
|
{ name: "nodered", containers: ["nodered", "mesh-nodered"] },
|
|
{ name: "searxng", containers: ["valkey", "searxng", "mesh-searxng"] },
|
|
{ name: "unifi", containers: ["unifi-controller", "mesh-unifi"] },
|
|
{ name: "portainer", containers: ["portainer", "mesh-portainer"] },
|
|
];
|
|
const CORE_ACE = new Set([
|
|
"postgres", "redis", "mssql",
|
|
"sonarr", "radarr", "lidarr", "jackett", "tautulli", "bookshelf",
|
|
"mosquitto", "influxdb", "grafana", "baserow", "nodered", "searxng", "unifi", "portainer",
|
|
]);
|
|
const GAPS_ACE = new Set(["plex", "bazarr", "nzbget", "qbittorrent", "ombi", "home-assistant", "letta"]);
|
|
|
|
/** The two workstations run one light module each, to prove a real module converges and joins the overlay. */
|
|
const LIGHT: Mod[] = [{ name: "portainer", containers: ["portainer", "mesh-portainer"] }];
|
|
const CORE_LIGHT = new Set(["portainer"]);
|
|
const GAPS_LIGHT = new Set<string>();
|
|
|
|
const PLAN: { node: string; mods: Mod[]; core: Set<string>; gaps: Set<string> }[] = [
|
|
{ node: "novox", mods: NOVOX, core: CORE_NOVOX, gaps: GAPS_NOVOX },
|
|
{ node: "ace", mods: ACE, core: CORE_ACE, gaps: GAPS_ACE },
|
|
{ node: "shanks", mods: LIGHT, core: CORE_LIGHT, gaps: GAPS_LIGHT },
|
|
{ node: "g14", mods: LIGHT, core: CORE_LIGHT, gaps: GAPS_LIGHT },
|
|
];
|
|
|
|
/**
|
|
* Host-port remaps (per module; host ports are per-VM so novox's and ace's never clash across nodes).
|
|
* The two FOUNDATION collisions are the new ones: postgres 5432 and lavinmq 5672 are moved off the
|
|
* foundation store/broker's host ports, which only exist on novox because that is where the foundation
|
|
* runs. The rest break the novox web/app host-port collisions (route-proxy fronts 80/443).
|
|
*/
|
|
const REMAP: Record<string, Record<string, string>> = {
|
|
// Moved, NOT bound to loopback. These two carried `127.0.0.1:` and it broke a consumer on a node
|
|
// with no foundation at all: a module is told to reach its provider at `<node>.internal`, that
|
|
// name resolves to the node's overlay address, and a provider listening only on loopback refuses
|
|
// it. letta on ace died of exactly this — "is the server running on that host and accepting
|
|
// TCP/IP connections?" — while postgres sat healthy beside it.
|
|
//
|
|
// The collision needed a different PORT, which is all the other entries here do. The address was
|
|
// never part of it, and adding it made the provider unreachable by the one name the mesh hands
|
|
// its consumers.
|
|
postgres: { "5432": "15432:5432" },
|
|
lavinmq: { "5672": "15673:5672" },
|
|
nextcloud: { "80": "8090:80" },
|
|
umami: { "3000": "3090:3000" },
|
|
invoicing: { "80": "8091:80", "9000": "9091:9000" },
|
|
qbittorrent: { "8080": "8090:8080" },
|
|
searxng: { "8080": "8092:8080" },
|
|
nzbget: { "6789": "6790:6789" },
|
|
};
|
|
|
|
/** Operator-provided app credentials, delivered as fake values through the real `secret accept` path. */
|
|
const CREDENTIALS: { node: string; module: string; name: string; crash: string }[] = [
|
|
{ node: "ace", module: "plex", name: "token", crash: "no Plex token" },
|
|
{ node: "ace", module: "bazarr", name: "api-key", crash: "no Bazarr API key" },
|
|
{ node: "ace", module: "ombi", name: "api-key", crash: "no Ombi API key" },
|
|
{ node: "ace", module: "home-assistant", name: "token", crash: "no Home Assistant token" },
|
|
{ node: "ace", module: "nzbget", name: "password", crash: "NZBGet not configured" },
|
|
{ node: "ace", module: "qbittorrent", name: "password", crash: "qBittorrent not configured" },
|
|
{ node: "novox", module: "umami", name: "admin", crash: "admin password is not set" },
|
|
];
|
|
|
|
/** Operator secrets for the credential modules that own-secret their whole app (mailu, de-spiegel). */
|
|
const OPERATOR_SECRETS: { node: string; module: string; name: string; value: string }[] = [
|
|
{ node: "novox", module: "mailu", name: "secret-key", value: "0123456789abcdef0123456789abcdef" },
|
|
{ node: "novox", module: "mailu", name: "admin", value: "MailuAdminFakePass123" },
|
|
{ node: "novox", module: "mailu", name: "api-token", value: "mailuapitokenfake0123456789abcd" },
|
|
{ node: "novox", module: "de-spiegel", name: "smtp-user", value: "despiegel-smtp-fake" },
|
|
{ node: "novox", module: "de-spiegel", name: "smtp-pass", value: "despiegel-pass-fake" },
|
|
{ node: "novox", module: "amqp-email-forwarder", name: "smtp-user", value: "eef-smtp-fake" },
|
|
{ node: "novox", module: "amqp-email-forwarder", name: "smtp-pass", value: "eef-pass-fake" },
|
|
];
|
|
|
|
let instanceId = "";
|
|
let held: HeldImage[] = [];
|
|
|
|
function quote(s: string): string {
|
|
return `'${s.replaceAll("'", `'\\''`)}'`;
|
|
}
|
|
|
|
async function on(machine: string, command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> {
|
|
const { stdout } = await exec(instanceId, machine, [
|
|
"sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`,
|
|
], timeoutMs);
|
|
const marker = stdout.lastIndexOf("__exit=");
|
|
if (marker < 0) return { out: stdout, ok: false };
|
|
return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" };
|
|
}
|
|
|
|
async function must(machine: string, command: string, timeoutMs?: number): Promise<string> {
|
|
const { out, ok } = await on(machine, command, timeoutMs);
|
|
if (!ok) throw new Error(`${machine}: ${command}\n${out}`);
|
|
return out;
|
|
}
|
|
|
|
/** The control plane, a container on novox (the anchor). */
|
|
async function mesh(command: string, timeoutMs?: number): Promise<string> {
|
|
return must(CONTROL, `docker exec mesh-controller /mesh-controller ${command}`, timeoutMs);
|
|
}
|
|
|
|
/** What a manifest's image reference becomes on the machine — ours by ID, everything else as written. */
|
|
function pinned(reference: string): string {
|
|
return onTheMachine(reference, held);
|
|
}
|
|
|
|
function bundleFor(images: HeldImage[]): string {
|
|
return foundationBundle(bundle, images);
|
|
}
|
|
|
|
function loadManifest(name: string): { manifest: string; broker: boolean } {
|
|
const path = resolve(catalogDir, name, "module.json");
|
|
const m = JSON.parse(readFileSync(path, "utf8")) as {
|
|
resources?: { type: string; image?: string; artifact?: string; ports?: string[] }[];
|
|
};
|
|
const remap = REMAP[name] ?? {};
|
|
for (const r of m.resources ?? []) {
|
|
if (r.type !== "container") continue;
|
|
// **A container naming an artifact is a module the mesh builds, and this bed does not build.**
|
|
// It pre-builds the same images on the workstation and stocks them, which is the lab standing
|
|
// in for the builder — so it does here what the builder does: replace the artifact with the
|
|
// reference the machine actually holds. Without this the unresolved field travels to the
|
|
// machine, whose declaration language has no such field, and the whole declaration is refused.
|
|
//
|
|
// The repository is `mesh-runtime-<module>`, which is not a guess: it is what this repository's
|
|
// own `scripts/build-module-runtime.sh <module>` produces and what the scenarios stock by name.
|
|
if (typeof r.artifact === "string" && typeof r.image !== "string") {
|
|
const reference = referenceFor(held, `mesh-runtime-${name}`);
|
|
assert.ok(reference,
|
|
`${name} declares the "${r.artifact}" artifact and this scenario stocked no ` +
|
|
`mesh-runtime-${name}. The mesh would have to build it, and this bed does not build — ` +
|
|
`add it to the machine's images: in the scenario, or build it with ` +
|
|
`scripts/build-module-runtime.sh ${name}`);
|
|
r.image = reference;
|
|
delete r.artifact;
|
|
}
|
|
if (typeof r.image === "string") r.image = pinned(r.image);
|
|
if (Array.isArray(r.ports)) r.ports = r.ports.map((p) => remap[p] ?? p);
|
|
}
|
|
const manifest = JSON.stringify(m);
|
|
return { manifest, broker: manifest.includes("MESH_BROKER_FILE") };
|
|
}
|
|
|
|
function tokenFrom(said: string): string {
|
|
const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" "));
|
|
assert.ok(found, `no token in:\n${said}`);
|
|
return found;
|
|
}
|
|
|
|
interface NodeState {
|
|
reached: boolean;
|
|
applied: boolean;
|
|
current: boolean;
|
|
waiting: boolean;
|
|
wrong?: { outcome: string; refused?: string | undefined; failed?: { id: string; error: string }[] | undefined } | undefined;
|
|
raw: string;
|
|
}
|
|
|
|
async function nodeState(node: string): Promise<NodeState> {
|
|
const asked = await on(CONTROL, `docker exec mesh-controller /mesh-controller status --json`);
|
|
if (!asked.ok) return { reached: false, applied: false, current: false, waiting: false, raw: asked.out };
|
|
let state: {
|
|
wrong: { node: string; outcome: string; refused?: string; failed?: { id: string; error: string }[] }[];
|
|
waiting: { node: string }[];
|
|
reported: { node: string; outcome: string; current: boolean }[];
|
|
};
|
|
try {
|
|
state = JSON.parse(asked.out);
|
|
} catch {
|
|
return { reached: false, applied: false, current: false, waiting: false, raw: asked.out };
|
|
}
|
|
const word = state.reported.find((r) => r.node === node);
|
|
const bad = state.wrong.find((w) => w.node === node);
|
|
return {
|
|
reached: true,
|
|
applied: word?.outcome === "applied",
|
|
current: !!word?.current,
|
|
waiting: state.waiting.some((w) => w.node === node),
|
|
wrong: bad ? { outcome: bad.outcome, refused: bad.refused, failed: bad.failed } : undefined,
|
|
raw: asked.out,
|
|
};
|
|
}
|
|
|
|
async function psMapOf(node: string): Promise<Map<string, string>> {
|
|
const out = (await on(node, `docker ps -a --format '{{.Names}}\t{{.Status}}'`)).out;
|
|
const map = new Map<string, string>();
|
|
for (const line of out.split("\n")) {
|
|
const [n, ...rest] = line.split("\t");
|
|
if (n) map.set(n.trim(), rest.join("\t").trim());
|
|
}
|
|
return map;
|
|
}
|
|
|
|
/**
|
|
* ADR 0066: the internal CA is initialised FROM AN OPERATOR'S ROOT — it does not mint its own.
|
|
*
|
|
* So the bed has to be an operator. The material is made on the anchor with openssl and handed to
|
|
* the mesh through the real `secret accept` path, exactly as a person would: the mesh cannot invent
|
|
* a PEM, and the random 32 bytes it makes for an own-secret nobody supplied would leave step-ca
|
|
* crash-looping on a root key that is not a key.
|
|
*/
|
|
async function deliverCaRoot(): Promise<boolean> {
|
|
const made = await on(CONTROL, [
|
|
"set -e",
|
|
"mkdir -p /tmp/ca && cd /tmp/ca",
|
|
// No trailing newline on a password file: step-ca reads the file as the password itself.
|
|
"openssl rand -hex 16 | tr -d '\\n' > key-password",
|
|
"openssl ecparam -genkey -name prime256v1 -out root.unenc",
|
|
"openssl ec -in root.unenc -aes256 -passout file:key-password -out root.key",
|
|
"rm -f root.unenc",
|
|
"openssl req -x509 -new -key root.key -passin file:key-password -sha256 -days 3650" +
|
|
` -out root.crt -subj "/CN=Mesh Internal CA/O=Novox Mesh Lab"`,
|
|
// Readable by the control plane, which is not root. Its image is FROM scratch and runs as
|
|
// 65534, and `docker cp` keeps the ownership and mode a file had outside — openssl writes a
|
|
// private key 0600 root-owned, so the copy landed unreadable and `secret accept` failed with
|
|
// `open /ca-root-key: permission denied`. The CA then crash-looped on a root it never got.
|
|
// Chowning it inside the container is not available: there is no shell in there to do it with.
|
|
//
|
|
// Safe here and nowhere else: these three exist for the seconds between being written and
|
|
// being sealed to the machine, on a lab node, for a CA thrown away with the scenario.
|
|
"chmod 0644 /tmp/ca/root.crt /tmp/ca/root.key /tmp/ca/key-password",
|
|
"docker cp /tmp/ca/root.crt mesh-controller:/ca-root-cert",
|
|
"docker cp /tmp/ca/root.key mesh-controller:/ca-root-key",
|
|
"docker cp /tmp/ca/key-password mesh-controller:/ca-root-key-password",
|
|
].join("\n"), 180_000);
|
|
if (!made.ok) {
|
|
console.log(`CA ROOT NOT MADE on ${CONTROL}:\n${made.out.split("\n").slice(-8).join("\n")}`);
|
|
return false;
|
|
}
|
|
for (const [name, file] of [
|
|
["root-cert", "/ca-root-cert"],
|
|
["root-key", "/ca-root-key"],
|
|
["root-key-password", "/ca-root-key-password"],
|
|
] as const) {
|
|
try {
|
|
await mesh(`secret accept ${CONTROL} step-ca ${name} --from ${file}`);
|
|
} catch (err) {
|
|
console.log(`CA ROOT ACCEPT FAILED (${name}): ${(err as Error).message.split("\n").slice(0, 3).join(" | ")}`);
|
|
return false;
|
|
}
|
|
}
|
|
return true;
|
|
}
|
|
|
|
/** What a module's manifest says its route label is, or "" if it contributes no route. */
|
|
function routeLabelOf(name: string): string {
|
|
const path = resolve(catalogDir, name, "module.json");
|
|
const m = JSON.parse(readFileSync(path, "utf8")) as {
|
|
contributes?: { route?: { label?: string } };
|
|
};
|
|
return m.contributes?.route?.label ?? "";
|
|
}
|
|
|
|
/** A node's overlay (mesh0) address, or "" if it has none yet. */
|
|
async function overlayAddr(node: string): Promise<string> {
|
|
const out = (await on(node, `ip -4 -o addr show mesh0 2>/dev/null | awk '{print $4}' | cut -d/ -f1`)).out;
|
|
return out.split("\n").map((l) => l.trim()).find(Boolean) ?? "";
|
|
}
|
|
|
|
// ==================================================================================================
|
|
// PHASE 1 — GENESIS. novox is brought into existence by the installer.
|
|
// ==================================================================================================
|
|
|
|
/** What genesis did, or where it stopped. */
|
|
interface GenesisResult {
|
|
ok: boolean;
|
|
/** `step 7 of 10, registry` — the installer's own words, so the bed reports the cause. */
|
|
step: string;
|
|
why: string;
|
|
report: string[];
|
|
}
|
|
|
|
/** The step a failed `mesh-bootstrap` names, as it prints it, or "" if it named none. */
|
|
function stepIn(said: string): string {
|
|
return said.match(/^mesh-bootstrap: (step \d+ of \d+, [a-z-]+):/m)?.[1] ?? "";
|
|
}
|
|
|
|
/**
|
|
* Put on the anchor what the installer needs to read, and run it.
|
|
*
|
|
* **This is the whole of what changed, and it is not a refactor.** The bed used to apply the
|
|
* foundation bundle itself, by hand, and then enrol four machines in one loop. It got the order
|
|
* right by accident and it modelled the wrong shape: an install procedure that exists only as a
|
|
* test fixture is exercised by whoever writes tests and never by whoever installs, which is why
|
|
* every bootstrap fault this year was found late (novox/hq ADR 0067). The anchor is now raised by
|
|
* running the same program a bare machine runs, and the bed only reads the result.
|
|
*
|
|
* It is run up to three times. Not to paper over a failure — every attempt's failing step is
|
|
* printed — but because `mesh-bootstrap` is idempotent by design and says so, and because the one
|
|
* thing here that fails for a reason which goes away by itself is a pull: the store, the broker and
|
|
* the registry come from the internet, through a household gateway's masquerade, and Docker Hub
|
|
* rate-limiting an anonymous pull is not this mesh's fault. A re-run is the retry, and it is the
|
|
* retry the installer's own documentation names.
|
|
*/
|
|
async function genesis(images: HeldImage[]): Promise<GenesisResult> {
|
|
const report: string[] = ["================ GENESIS: novox becomes a mesh of one ================"];
|
|
const stop = (step: string, why: string): GenesisResult => {
|
|
report.push(`\nSTOPPED at ${step || "(no step named)"}: ${why}`);
|
|
return { ok: false, step, why, report };
|
|
};
|
|
|
|
// The installer, beside the host binary. Everything else on this machine was placed by `raise`;
|
|
// this one is placed here because only the anchor is bootstrapped.
|
|
const name = await instanceNameOf(instanceId, CONTROL);
|
|
const version = await placeBootstrap(name, CONTROL, installer as string,
|
|
(m) => console.log(`genesis:${m}`));
|
|
report.push(` installer ${version} at ${BOOTSTRAP_PATH}`);
|
|
report.push(` builds from ${source} at ${sourceRef.slice(0, 8)}`);
|
|
|
|
// The catalogue. `mesh-bootstrap --catalog` reads manifests from a CHECKOUT on the machine,
|
|
// because at this moment the mesh has no forge, no build machine and — until step 7 finishes —
|
|
// no registry. A manifest is a file, and somebody has to have put it there.
|
|
await must(CONTROL, `mkdir -p ${CATALOGUE_MODULES.map((m) => `${CATALOGUE_ON_MACHINE}/modules/${m}`).join(" ")}`);
|
|
for (const module of CATALOGUE_MODULES) {
|
|
const from = resolve(catalogDir, module, "module.json");
|
|
assert.ok(existsSync(from), `the catalogue has no ${module}/module.json at ${from}`);
|
|
await push(instanceId, CONTROL, from, `${CATALOGUE_ON_MACHINE}/modules/${module}/module.json`);
|
|
}
|
|
report.push(` catalogue ${CATALOGUE_MODULES.join(", ")} at ${CATALOGUE_ON_MACHINE}`);
|
|
|
|
// The foundation TEMPLATE — not the bundle. The installer produces the bundle from it: it replaces
|
|
// the control plane's image with the id of the image it carries, renames that container
|
|
// `temp-mesh-controller`, and writes the result where a person can read it.
|
|
//
|
|
// Two substitutions still happen here, and both belong to the bed rather than to the installer.
|
|
// The example names three images at a registry the lab no longer raises: the store and the broker
|
|
// become the upstream references mesh-catalog pins (harness), and the machine pulls them over its
|
|
// uplink like any first node. The third, mesh-controller, is deliberately LEFT naming that dead
|
|
// registry — the installer overwrites it, and leaving it proves that it does.
|
|
//
|
|
// And the broker's advertised address. The template hardcodes 192.0.2.10:5671, the old
|
|
// separate-anchor address; a token carries MESH_BROKER_ADDRESS verbatim as the endpoint an
|
|
// enrolling node dials, so with the foundation on novox it must be novox's own public address or
|
|
// every node would enrol against a dead one. The installer refuses to guess this and says so
|
|
// loudly, which is right — it does not know what this machine is called from outside.
|
|
const template = bundleFor(images).replaceAll("192.0.2.10:5671", "192.0.2.20:5671");
|
|
const local = join(tmpdir(), `mesh-lab-foundation-${process.pid}.lock`);
|
|
writeFileSync(local, template);
|
|
await push(instanceId, CONTROL, local, "/tmp/foundation-template.lock");
|
|
|
|
const command = [
|
|
BOOTSTRAP_PATH,
|
|
`--source ${source}`,
|
|
`--source-ref ${sourceRef}`,
|
|
`--bundle /tmp/foundation-template.lock`,
|
|
`--catalog ${CATALOGUE_ON_MACHINE}`,
|
|
`--node ${CONTROL}`,
|
|
`--registry ${MESH_REGISTRY}`,
|
|
`--host ${HOST_PATH}`,
|
|
// The lab has no unit to supervise the host with, and the installer refuses to invent one — a
|
|
// unit file is a packaging decision. This is the arrangement it offers instead, and it is loud
|
|
// about what it is: a host started this way does not survive a reboot.
|
|
`--host-in-background`,
|
|
].join(" ");
|
|
|
|
let said = "";
|
|
let step = "";
|
|
for (let attempt = 1; attempt <= 3; attempt++) {
|
|
const ran = await on(CONTROL, command, 2_400_000);
|
|
said = ran.out;
|
|
console.log(`\n---- mesh-bootstrap on ${CONTROL} (attempt ${attempt}) ----\n${said}`);
|
|
if (ran.ok) {
|
|
step = "";
|
|
break;
|
|
}
|
|
step = stepIn(said);
|
|
if (attempt < 3) {
|
|
console.log(`genesis attempt ${attempt} stopped at ${step || "an unnamed step"}; ` +
|
|
`re-running in 30s — every step it already did will say so`);
|
|
await new Promise((r) => setTimeout(r, 30_000));
|
|
}
|
|
}
|
|
if (step) {
|
|
return stop(step, said.split("\n").filter(Boolean).slice(-6).join("\n"));
|
|
}
|
|
|
|
// ------------------------------------------------------------------------------------------
|
|
// And now the only thing that matters: is novox a WORKING MESH OF ONE? Asked of the machine,
|
|
// never inferred from the installer exiting zero (novox/hq ADR 0018).
|
|
// ------------------------------------------------------------------------------------------
|
|
|
|
// 1. The control plane answers. Asked of the PERMANENT container by name — `status` opens all
|
|
// three stores, so a reply proves the connections it was given are the foundation's own.
|
|
const answered = await on(CONTROL, `docker exec mesh-controller /mesh-controller status`, 60_000);
|
|
report.push(` control plane ${answered.ok ? answered.out.split("\n")[0] : "NO ANSWER"}`);
|
|
if (!answered.ok) return stop("after step 10", `mesh-controller does not answer:\n${answered.out}`);
|
|
|
|
// 2. The registry replies on /v2/ — the registry API's own "yes, I am one and I am ready". A
|
|
// container that is up is not a registry that serves.
|
|
const v2 = await on(CONTROL,
|
|
`curl -s -o /dev/null -w '%{http_code}' --max-time 10 http://${MESH_REGISTRY}/v2/`);
|
|
const v2Code = v2.out.trim();
|
|
report.push(` registry /v2/ ${v2Code || "no answer"}`);
|
|
if (v2Code !== "200") return stop("after step 10", `the mesh's own registry answered ${v2Code || "nothing"}`);
|
|
|
|
// 3. THE PIVOT COMPLETED. ADR 0067 states this check in as many words: after installing, the
|
|
// running control plane's image is pinned by a digest THE MESH'S OWN REGISTRY ASSIGNED — not
|
|
// by an image id. If it is still an image id, the foundation's container is what is running,
|
|
// nothing was published, and this mesh can never roll out its own upgrades.
|
|
const pinnedTo = (await on(CONTROL, `docker inspect --format '{{.Config.Image}}' mesh-controller`))
|
|
.out.trim();
|
|
report.push(` pinned to ${pinnedTo || "(nothing)"}`);
|
|
if (/^sha256:[0-9a-f]{64}$/.test(pinnedTo)) {
|
|
return stop("after step 10",
|
|
`mesh-controller is running from ${pinnedTo}, which is an IMAGE ID — the digest of the image's ` +
|
|
`own configuration, which no registry ever served. The pivot did not happen: what is ` +
|
|
`running is the image the installer carried, not one this mesh published, so this mesh ` +
|
|
`cannot upgrade itself (novox/hq ADR 0067, "the pivot completed").`);
|
|
}
|
|
if (!new RegExp(`^${MESH_REGISTRY.replaceAll(".", "\\.")}/mesh-controller@sha256:[0-9a-f]{64}$`)
|
|
.test(pinnedTo)) {
|
|
return stop("after step 10",
|
|
`mesh-controller is running from ${pinnedTo || "nothing this bed could read"}, which is not a ` +
|
|
`digest assigned by ${MESH_REGISTRY}.`);
|
|
}
|
|
|
|
// 3a. THE CONTROL PLANE WAS BUILT, not carried.
|
|
//
|
|
// **The distinction the installer now exists to make** (novox/hq ADR 0073). A mesh running an
|
|
// image it was handed cannot rebuild the thing that runs it, and looks identical from the
|
|
// outside to one that can — same container, same digest, same registry. The difference is
|
|
// whether a build happened, and the only place that is visible is the installer saying so.
|
|
const wanted = sourceRef.slice(0, 8);
|
|
if (!new RegExp(`built mesh-controller from ${wanted}`).test(said)) {
|
|
return stop("after the last step",
|
|
`the installer never said it built mesh-controller from ${wanted}. What runs may have been ` +
|
|
`carried rather than made here, which is a mesh that cannot rebuild its own control plane.`);
|
|
}
|
|
report.push(` built here mesh-controller from ${wanted}, by the carried builder`);
|
|
|
|
// 3b. And the registry really serves it, asked of the registry rather than of the container. A
|
|
// reference is a claim; a tag list is the registry agreeing.
|
|
const tags = await on(CONTROL,
|
|
`curl -s --max-time 10 http://${MESH_REGISTRY}/v2/mesh-controller/tags/list`);
|
|
report.push(` registry holds ${tags.out.trim() || "nothing"}`);
|
|
if (!tags.out.includes("genesis")) {
|
|
return stop("after step 10",
|
|
`${MESH_REGISTRY} does not serve mesh-controller, so the digest the container is pinned to ` +
|
|
`names an image nothing can pull: ${tags.out.trim()}`);
|
|
}
|
|
|
|
// 4. The temporary control plane is GONE. Two control planes is the half-finished state, and the
|
|
// name is the audit: a machine running mesh-controller and not temp-mesh-controller has pivoted.
|
|
const temp = await on(CONTROL, `docker inspect --format '{{.State.Status}}' temp-mesh-controller`);
|
|
report.push(` temp-mesh-controller ${temp.ok ? `STILL HERE (${temp.out.trim()})` : "gone"}`);
|
|
if (temp.ok) {
|
|
return stop("after step 10",
|
|
`temp-mesh-controller is still ${temp.out.trim()}. Two control planes are consuming this ` +
|
|
`mesh's broker queues; neither is wrong and the pivot is not finished.`);
|
|
}
|
|
|
|
// 5. And the mesh has heard from its one node. Everything the join phase does next depends on it.
|
|
const nodes = await on(CONTROL, `docker exec mesh-controller /mesh-controller node list`);
|
|
report.push(` node list ${nodes.out.trim().split("\n").join(" | ")}`);
|
|
const line = nodes.out.split("\n").map((l) => l.trim()).find((l) => l.startsWith(`${CONTROL} `));
|
|
if (!line || !/^\S+\s+here\b/.test(line)) {
|
|
return stop("after step 10",
|
|
`the mesh has not heard from ${CONTROL}: ${line ?? "it has no record of it at all"}`);
|
|
}
|
|
|
|
report.push(`\nVERDICT: ${CONTROL} is a working mesh of one, bootstrapped through the installer.`);
|
|
return { ok: true, step: "", why: "", report };
|
|
}
|
|
|
|
// ==================================================================================================
|
|
// PHASE 2 — JOINING. Everything else is a machine joining a mesh that already exists.
|
|
// ==================================================================================================
|
|
|
|
/**
|
|
* ace, shanks and g14 join. Host binary plus a token — no bootstrap, no foundation, no registry.
|
|
*
|
|
* **novox is not in this loop.** It was enrolled by the installer, as part of becoming a mesh, and
|
|
* enrolling it again would present the mesh with a second identity for a node it already knows —
|
|
* which `mesh-host enrol` refuses, and rightly.
|
|
*
|
|
* The home nodes reach novox's public 192.0.2.20:5671 by dialling OUT through the household
|
|
* gateway, so the enrol itself is the first proof that outbound home→public works.
|
|
*/
|
|
async function joinTheMesh(): Promise<void> {
|
|
// ADR 0066: said as soon as the record exists, because everything routed is composed from it. A
|
|
// node that faces the outside has one; the workstations do not, and are given none. The anchor's
|
|
// is set here rather than in genesis because it is a fact about the mesh, not part of raising
|
|
// one — and the installer has an opinion about neither.
|
|
const anchorDomain = PUBLIC_DOMAIN[CONTROL];
|
|
if (anchorDomain) await mesh(`node public-domain ${CONTROL} ${anchorDomain}`);
|
|
|
|
for (const machine of HOME_NODES) {
|
|
await mesh(`node add ${machine}`);
|
|
const domain = PUBLIC_DOMAIN[machine];
|
|
if (domain) await mesh(`node public-domain ${machine} ${domain}`);
|
|
const token = tokenFrom(await mesh(`token issue --node ${machine}`));
|
|
const said = await must(machine, `${HOST_PATH} enrol --token ${quote(token)}`, 180_000);
|
|
assert.match(said, new RegExp(`enrolled as ${machine}`), said);
|
|
await must(machine, `nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`);
|
|
}
|
|
}
|
|
|
|
before(async () => {
|
|
if (skip) return;
|
|
assert.ok(existsSync(catalogDir), `mesh-catalog modules not found at ${catalogDir}`);
|
|
|
|
const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), {
|
|
onProgress: (m) => console.log(`raise: ${m}`),
|
|
...(FIXED_ID ? { instanceId: FIXED_ID } : {}),
|
|
});
|
|
instanceId = raised.instanceId;
|
|
held = raised.images;
|
|
console.log(`INSTANCE ${instanceId}${KEEP ? " (KEEP — will be left standing)" : ""}`);
|
|
|
|
// ---- PHASE 1, and it GATES phase 2 ---------------------------------------------------------
|
|
//
|
|
// Caught rather than allowed to propagate, so that a failure BEFORE the installer ran — a
|
|
// manifest that is not where the bed thought, a binary that would not copy — is reported in the
|
|
// same shape as one the installer itself named, instead of as a bare stack trace from a helper.
|
|
let genesisResult: GenesisResult;
|
|
try {
|
|
genesisResult = await genesis(raised.images);
|
|
} catch (err) {
|
|
genesisResult = {
|
|
ok: false,
|
|
step: "getting the anchor ready to be bootstrapped — the installer never ran",
|
|
why: (err as Error).message,
|
|
report: ["================ GENESIS: novox becomes a mesh of one ================"],
|
|
};
|
|
}
|
|
console.log(genesisResult.report.join("\n"));
|
|
assert.ok(genesisResult.ok,
|
|
`GENESIS FAILED — ${genesisResult.step || "no step named"}.\n\n${genesisResult.why}\n\n` +
|
|
`No other machine was asked to join. A second machine joining a mesh that is not ready is a ` +
|
|
`different failure with a different cause, and running it now would bury this one under it.\n\n` +
|
|
genesisResult.report.join("\n"));
|
|
|
|
// ---- PHASE 2 --------------------------------------------------------------------------------
|
|
await joinTheMesh();
|
|
|
|
// The operator provides ace's media library (ADR 0051 accesses confirm the paths, create nothing).
|
|
await must("ace", `mkdir -p ${MEDIA_DIRS.join(" ")}`);
|
|
}, { timeout: 5_400_000 });
|
|
|
|
after(async () => {
|
|
if (KEEP) {
|
|
console.log(`\nLEFT STANDING: ${instanceId} — not destroyed (MESH_LAB_KEEP).`);
|
|
return;
|
|
}
|
|
if (instanceId) await destroy(instanceId);
|
|
await destroyAll(`${SCENARIO}-`);
|
|
}, { timeout: 900_000 });
|
|
|
|
test("the full mesh forms across the access point and both server sets converge", {
|
|
skip, timeout: 5_400_000,
|
|
}, async () => {
|
|
// ================================================================================================
|
|
// PHASE A — THE HEADLINE. Place the overlay (hub on novox at its public endpoint; the home nodes
|
|
// dial out, no endpoint of their own), assign networking to every node, push, and VERIFY the tunnel
|
|
// forms ACROSS the gateway. This runs BEFORE any heavy module, so the cross-segment-overlay verdict
|
|
// is captured whatever the module convergence then does.
|
|
// ================================================================================================
|
|
await mesh("overlay place novox --hub --endpoint 192.0.2.20:51820 --site hosting");
|
|
for (const node of HOME_NODES) await mesh(`overlay place ${node} --site home`);
|
|
for (const node of NODES) await mesh(`assign ${node} networking`);
|
|
for (const node of NODES) {
|
|
try {
|
|
await mesh(`push ${node}`, 180_000);
|
|
} catch (err) {
|
|
console.log(`networking push rejected (${node}): ${(err as Error).message.split("\n").slice(0, 4).join(" | ")}`);
|
|
}
|
|
}
|
|
|
|
// Give the home nodes time to dial the hub and complete a handshake through the NAT.
|
|
const overlay: Record<string, string> = {};
|
|
const deadline = Date.now() + 300_000;
|
|
while (Date.now() < deadline) {
|
|
for (const node of NODES) if (!overlay[node]) overlay[node] = await overlayAddr(node);
|
|
if (NODES.every((n) => overlay[n])) break;
|
|
await new Promise((r) => setTimeout(r, 8000));
|
|
}
|
|
// A little longer for handshakes to settle (keepalive interval).
|
|
await new Promise((r) => setTimeout(r, 30000));
|
|
|
|
const overlayReport: string[] = ["================ CROSS-SEGMENT OVERLAY (the headline) ================"];
|
|
for (const node of NODES) overlayReport.push(` ${node.padEnd(8)} mesh0 = ${overlay[node] || "NONE"}`);
|
|
|
|
// The hub's WireGuard peers and their handshakes, from novox.
|
|
const hubWg = (await on("novox", `wg show 2>&1 || echo 'wg tool absent'`)).out;
|
|
overlayReport.push(`\n---- novox (hub) wg show ----\n${hubWg}`);
|
|
|
|
// From each home node: its wg peer state (endpoint should be 192.0.2.20:51820, with a recent
|
|
// handshake) AND a ping to novox's overlay address — the functional proof the tunnel carries
|
|
// traffic across the gateway.
|
|
const overlayFormed: Record<string, boolean> = {};
|
|
const novoxOverlay = overlay["novox"] ?? "";
|
|
for (const node of HOME_NODES) {
|
|
const wg = (await on(node, `wg show 2>&1 || echo 'wg tool absent'`)).out;
|
|
const handshake = (await on(node, `wg show all latest-handshakes 2>/dev/null | awk '{print $2}' | sort -rn | head -1`)).out.trim();
|
|
const ping = novoxOverlay
|
|
? await on(node, `ping -c 3 -W 2 ${novoxOverlay} 2>&1 | tail -3`)
|
|
: { out: "novox has no overlay address to ping", ok: false };
|
|
const handshakeSecs = Number(handshake) || 0;
|
|
// Formed = we can reach novox over the overlay from this home node (traffic across the NAT).
|
|
overlayFormed[node] = ping.ok;
|
|
overlayReport.push(`\n---- ${node} (home) ----`);
|
|
overlayReport.push(wg.split("\n").map((l) => ` ${l}`).join("\n"));
|
|
overlayReport.push(` latest-handshake epoch: ${handshake || "none"}${handshakeSecs ? "" : " (no handshake recorded)"}`);
|
|
overlayReport.push(` ping novox(${novoxOverlay}) over overlay: ${ping.ok ? "REPLIES" : "NO REPLY"}`);
|
|
overlayReport.push(ping.out.split("\n").map((l) => ` ${l}`).join("\n"));
|
|
}
|
|
const anyHomeFormed = HOME_NODES.some((n) => overlayFormed[n]);
|
|
const allHomeFormed = HOME_NODES.every((n) => overlayFormed[n]);
|
|
overlayReport.push(`\nVERDICT: overlay across the access point ${allHomeFormed ? "FORMED for all home nodes" : anyHomeFormed ? "FORMED for some home nodes" : "DID NOT FORM"}.`);
|
|
const overlaySummary = overlayReport.join("\n");
|
|
console.log(overlaySummary);
|
|
|
|
// ================================================================================================
|
|
// PHASE B — converge the full node sets on top of the overlay.
|
|
// ================================================================================================
|
|
const added = new Map<string, boolean>();
|
|
async function ensureAdded(name: string): Promise<boolean> {
|
|
const known = added.get(name);
|
|
if (known !== undefined) return known;
|
|
const { manifest, broker } = loadManifest(name);
|
|
await must(CONTROL, `printf %s ${quote(manifest)} > /tmp/${name}.json && docker cp /tmp/${name}.json mesh-controller:/${name}.json`);
|
|
await mesh(`module add /${name}.json`);
|
|
added.set(name, broker);
|
|
return broker;
|
|
}
|
|
|
|
const assigned: Record<string, Set<string>> = { novox: new Set(), ace: new Set(), shanks: new Set(), g14: new Set() };
|
|
const refused: Record<string, { name: string; why: string }[]> = { novox: [], ace: [], shanks: [], g14: [] };
|
|
for (const { node, mods } of PLAN) {
|
|
for (const { name } of mods) {
|
|
try {
|
|
const broker = await ensureAdded(name);
|
|
if (broker) await mesh(`module issue ${name} --node ${node}`);
|
|
await mesh(`assign ${node} ${name}`);
|
|
assigned[node]!.add(name);
|
|
} catch (err) {
|
|
const why = (err as Error).message.split("\n").map((l) => l.trim()).filter(Boolean).slice(1, 5).join(" | ");
|
|
refused[node]!.push({ name, why });
|
|
console.log(`NOT ASSIGNED ${node}/${name}: ${why}`);
|
|
}
|
|
}
|
|
}
|
|
|
|
// Operator-provided app credentials (own-secrets), delivered as fake values through `secret accept`.
|
|
const credentialDelivered = new Map<string, boolean>();
|
|
for (const name of new Set(CREDENTIALS.map((c) => c.name))) {
|
|
await must(CONTROL, `printf %s ${quote(`fake-${name}-value`)} > /tmp/fake-${name} && docker cp /tmp/fake-${name} mesh-controller:/fake-${name}`);
|
|
}
|
|
for (const c of CREDENTIALS) {
|
|
if (!assigned[c.node]!.has(c.module)) {
|
|
credentialDelivered.set(`${c.node}/${c.module}`, false);
|
|
continue;
|
|
}
|
|
try {
|
|
await mesh(`secret accept ${c.node} ${c.module} ${c.name} --from /fake-${c.name}`);
|
|
credentialDelivered.set(`${c.node}/${c.module}`, true);
|
|
} catch (err) {
|
|
credentialDelivered.set(`${c.node}/${c.module}`, false);
|
|
console.log(`CREDENTIAL ACCEPT FAILED ${c.node}/${c.module}: ${(err as Error).message.split("\n").slice(0, 3).join(" | ")}`);
|
|
}
|
|
}
|
|
// Whole-app own-secrets (mailu/de-spiegel/amqp-email-forwarder).
|
|
for (const s of OPERATOR_SECRETS) {
|
|
if (!assigned[s.node]!.has(s.module)) continue;
|
|
try {
|
|
const inControl = `/secret-${s.module}-${s.name}`;
|
|
await must(CONTROL, `printf %s ${quote(s.value)} > /tmp${inControl} && docker cp /tmp${inControl} mesh-controller:${inControl}`);
|
|
await mesh(`secret accept ${s.node} ${s.module} ${s.name} --from ${inControl}`);
|
|
} catch (err) {
|
|
console.log(`OPERATOR SECRET FAILED ${s.node}/${s.module}/${s.name}: ${(err as Error).message.split("\n").slice(0, 2).join(" | ")}`);
|
|
}
|
|
}
|
|
|
|
// ADR 0066: the CA's root, before the push that would otherwise deliver a random 32 bytes for it.
|
|
const caRootDelivered = assigned["novox"]!.has("step-ca") ? await deliverCaRoot() : false;
|
|
if (!caRootDelivered) console.log("ADR 0066: no operator root delivered; step-ca cannot initialise.");
|
|
|
|
// ONE push per node (workstations first — cheap — then the heavy service nodes).
|
|
const pushError: Record<string, string> = {};
|
|
for (const node of ["shanks", "g14", "novox", "ace"]) {
|
|
try {
|
|
await mesh(`push ${node}`, 300_000);
|
|
} catch (err) {
|
|
pushError[node] = (err as Error).message;
|
|
console.log(`PUSH REJECTED (${node}):\n${pushError[node]!.split("\n").slice(0, 6).join("\n")}`);
|
|
}
|
|
}
|
|
|
|
// Wait for each node's CORE containers to come up (all nodes pull concurrently from the one registry).
|
|
const psMaps: Record<string, Map<string, string>> = { novox: new Map(), ace: new Map(), shanks: new Map(), g14: new Map() };
|
|
for (const { node, mods, core } of PLAN) {
|
|
if (pushError[node]) continue;
|
|
const coreContainers = mods.filter((m) => core.has(m.name) && assigned[node]!.has(m.name)).flatMap((m) => m.containers);
|
|
const until = Date.now() + 3_000_000;
|
|
while (Date.now() < until) {
|
|
psMaps[node] = await psMapOf(node);
|
|
if (coreContainers.every((c) => (psMaps[node]!.get(c) ?? "").startsWith("Up"))) break;
|
|
await new Promise((r) => setTimeout(r, 10000));
|
|
}
|
|
}
|
|
await new Promise((r) => setTimeout(r, 20000));
|
|
|
|
// ================================================================================================
|
|
// Per-node convergence report.
|
|
// ================================================================================================
|
|
const users = (await on("novox", `docker exec mesh-broker lavinmqctl list_users 2>&1`)).out;
|
|
const allProblems: string[] = [];
|
|
const report: string[] = ["================ FULL MESH CONVERGENCE ================"];
|
|
|
|
for (const { node, mods, core, gaps } of PLAN) {
|
|
const psMap = psMaps[node] = await psMapOf(node);
|
|
const st = await nodeState(node);
|
|
const running = (name: string): boolean => (psMap.get(name) ?? "").startsWith("Up");
|
|
const ranOnce = (name: string): boolean => !psMap.has(name) || /^(Up|Exited \(0\))/.test(psMap.get(name) ?? "");
|
|
const failedResources = st.wrong?.failed ?? [];
|
|
|
|
report.push(`\n---- node ${node}: reached=${st.reached} applied=${st.applied} current=${st.current} waiting=${st.waiting} ----`);
|
|
if (pushError[node]) report.push(` PUSH REJECTED: ${pushError[node]!.split("\n").slice(0, 6).join("\n ")}`);
|
|
if (st.wrong) {
|
|
report.push(` NODE WRONG: outcome=${st.wrong.outcome}`);
|
|
for (const f of failedResources) report.push(` failed ${f.id}: ${f.error}`);
|
|
}
|
|
for (const r of refused[node]!) report.push(` REFUSED ${r.name}: ${r.why}`);
|
|
|
|
const coreFailures: string[] = [];
|
|
for (const mod of mods) {
|
|
if (!assigned[node]!.has(mod.name)) continue;
|
|
if (mod.node) {
|
|
report.push(` ${core.has(mod.name) ? "*" : " "} ${mod.name.padEnd(20)} ${core.has(mod.name) ? "CORE" : "gap "} node-service`);
|
|
continue;
|
|
}
|
|
const states = mod.containers.map((c) => `${c}:${running(c) ? "UP" : (psMap.get(c) ?? "MISSING")}`);
|
|
const ok = mod.containers.every(running) && (mod.runOnce ?? []).every(ranOnce);
|
|
const tag = core.has(mod.name) ? (ok ? "OK " : "FAIL") : (ok ? "ok " : "GAP ");
|
|
report.push(` ${core.has(mod.name) ? "*" : " "} ${mod.name.padEnd(20)} ${tag} ${states.join(" ")}`);
|
|
if (core.has(mod.name) && !ok) coreFailures.push(mod.name);
|
|
}
|
|
const issuedHere = mods.filter((m) => new RegExp(`${node}-${m.name}\\b`).test(users)).length;
|
|
report.push(` broker accounts: ${issuedHere} present for ${node}`);
|
|
|
|
const gapOwnerOf = (f: { id: string; error: string }): string => {
|
|
const m = f.error.match(/applying "([^".]+)\./);
|
|
return m?.[1] ?? (f.id.split(".")[0] ?? "");
|
|
};
|
|
if (pushError[node]) allProblems.push(`${node}: push rejected`);
|
|
if (coreFailures.length) allProblems.push(`${node}: CORE not converged: ${coreFailures.join(", ")}`);
|
|
const nonGapFailed = failedResources.filter((f) => !gaps.has(gapOwnerOf(f)));
|
|
if (nonGapFailed.length) allProblems.push(`${node}: non-gap resource failed: ${nonGapFailed.map((f) => `${f.id} (${f.error.slice(0, 60)})`).join(", ")}`);
|
|
}
|
|
|
|
const summary = report.join("\n");
|
|
console.log(summary);
|
|
|
|
// Diagnostics for any CORE container that did not come up.
|
|
for (const { node, mods, core } of PLAN) {
|
|
const psMap = psMaps[node]!;
|
|
for (const mod of mods) {
|
|
if (!core.has(mod.name) || !assigned[node]!.has(mod.name)) continue;
|
|
for (const c of mod.containers) {
|
|
if (psMap.has(c) && !(psMap.get(c) ?? "").startsWith("Up")) {
|
|
console.log(`\n---- ${node} logs: ${c} (${psMap.get(c)}) ----\n${(await on(node, `docker logs ${c} 2>&1 | tail -25`)).out}`);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// ================================================================================================
|
|
// ADR 0066 — ROUTE NAMES AND THE INTERNAL CA. Additive, and deliberately only the cheap half.
|
|
//
|
|
// What is checked here is the part that is DECIDED and costs one file read: a module contributes a
|
|
// LABEL, the node carries a PUBLIC DOMAIN, and the mesh joins them — `<label>.<public-domain>`,
|
|
// with `@` composing to the bare domain. That join is what makes a routed module reachable at all,
|
|
// and before this bed set a public domain it composed to nothing on every node, silently.
|
|
//
|
|
// What is NOT checked here is issuance: whether route-proxy actually obtains a certificate from
|
|
// step-ca over ACME. That path is being fixed in mesh-controller as this is written, and a bed that
|
|
// gated on it would be reporting somebody else's in-flight work as this bed's failure.
|
|
// ================================================================================================
|
|
const adr: string[] = ["================ ADR 0066: LABELLED ROUTES ================"];
|
|
|
|
const wanted: { node: string; module: string; label: string; name: string }[] = [];
|
|
for (const { node, mods } of PLAN) {
|
|
const domain = PUBLIC_DOMAIN[node];
|
|
if (!domain) continue;
|
|
for (const { name } of mods) {
|
|
if (!assigned[node]!.has(name)) continue;
|
|
const label = routeLabelOf(name);
|
|
if (!label) continue;
|
|
wanted.push({ node, module: name, label, name: label === "@" ? domain : `${label}.${domain}` });
|
|
}
|
|
}
|
|
|
|
// The composed names as the MESH wrote them, read from the proxy's own received-routes file —
|
|
// the mesh's answer, on the machine, rather than this test's arithmetic checked against itself.
|
|
const routesFile = (await on("novox", `cat /var/lib/route-proxy/routes/mesh.json 2>&1`)).out;
|
|
const missing: string[] = [];
|
|
const composed: typeof wanted = [];
|
|
for (const w of wanted.filter((w) => w.node === "novox")) {
|
|
const present = routesFile.includes(`"${w.name}"`);
|
|
adr.push(` ${w.module.padEnd(20)} label ${w.label.padEnd(10)} -> ${w.name.padEnd(28)} ${present ? "COMPOSED" : "MISSING"}`);
|
|
if (present) composed.push(w);
|
|
else missing.push(`${w.module} (${w.label} -> ${w.name})`);
|
|
}
|
|
|
|
// And that the proxy answers for one of them. Over HTTP, on the anchor: a certificate is the
|
|
// issuance question, and this one is only whether the name reaches the proxy at all.
|
|
const probe = composed[0];
|
|
const servedCode = probe
|
|
? (await on("novox", `curl -s -o /dev/null -w '%{http_code}' --max-time 10 -H 'Host: ${probe.name}' http://127.0.0.1/`)).out.trim()
|
|
: "";
|
|
adr.push(`\n proxy answers for ${probe?.name ?? "(nothing composed)"}: ${servedCode || "no answer"}`);
|
|
adr.push(` operator root delivered to step-ca: ${caRootDelivered}`);
|
|
adr.push(` step-ca container: ${psMaps["novox"]?.get("step-ca") ?? "MISSING"}`);
|
|
console.log(adr.join("\n"));
|
|
|
|
// ================================================================================================
|
|
// GATING. The headline gates: the cross-segment overlay must FORM for at least one home node
|
|
// (that is the thing this bed exists to prove). Convergence gates on each node's CORE and no
|
|
// non-gap resource failing. The KEEP run is about leaving a browsable instance, so its convergence
|
|
// is reported but not hard-gated; a normal run gates fully.
|
|
// ================================================================================================
|
|
assert.ok(anyHomeFormed,
|
|
`the overlay did NOT form across the access point — no home node could reach novox over the overlay:\n${overlaySummary}`);
|
|
|
|
if (!KEEP) {
|
|
// ADR 0066, the cheap half. Reported on a KEEP run like everything else there.
|
|
assert.deepEqual(missing, [],
|
|
`these routed modules composed no name — a label with no public domain to join it to is a ` +
|
|
`module nothing can reach, and it fails silently:\n${adr.join("\n")}\n\nroutes file:\n${routesFile}`);
|
|
assert.ok(probe && servedCode !== "" && servedCode !== "000",
|
|
`the proxy did not answer for ${probe?.name ?? "any composed name"} (got "${servedCode}"). ` +
|
|
`The name composes, so this is the proxy, not the join:\n${adr.join("\n")}`);
|
|
}
|
|
|
|
if (!KEEP) {
|
|
assert.deepEqual(allProblems, [], `the full mesh did not converge:\n ${allProblems.join("\n ")}\n\n${summary}`);
|
|
} else if (allProblems.length) {
|
|
console.log(`\nCONVERGENCE PROBLEMS (reported, not gated on a KEEP run):\n ${allProblems.join("\n ")}`);
|
|
}
|
|
});
|