Register R92 as a bed: genesis on a runtime that trusts no registry (hq issue 92)

The lab's base image trusts the documentation ranges as plain-HTTP registries, so no genesis bed could
see the push to an untrusted registry that issue 92 is about (reviews 73 and 76 on mesh-host #91). The
bed takes that trust off the anchor before the installer runs and asks the runtime it trusts loopback
only, then runs genesis with --registry 192.0.2.10:5000 and holds that the control plane reaches the
registry, trusted by the mesh's name too.

The prover gains a bed kind: it builds the node-engine and its installer at each commit, the carried
builder from the pinned controller, and raises the bed; a bed that never reached the step it replays
skips, and a skip is an error, never the fault reproduced.

Not proved yet: genesis stops at step 6 on both sides, because the temporary controller has no
MESH_BUS_NATS and the bus template is refused by the installer (issue 146).

test/integration/replay92.test.ts

@@ -0,0 +1,238 @@
+/**
+ * REPLAY R92 — genesis pushes to its own registry on a runtime nobody told to trust it (novox/hq issue 92).
+ *
+ * On the first real genesis the push of the control plane's image into the registry genesis had just
+ * raised was refused: `Get "https://<address>/v2/": http: server gave HTTP response to HTTPS client`. The
+ * registry speaks plain HTTP on purpose (ADR 0082), and the trust that lets a runtime speak to it was
+ * written by the `docker` module, installed six steps after the push. The fix (mesh-host #91) has genesis
+ * write that trust into the runtime's configuration itself, and confirm it through `docker info`, before
+ * anything is pushed.
+ *
+ * **The lab hid it.** Its base image trusts the documentation ranges as plain-HTTP registries before any bed
+ * starts (src/lifecycle/base.ts), so every genesis bed pushed to a registry its runtime already trusted. This
+ * bed takes that trust back off the anchor — the base's file removed, the runtime restarted, and the runtime
+ * asked that it now trusts loopback and nothing else — and then runs the installer as an operator does, with
+ * `--registry 192.0.2.10:5000`.
+ *
+ * What it holds: genesis gets past its publish step, the registry it raised serves the control plane's image
+ * it pushed by the address it was given, and the runtime trusts that registry by the mesh's name too, which
+ * the builder pushes to later. Before the fix the push is refused, and the bed fails naming it.
+ *
+ * **A bed that could not reach the step it replays says so and does not fail**: a genesis stopped before
+ * its registry step — a pull the internet refused, a build that broke — proves nothing about the order this
+ * replays, and the test is skipped with the reason, which the prover reads as an error, never as the fault
+ * reproduced. Genesis is run once and not asked to finish: the steps after the control plane read
+ * repositories this bed does not serve, and are not what it replays.
+ *
+ *   MESH_LAB_HOST_BINARY=<mesh-host>/mesh-host        MESH_LAB_BOOTSTRAP_BINARY=<mesh-host>/mesh-bootstrap
+ *   MESH_LAB_BUNDLE=<mesh-host>/examples/foundation-first-node.lock (the installer refuses the bus one)
+ *   MESH_LAB_CATALOG=<mesh-catalog>/modules           MESH_REPLAY_CONTROLLER=<a mesh-controller checkout>
+ *   MESH_LAB_KEEP=1   leave it standing
+ *
+ * Run by `go run ./replays/cmd/prove R92`, which builds all of that at the fix and at the commit before it.
+ */
+import { test, before, after } from "node:test";
+import assert from "node:assert/strict";
+import { existsSync, mkdtempSync, rmSync } from "node:fs";
+import { execFileSync } from "node:child_process";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+import { loadScenario } from "../../src/declaration/parse.ts";
+import { raise } from "../../src/lifecycle/raise.ts";
+import { destroy, exec, push } from "../../src/lifecycle/operate.ts";
+import { bootstrapBinaryPath, hostBinaryPath } from "../../src/lifecycle/place.ts";
+import { labIsUsable, destroyAll, foundationBundle } from "./harness.ts";
+import { genesis, type GenesisResult } from "./genesis.ts";
+
+const SCENARIO = "genesis-untrusted";
+const ANCHOR = "anchor";
+const FORGE = "forge";
+const ANCHOR_ADDRESS = "192.0.2.10";
+const FORGE_ADDRESS = "192.0.2.30";
+/** The registry as the operator gives it: the anchor's own address and the registry's port. */
+const REGISTRY = `${ANCHOR_ADDRESS}:5000`;
+/** And as the mesh names it: the node, the mesh's internal suffix, the same port. */
+const REGISTRY_BY_NAME = `${ANCHOR}.internal:5000`;
+/** The installer's steps up to and including the one this replays, in its order. */
+const UP_TO_PUBLISH = ["preflight", "load", "build", "bundle", "apply", "verify", "enrol", "operator", "registry", "publish"];
+/** The steps before the registry's: a stop there never reached what this replays. */
+const BEFORE_REGISTRY = UP_TO_PUBLISH.slice(0, UP_TO_PUBLISH.indexOf("registry"));
+
+const capability = await labIsUsable();
+const binary = hostBinaryPath();
+const installer = bootstrapBinaryPath();
+const bundle = process.env["MESH_LAB_BUNDLE"] ?? "";
+const catalogDir = process.env["MESH_LAB_CATALOG"] ?? "";
+const controller = process.env["MESH_REPLAY_CONTROLLER"] ?? "";
+const KEEP = !!process.env["MESH_LAB_KEEP"];
+const FIXED_ID = process.env["MESH_LAB_INSTANCE_ID"] ?? (KEEP ? "genesis-untrusted-live" : undefined);
+
+const skip =
+  !capability.usable ? capability.why :
+  !binary ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" :
+  !installer ? "MESH_LAB_BOOTSTRAP_BINARY is not set to a built mesh-bootstrap" :
+  !bundle || !existsSync(bundle) ? "MESH_LAB_BUNDLE is not set to a foundation template" :
+  !catalogDir || !existsSync(catalogDir) ? "MESH_LAB_CATALOG is not set to mesh-catalog/modules" :
+  !controller || !existsSync(resolve(controller, ".git")) ? "MESH_REPLAY_CONTROLLER is not set to a mesh-controller checkout" :
+  false;
+
+let instanceId = "";
+/** Why the bed never reached the step it replays, or "" when it did. */
+let unreached = "";
+let result: GenesisResult | undefined;
+let untrusted = "";
+const report: string[] = [];
+
+async function on(machine: string, command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> {
+  const { stdout } = await exec(instanceId, machine, ["sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`], timeoutMs);
+  const marker = stdout.lastIndexOf("__exit=");
+  if (marker < 0) return { out: stdout, ok: false };
+  return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" };
+}
+async function must(machine: string, command: string, timeoutMs?: number): Promise<string> {
+  const { out, ok } = await on(machine, command, timeoutMs);
+  if (!ok) throw new Error(`${machine}: ${command}\n${out}`);
+  return out;
+}
+
+/** The control plane's repository at the checkout's HEAD, served from the forge machine over git's protocol. */
+async function raiseForge(): Promise<string> {
+  const dir = mkdtempSync(join(tmpdir(), "mesh-lab-r92-forge-"));
+  const head = execFileSync("git", ["-C", controller, "rev-parse", "HEAD"], { encoding: "utf8" }).trim();
+  try {
+    const bare = join(dir, "mesh-controller.git");
+    execFileSync("git", ["init", "-q", "--bare", bare]);
+    // No collection after the push: one run in the background rewrites the objects while they are carried.
+    execFileSync("git", ["-C", bare, "config", "receive.autogc", "false"]);
+    execFileSync("git", ["-C", controller, "push", "-q", "--force", bare, "HEAD:refs/heads/main"], { stdio: "pipe" });
+    const tar = join(tmpdir(), `mesh-lab-r92-forge-${process.pid}.tar`);
+    execFileSync("tar", ["-cf", tar, "-C", dir, "."]);
+    await push(instanceId, FORGE, tar, "/tmp/forge.tar");
+    rmSync(tar, { force: true });
+  } finally {
+    rmSync(dir, { recursive: true, force: true });
+  }
+  // Owned by the daemon's user, or git refuses to serve it ("dubious ownership") and says nothing.
+  await must(FORGE, `mkdir -p /srv/git && tar --no-same-owner -xf /tmp/forge.tar -C /srv/git && chown -R root:root /srv/git && ` +
+    `git daemon --base-path=/srv/git --export-all --reuseaddr --detach --listen=${FORGE_ADDRESS} --pid-file=/run/git-daemon.pid`);
+  for (let i = 0; ; i++) {
+    if ((await on(ANCHOR, `git ls-remote git://${FORGE_ADDRESS}/mesh-controller.git main`)).ok) break;
+    if (i >= 30) throw new Error(`the anchor cannot reach the forge at git://${FORGE_ADDRESS}/`);
+    await new Promise((r) => setTimeout(r, 2_000));
+  }
+  return head;
+}
+
+/**
+ * The anchor's runtime made a bare machine's: the base's trust of the documentation ranges taken off, and
+ * the runtime asked that it trusts no registry but loopback. Before genesis, so nothing runs that a restart
+ * would stop. Answers what the runtime said.
+ */
+async function untrust(): Promise<string> {
+  await must(ANCHOR, `rm -f /etc/docker/daemon.json && systemctl restart docker`, 180_000);
+  for (let i = 0; ; i++) {
+    if ((await on(ANCHOR, `docker info >/dev/null`)).ok) break;
+    if (i >= 60) throw new Error("the anchor's runtime did not answer after its trust was taken off");
+    await new Promise((r) => setTimeout(r, 2_000));
+  }
+  const config = (await must(ANCHOR, `docker info --format '{{json .RegistryConfig}}'`)).trim();
+  const parsed = JSON.parse(config) as { InsecureRegistryCIDRs?: string[]; IndexConfigs?: Record<string, { Secure?: boolean }> };
+  const cidrs = parsed.InsecureRegistryCIDRs ?? [];
+  const insecureNames = Object.entries(parsed.IndexConfigs ?? {}).filter(([, c]) => c.Secure === false).map(([n]) => n);
+  if (cidrs.some((c) => c !== "127.0.0.0/8" && c !== "::1/128") || insecureNames.length > 0) {
+    throw new Error(`the anchor's runtime still trusts more than loopback after the base's trust was taken off: ${config}`);
+  }
+  return `trusts ${cidrs.join(", ") || "nothing"} as plain-HTTP registries, and no registry by name`;
+}
+
+before(async () => {
+  if (skip) return;
+  const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), {
+    onProgress: (m) => console.log(`raise: ${m}`),
+    ...(FIXED_ID ? { instanceId: FIXED_ID } : {}),
+  });
+  instanceId = raised.instanceId;
+  console.log(`INSTANCE ${instanceId}${KEEP ? " (KEEP — will be left standing)" : ""}`);
+
+  let head = "";
+  try {
+    head = await raiseForge();
+    report.push(`  forge             git://${FORGE_ADDRESS}/mesh-controller.git at ${head.slice(0, 8)}`);
+    untrusted = await untrust();
+    report.push(`  anchor's runtime  ${untrusted}`);
+  } catch (err) {
+    unreached = `the bed was not prepared, so the installer never ran: ${(err as Error).message}`;
+    return;
+  }
+
+  const forge = (repo: string) => `git://${FORGE_ADDRESS}/${repo}.git`;
+  try {
+    result = await genesis({
+      instanceId,
+      node: ANCHOR,
+      installer: installer as string,
+      catalogDir,
+      // The registry's manifest is all the steps this replays read from the catalogue.
+      catalogueModules: ["distribution"],
+      bundleTemplate: foundationBundle(bundle, []),
+      registry: REGISTRY,
+      source: forge("mesh-controller"),
+      sourceRef: head,
+      // Named because the installer requires them; served by nobody, since the steps that clone them
+      // come after the one replayed here.
+      toolsSource: forge("mesh-tools"),
+      sdkSource: forge("mesh-sdk"),
+      catalogSource: forge("mesh-catalog"),
+      attempts: 1,
+      verify: false,
+      log: (m) => console.log(m),
+    });
+  } catch (err) {
+    unreached = `the installer could not be run: ${(err as Error).message}`;
+    return;
+  }
+  report.push(...result.report);
+  const stoppedAt = result.step.replace(/^step \d+ of \d+, /, "");
+  if (!result.ok && (BEFORE_REGISTRY.includes(stoppedAt) || !result.step)) {
+    unreached = `genesis stopped at ${result.step || "no step it named"}, before its registry step, so the order ` +
+      `this replays was never reached:\n${result.why}`;
+  }
+}, { timeout: 3_600_000 });
+
+after(async () => {
+  if (KEEP) {
+    console.log(`\nLEFT STANDING: ${instanceId} — not destroyed (MESH_LAB_KEEP).`);
+    return;
+  }
+  if (instanceId) await destroy(instanceId);
+  await destroyAll(`${SCENARIO}-`);
+}, { timeout: 900_000 });
+
+test("genesis pushes to the registry it raised on a runtime nobody told to trust it", {
+  skip, timeout: 3_600_000,
+}, async (t) => {
+  console.log(report.join("\n"));
+  if (unreached) {
+    t.skip(`R92 NOT REACHED: ${unreached}`);
+    return;
+  }
+  const ran = result as GenesisResult;
+  const stoppedAt = ran.step.replace(/^step \d+ of \d+, /, "");
+
+  // The incident's own words, and the step it stopped at.
+  assert.doesNotMatch(ran.said, /server gave HTTP response to HTTPS client/,
+    `the runtime refused to push to ${REGISTRY} over plain HTTP — the registry was not trusted before the push ` +
+    `(stopped at ${ran.step || "no step"}):\n${ran.why}`);
+  assert.ok(ran.ok || !UP_TO_PUBLISH.includes(stoppedAt),
+    `genesis stopped at ${ran.step}, before its control plane was published to ${REGISTRY}:\n${ran.why}`);
+
+  // The registry agreeing: the image genesis pushed, by the address it was given.
+  const tags = (await on(ANCHOR, `curl -s --max-time 10 http://${REGISTRY}/v2/mesh-controller/tags/list`)).out.trim();
+  assert.match(tags, /"genesis"/, `${REGISTRY} does not serve the control plane genesis pushed: ${tags || "no answer"}`);
+
+  // And the runtime trusts the registry by the mesh's name as well: the builder pushes there later.
+  const config = (await must(ANCHOR, `docker info --format '{{json .RegistryConfig.IndexConfigs}}'`)).trim();
+  const byName = (JSON.parse(config) as Record<string, { Secure?: boolean }>)[REGISTRY_BY_NAME];
+  assert.ok(byName && byName.Secure === false,
+    `the runtime does not trust ${REGISTRY_BY_NAME}, the name the mesh pushes to its registry by: ${config}`);
+});