Run the forge, on a database the mesh gave it

Everything up to now stopped at composing a declaration. That proves the
control plane and the host agree, and proves nothing about whether the
thing described works — which is how five modules sat pinned to images
that did not exist while parsing and resolving perfectly.

The forge is the right one to run first. It needs a database from
another module, a password it did not choose, and a connection string it
could not have written itself: the address and port come from what the
database serves, the user name from what the mesh decided both ends
would call it. If any of that is wrong it cannot start, and nothing else
in this suite would notice.

The test checks the chain in the order it has to happen — the login
exists, the database it owns exists, the forge answers, and its log does
not say authentication failed. That last one matters: a forge that
started and could not reach its database would still answer on its port.

Rewriting image references is now shared rather than copied from the
bundle, which had the same problem first. A digest is not knowable until
something is built, and when it is, it belongs to whichever registry
served it — so the text says which image and the scenario says which
copy. Matching is on the repository, with a test that a repository
ending in another one is not half-replaced.

Also makes the planning test put the machine back. Tests here share one
mesh, so the five modules it assigned were inherited by whatever ran
next; harmless while nothing pushed, and not harmless now.

scenarios/two-nodes.yml

@@ -40,6 +40,10 @@ images:
   - mesh-provision-postgres:development
   # And the proxy, which is what turns a route grant into traffic actually arriving.
   - mesh-route-proxy:development
+  # And a forge, so one of the real module descriptions can be started rather than only planned.
+  # It is the first of them to run: it needs a database from another module, a credential it did
+  # not choose, and a connection string it could not have written itself.
+  - gitea/gitea:1.22
 
 place:
   all: [host, runtime]

src/pinning.ts

@@ -0,0 +1,56 @@
+/**
+ * Rewriting an image reference to the one a scenario's own registry serves.
+ *
+ * **A digest is not knowable until something is built** (novox/hq 04-ISSUES/025). A manifest in a
+ * repository can pin a third-party image, because somebody can ask a registry what a tag points
+ * at. It cannot pin an image the mesh builds itself: that image does not exist yet, and when it
+ * does its digest belongs to whichever registry served it.
+ *
+ * The bundle has always had this problem and solves it by rewriting references once the scenario's
+ * registry is up and its digests are known. Modules have exactly the same problem and were solving
+ * it by shipping sixty-four zeros, which parses, resolves, composes — and stops on the machine.
+ *
+ * So the rewriting is shared rather than copied, and matches on the **repository**, because that
+ * is the part a person writes and the only part that survives being served somewhere else.
+ */
+
+/** `192.0.2.250:5000/ghcr.io/mailu/admin@sha256:…` → `ghcr.io/mailu/admin` */
+export function repositoryOf(pinned: string): string {
+  const at = pinned.indexOf("@");
+  const body = at === -1 ? pinned : pinned.slice(0, at);
+  const slash = body.indexOf("/");
+  // Everything after the registry. A reference with no slash at all is its own repository.
+  return slash === -1 ? body : body.slice(slash + 1);
+}
+
+/**
+ * Replace every reference to a stocked repository with the reference this scenario serves.
+ *
+ * Matching is on the repository and ignores whatever registry and digest were written down —
+ * a file may name `postgres@sha256:7456…` or `mesh-provision-postgres@sha256:0000…` and both mean
+ * *the postgres this scenario has*. That is the whole point: the text says which image, the
+ * scenario says which copy.
+ *
+ * A repository the scenario did not stock is left alone rather than blanked. It may be reachable
+ * some other way, and silently emptying a reference would produce the exact failure this exists to
+ * prevent.
+ */
+export function pinnedInto(text: string, served: string[]): string {
+  let out = text;
+  for (const pinned of served) {
+    const repository = repositoryOf(pinned);
+    const escaped = repository.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+    // Optionally a registry, then the repository, then any digest. Anchored on a quote or
+    // whitespace so a longer repository ending in a shorter one is not half-replaced.
+    out = out.replaceAll(
+      new RegExp(`(?<=^|["\\s])(?:[A-Za-z0-9_.:-]+\\/)*${escaped}@sha256:[0-9a-f]{64}`, "g"),
+      pinned,
+    );
+  }
+  return out;
+}
+
+/** Whether anything is still pinned to a placeholder, which would fail on the machine. */
+export function stillUnpinned(text: string): string[] {
+  return [...text.matchAll(/([A-Za-z0-9_.:/-]+)@sha256:0{64}/g)].map((m) => m[1]!);
+}

test/integration/mesh.test.ts

@@ -21,6 +21,7 @@ import assert from "node:assert/strict";
 import { existsSync, readFileSync } from "node:fs";
 import { loadScenario } from "../../src/declaration/parse.ts";
 import { raise } from "../../src/lifecycle/raise.ts";
+import { pinnedInto, stillUnpinned } from "../../src/pinning.ts";
 import { destroy, exec } from "../../src/lifecycle/operate.ts";
 import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts";
 import { labIsUsable, destroyAll } from "./harness.ts";
@@ -1800,4 +1801,75 @@ test("the real modules resolve together, and compose a declaration a host accept
         `${c.name} carries something secret-shaped in env.${key}, which the broker would see`);
     }
   }
+
+  // Put the machine back. **Tests here share one mesh**, so what this one assigns is what the
+  // next one inherits — and this one assigns five modules whose images are mostly not stocked.
+  // Planning them is harmless; leaving them assigned makes the next push try to start them.
+  for (const name of modules) await mesh(`unassign anchor ${name}`);
+});
+
+// The first of the real module descriptions to actually run.
+//
+// **Everything before this stopped at composing a declaration.** That proves the control plane and
+// the host agree, and proves nothing about whether the thing described works — which is how five
+// modules sat pinned to images that did not exist, parsing and resolving perfectly
+// (novox/hq 04-ISSUES/025).
+//
+// The forge is the one worth running first. It needs a database from another module, a password it
+// did not choose, and a connection string it could not have written itself: the address and port
+// come from what the database serves, and the user name from what the mesh decided both ends would
+// call it (04-ISSUES/022 and 023). If any of that is wrong it cannot start, and nothing else in
+// this file would notice.
+test("the forge runs, on a database the mesh gave it", { skip, timeout: 900_000 }, async () => {
+  for (const name of ["postgres", "gitea"]) {
+    const raw = readFileSync(`${process.env["MESH_LAB_MODULES"]}/${name}.json`, "utf8");
+    // An image the mesh builds has no digest until it is built, and one it does not build belongs
+    // to whichever registry served it. Both are answered by this scenario's own registry.
+    const pinned = pinnedInto(raw, stocked);
+    assert.deepEqual(stillUnpinned(pinned), [],
+      `${name} still names an image nothing serves, so it could not start`);
+    await must("anchor", `printf %s ${quote(pinned)} > /run-${name}.json`);
+    await must("anchor", `docker cp /run-${name}.json mesh-control:/run-${name}.json`);
+    await mesh(`module add /run-${name}.json`);
+    await mesh(`assign anchor ${name}`);
+  }
+  await mesh("push anchor", 300_000);
+
+  // The database first: until the provisioner has made the login, the forge has nothing to
+  // connect to and its own start would prove only that it retries.
+  const psql = async (q: string) =>
+    (await on("anchor",
+      `docker exec postgres psql -U postgres -qAt -c ${quote(q)}`, 60_000)).out.trim();
+
+  let made = "";
+  for (let i = 0; i < 40 && made !== "t"; i++) {
+    made = await psql("select true from pg_roles where rolname = 'mesh_anchor_gitea'");
+    if (made !== "t") await new Promise((r) => setTimeout(r, 3000));
+  }
+  assert.equal(made, "t",
+    `no login was created for the forge:\n${(await on("anchor", "docker logs mesh-provision-postgres 2>&1 | tail -20")).out}`);
+  assert.equal(await psql("select true from pg_database where datname = 'gitea'"), "t",
+    "the login exists and the database it owns does not");
+
+  // And the forge itself, answering. Not that its container exists — that it serves.
+  let answered = false;
+  let said = { out: "", ok: false };
+  for (let i = 0; i < 60 && !answered; i++) {
+    said = await on("anchor", `curl -sf -o /dev/null -w '%{http_code}' --max-time 5 http://127.0.0.1:3000/`, 30_000);
+    answered = said.out.trim().startsWith("2") || said.out.trim() === "303";
+    if (!answered) await new Promise((r) => setTimeout(r, 5000));
+  }
+  assert.ok(answered,
+    `the forge never answered (last: ${said.out.trim()}):\n` +
+      `${(await on("anchor", "docker logs gitea 2>&1 | tail -25")).out}`);
+
+  // **The credential actually worked.** A forge that started and could not reach its database
+  // would still answer on its port, so the log is where the difference lives.
+  const log = (await on("anchor", "docker logs gitea 2>&1 | tail -60")).out;
+  assert.doesNotMatch(log, /password authentication failed|connection refused|does not exist/i,
+    `the forge started and could not use the database it was given:\n${log}`);
+
+  await mesh("unassign anchor gitea");
+  await mesh("unassign anchor postgres");
+  await mesh("push anchor", 300_000);
 });

test/pinning.test.ts

@@ -0,0 +1,73 @@
+import { test } from "node:test";
+import assert from "node:assert/strict";
+
+import { pinnedInto, repositoryOf, stillUnpinned } from "../src/pinning.ts";
+
+const SERVED = [
+  "192.0.2.250:5000/postgres@sha256:" + "a".repeat(64),
+  "192.0.2.250:5000/mesh-provision-postgres@sha256:" + "b".repeat(64),
+  "192.0.2.250:5000/gitea/gitea@sha256:" + "c".repeat(64),
+  "192.0.2.250:5000/ghcr.io/mailu/admin@sha256:" + "d".repeat(64),
+];
+
+test("the repository is what survives being served somewhere else", () => {
+  assert.equal(repositoryOf(SERVED[0]!), "postgres");
+  assert.equal(repositoryOf(SERVED[2]!), "gitea/gitea");
+  assert.equal(repositoryOf(SERVED[3]!), "ghcr.io/mailu/admin");
+  assert.equal(repositoryOf("alpine"), "alpine");
+});
+
+// The case this exists for: an image the mesh builds has no digest until it is built, so a
+// manifest ships sixty-four zeros and would stop on the machine (novox/hq 04-ISSUES/025).
+test("a placeholder for one of our own images becomes the one this scenario serves", () => {
+  const before = `"image": "mesh-provision-postgres@sha256:${"0".repeat(64)}"`;
+  const after = pinnedInto(before, SERVED);
+  assert.match(after, /192\.0\.2\.250:5000\/mesh-provision-postgres@sha256:b{64}/);
+  assert.deepEqual(stillUnpinned(after), []);
+});
+
+// And a real third-party digest is replaced too — the text says which image, the scenario says
+// which copy of it.
+test("a real digest is redirected to this scenario's copy", () => {
+  const before = `"image": "gitea/gitea@sha256:${"f".repeat(64)}"`;
+  assert.match(pinnedInto(before, SERVED), /192\.0\.2\.250:5000\/gitea\/gitea@sha256:c{64}/);
+});
+
+test("a reference that already carries a registry is still redirected", () => {
+  const before = `"image": "docker.io/postgres@sha256:${"e".repeat(64)}"`;
+  assert.match(pinnedInto(before, SERVED), /192\.0\.2\.250:5000\/postgres@sha256:a{64}/);
+});
+
+// **Left alone, not blanked.** A repository this scenario did not stock may be reachable some
+// other way, and emptying the reference would produce the exact failure this prevents.
+test("something the scenario does not serve is untouched", () => {
+  const before = `"image": "redis@sha256:${"9".repeat(64)}"`;
+  assert.equal(pinnedInto(before, SERVED), before);
+});
+
+// A longer repository ending in a shorter one must not be half-replaced.
+test("a repository that ends in another one is not partly rewritten", () => {
+  const before = `"image": "my-postgres@sha256:${"7".repeat(64)}"`;
+  assert.equal(pinnedInto(before, SERVED), before,
+    "'my-postgres' was rewritten because it ends in 'postgres'");
+});
+
+test("every image in a whole manifest is redirected at once", () => {
+  const manifest = JSON.stringify({
+    resources: [
+      { id: "db", image: `postgres@sha256:${"1".repeat(64)}` },
+      { id: "prov", image: `mesh-provision-postgres@sha256:${"0".repeat(64)}` },
+      { id: "app", image: `gitea/gitea@sha256:${"2".repeat(64)}` },
+    ],
+  });
+  const after = pinnedInto(manifest, SERVED);
+  assert.deepEqual(stillUnpinned(after), []);
+  for (const want of ["a".repeat(64), "b".repeat(64), "c".repeat(64)]) {
+    assert.ok(after.includes(want), `missing ${want.slice(0, 6)}… in ${after}`);
+  }
+});
+
+test("what is still a placeholder can be named", () => {
+  const text = `"image": "something-of-ours@sha256:${"0".repeat(64)}"`;
+  assert.deepEqual(stillUnpinned(text), ["something-of-ours"]);
+});