The lab had a registry that production does not, so it tested a fiction
The lab raised a `registry` VM, pushed ~73 images into it from the workstation, and rewrote every manifest reference — third-party ones included — to point at it. No production mesh has such a thing. So every bed proved that a machine could fetch an image from a registry that exists nowhere else, and the bootstrap problems that only appear when a machine has to fetch for itself went unfound. What replaces it is the two things that are true in the world: **Public images come from the public internet.** mesh-lab already created a NAT'd uplink for exactly this and attached it to any machine declaring `egress`; no scenario ever declared it. They do now, and third-party references are left exactly as the catalogue writes them. **The mesh's own images have no registry and never will.** mesh-control, mesh-builder, mesh-route-proxy and the per-module runtimes are built from source and exist in no registry. A machine gets them the way an operator's machine does — they are built here and loaded onto it — and is then named by the digest of its own image configuration, which mesh-host now accepts as "an image this machine already holds". `images:` therefore means only *ours*, and a third-party entry is refused rather than quietly loaded: otherwise the fiction returns one convenient line at a time. It is per-machine as well, because "everything, everywhere" was never a description of anything real — handing whole-mesh-full's union to its two 30GiB workstations would fill the disk with runtimes nothing on them will start. **The uplink and the declared gateway would have fought, silently.** A gateway container and the transit router reach the scenario and nothing else; a default route through either is a black hole for anything outside, and it beats the uplink's DHCP route on metric. So a machine with egress states the scenario's ranges explicitly — through the same gateway or transit it would have defaulted to, so the overlay-across-NAT path is unchanged — and leaves the default to the uplink. A range with no path inside the scenario becomes `unreachable` rather than falling through: 192.168.1.0/24 is an ordinary private range in fact, and letting it escape would put scenario traffic on whatever network the workstation is sitting on. `scenarioRoutesFor` is pure and tested, because a decision only a full raise could check is one nobody checks. The registry-reachability check the raise gained earlier is kept, pointed at the real thing: every machine with egress must resolve a name and reach the internet before the raise says it finished. Same failure it was written for — a raise that returns, an apply that dies on its first pull, an instance left a bare shell — now guarding the path that actually carries. The base image's trust of the documentation ranges as plain-HTTP registries STAYS. It was never only for the lab's registry: the mesh has one of its own, the `registry` module, serving artifacts to the whole mesh over plain HTTP from whatever node runs it. Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
This commit is contained in:
+129
-5
@@ -19,6 +19,7 @@ import { join } from "node:path";
|
||||
|
||||
import { incus, incusOk, succeeds } from "../incus/client.ts";
|
||||
import { around, log, shorten } from "../log.ts";
|
||||
import { isMeshBuilt, repositoryOf, type HeldImage } from "../pinning.ts";
|
||||
|
||||
/**
|
||||
* What this stage can put inside a machine.
|
||||
@@ -263,17 +264,17 @@ export async function placeImage(
|
||||
// sealed machine cannot reach. Measured, not assumed: the load says `Loaded image ID:`
|
||||
// instead of `Loaded image:`, and `docker images` then lists nothing.
|
||||
//
|
||||
// This collides with novox/hq ADR 0006, which pins bundle images BY DIGEST and has the host
|
||||
// refuse anything else. Reconciling the two needs a registry inside the scenario, which is
|
||||
// real design work — see 04-ISSUES/009.
|
||||
// What an archive DOES keep is the image's own ID — the digest of its configuration — and that
|
||||
// is how the mesh's own images are named once loaded. See {@link loadHeldImages}.
|
||||
if (reference.includes("@sha256:")) {
|
||||
throw new PlacementError(
|
||||
machine,
|
||||
`${reference} is pinned by digest, and an image placed from an archive cannot keep its ` +
|
||||
`digest — a repo digest only exists for an image a registry served.\n` +
|
||||
` Placing it would load an image with no name, and a container declaring that digest ` +
|
||||
`would try to reach a registry the machine cannot see.\n` +
|
||||
` Place it by tag, or give the scenario a registry (novox/hq 04-ISSUES/009).`,
|
||||
`would try to reach a registry.\n` +
|
||||
` Place it by tag. What survives being loaded is the image's own ID, which is what a ` +
|
||||
`declaration names it by (mesh-host: an image the machine already holds).`,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -397,3 +398,126 @@ async function waitForRuntime(instanceName: string, machine: string): Promise<vo
|
||||
` systemd is waiting on:\n${jobs.trim() || " (it said nothing)"}`,
|
||||
);
|
||||
}
|
||||
|
||||
// --- the mesh's own images ----------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Which machine is handed which of the mesh's own images.
|
||||
*
|
||||
* A machine that says nothing gets the lot, which is right for a one-machine bed. A machine that
|
||||
* lists some gets those. This is where a workstation running one small module stops paying for
|
||||
* forty runtimes it will never start.
|
||||
*/
|
||||
export function planHeldImages(scenario: Scenario): { machine: string; images: string[] }[] {
|
||||
const all = scenario.images ?? [];
|
||||
if (all.length === 0) return [];
|
||||
return Object.entries(scenario.machines)
|
||||
.map(([machine, spec]) => ({ machine, images: spec.images ?? all }))
|
||||
.filter((plan) => plan.images.length > 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* Put the mesh's own images onto the machines that need them, and say what they are now called.
|
||||
*
|
||||
* **This is what replaced the lab's registry**, and the difference is the whole point. A registry
|
||||
* inside the scenario served every image — third-party ones included — from an address that exists
|
||||
* in no production mesh, so a bootstrap that could only work against it went green here and would
|
||||
* have failed anywhere else. There is no such registry now: third-party images are pulled from the
|
||||
* internet over each machine's `egress` uplink, and the mesh's own arrive the way they arrive on an
|
||||
* operator's machine — somebody built them and put them there.
|
||||
*
|
||||
* The reference a declaration then uses is the image's **own ID**, the digest of its configuration.
|
||||
* `docker load` preserves it, so the name is identical on the workstation that built the image and
|
||||
* on every machine handed a copy — immutable, unforgeable, and requiring nothing to have served it
|
||||
* (mesh-host, *an image may be named by the digest of its own configuration*).
|
||||
*
|
||||
* Read back on both sides. The ID is taken from the workstation and then CONFIRMED on the machine,
|
||||
* because a load that lands a different image than the one exported is exactly the silent fault
|
||||
* this lab exists to catch — and the reference is what every manifest will be rewritten to.
|
||||
*/
|
||||
export async function loadHeldImages(
|
||||
scenario: Scenario,
|
||||
machineNames: Map<string, string>,
|
||||
log: (message: string) => void = () => {},
|
||||
): Promise<HeldImage[]> {
|
||||
const plans = planHeldImages(scenario);
|
||||
if (plans.length === 0) return [];
|
||||
|
||||
const held: HeldImage[] = [];
|
||||
for (const requested of scenario.images ?? []) {
|
||||
// Refused by the validator, so reaching here would be a validator bug — but the consequence
|
||||
// is a third-party image quietly loaded from the workstation instead of pulled, which is the
|
||||
// fiction all of this exists to remove. Cheap to check, expensive to miss.
|
||||
if (!isMeshBuilt(requested)) {
|
||||
throw new Error(
|
||||
`images: '${requested}' is not one of the mesh's own images. It is pulled from the ` +
|
||||
`internet by the machine that needs it, not loaded from this workstation.`,
|
||||
);
|
||||
}
|
||||
|
||||
const wanted = plans.filter((plan) => plan.images.includes(requested)).map((p) => p.machine);
|
||||
if (wanted.length === 0) continue;
|
||||
|
||||
const id = (await local(
|
||||
"docker", ["image", "inspect", "--format", "{{.Id}}", requested], 60_000,
|
||||
)).stdout.trim();
|
||||
if (!/^sha256:[0-9a-f]{64}$/.test(id)) {
|
||||
throw new Error(
|
||||
`${requested} is not on this workstation, so there is nothing to hand the machines.\n` +
|
||||
` It is one of the mesh's own images and exists in no registry — nothing can pull it.\n` +
|
||||
` Build it first (mesh-control's \`make image …\`, or scripts/build-module-runtime.sh).`,
|
||||
);
|
||||
}
|
||||
const image: HeldImage = { requested, repository: repositoryOf(requested), reference: id };
|
||||
|
||||
// Exported once, handed to each machine that asked for it. The archive is the expensive part
|
||||
// and it does not depend on the destination.
|
||||
const tar = join(tmpdir(), `mesh-lab-held-${process.pid}-${Date.now()}.tar`);
|
||||
const saved = await local("docker", ["save", requested, "-o", tar], 900_000);
|
||||
if (!saved.ok) {
|
||||
await unlink(tar).catch(() => {});
|
||||
throw new Error(`cannot export ${requested} from this workstation: ${saved.stderr.trim()}`);
|
||||
}
|
||||
|
||||
try {
|
||||
for (const machine of wanted) {
|
||||
const name = machineNames.get(machine);
|
||||
if (!name) continue;
|
||||
await waitForRuntime(name, machine);
|
||||
await incus(["file", "push", tar, `${name}/tmp/held.tar`], 900_000);
|
||||
const loaded = await incusOk(
|
||||
["exec", name, "--", "docker", "load", "-i", "/tmp/held.tar"], 900_000,
|
||||
);
|
||||
if (!loaded?.includes("Loaded image")) {
|
||||
throw new PlacementError(
|
||||
machine,
|
||||
`${requested} was pushed to ${machine} and did not load.\n` +
|
||||
` The runtime said: ${loaded?.trim() || "nothing"}`,
|
||||
);
|
||||
}
|
||||
// The reference every manifest is about to be rewritten to, confirmed present under
|
||||
// exactly that name. Asking for the tag would prove the load happened; asking for the ID
|
||||
// proves the thing a declaration will name is the thing that is there.
|
||||
const there = (await incusOk(
|
||||
["exec", name, "--", "docker", "image", "inspect", "--format", "{{.Id}}", id], 120_000,
|
||||
))?.trim();
|
||||
if (there !== id) {
|
||||
throw new PlacementError(
|
||||
machine,
|
||||
`${requested} loaded onto ${machine} and is not there as ${id}.\n` +
|
||||
` The runtime answered '${there || "nothing"}'.\n` +
|
||||
` Every manifest naming this image would be rewritten to a reference the machine ` +
|
||||
`does not hold, and nothing serves it — so the apply would stop at the container.`,
|
||||
);
|
||||
}
|
||||
await succeeds(["exec", name, "--", "rm", "-f", "/tmp/held.tar"], 60_000);
|
||||
}
|
||||
} finally {
|
||||
await unlink(tar).catch(() => {});
|
||||
}
|
||||
|
||||
held.push(image);
|
||||
log(` ${requested} → ${id.slice(0, 19)}… on ${wanted.join(", ")}`);
|
||||
}
|
||||
return held;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user