novox/mesh-lab
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

The beds name images the way a machine would find them

Browse Source 
Twenty-eight integration tests each carried their own copy of the same two helpers,
which pointed a manifest and the substrate bundle at whatever the lab's registry had
assigned. They now share two in the harness, and the difference is the point: ours is
rewritten to the ID the machine holds it under, and everything else is left exactly as
written so the machine pulls it.

**The substrate bundle is where the fiction was most load-bearing.** mesh-host's
`examples/substrate-first-node.lock` pins all three of its images at
`192.0.2.250:5000/…`, which is the address the lab's registry served from — it was
written for a target, and the target was the lab. Two of those are ordinary third-party
images and become the digests mesh-catalog's own postgres and lavinmq modules pin, so
the substrate's store and broker are literally the images the mesh runs. mesh-control
exists in no registry at all and becomes the ID the machine was handed. **The bundle
itself should be fixed in mesh-host and this substitution deleted with it.**

Beds that wrote a manifest by hand named an image by repository and let the rewrite
supply a digest. There is nothing to supply one now, so `onTheMachine` refuses an
unpinned reference and hands back the digest the catalogue pins — a bed runs the image
the mesh ships, and a bed that drifts from the catalogue is testing a different
postgres.

Three beds took a third-party image out of the raised list, which no longer contains
one: certificates (pebble), objectstore (minio and its client) and provisioner
(postgres) now name theirs and pull it. builds and mesh publish into the MESH's own
artifact store — the `registry` module's image, on the node, on 5000 — rather than into
scenery the lab raised. That is a different claim, and only one of them exists in
production.

New unit tests cover what a full raise would otherwise be the only way to check: the
routes an egress machine gets (that its gateway is still the path to the rest of the
scenario, that a range with no path is unreachable rather than leaked to the uplink,
that each family gets its own next hop), which machine is handed which of our images,
and the `images:` rule that refuses a third-party entry. The "shipped scenarios are
valid" test now loads every scenario rather than two of them.

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
This commit is contained in:
jschoubben
2026-09-10 23:16:41 +02:00
parent 5c91c0ecd2
commit 675facdb0d
40 changed files with 898 additions and 705 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/integration/mesh.test.ts
+42 -49
View File
@@ -21,10 +21,10 @@ import assert from "node:assert/strict";
import { existsSync, readFileSync } from "node:fs";
import { loadScenario } from "../../src/declaration/parse.ts";
import { raise } from "../../src/lifecycle/raise.ts";
import { pinnedInto, stillUnpinned } from "../../src/pinning.ts";
import { pinnedInto, stillUnpinned, type HeldImage } from "../../src/pinning.ts";
import { destroy, exec } from "../../src/lifecycle/operate.ts";
import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts";
import { labIsUsable, destroyAll } from "./harness.ts";
import { labIsUsable, destroyAll, substrateBundle, onTheMachine } from "./harness.ts";
import { incus } from "../../src/incus/client.ts";
import { machineName } from "../../src/lifecycle/names.ts";
import { ready, returnTo, keep, rememberStock, warmStock } from "../../src/warm.ts";
@@ -49,23 +49,27 @@ const skip = !capability.usable
const SCENARIO = "two-nodes";
let instanceId = "";
/** The scenario's own registry, which serves the images a module may mirror. */
let registry = "";
/** What that registry actually serves, by repository. */
let stocked: string[] = [];
/**
* The pinned reference for one of the scenario's images.
* The MESH's own artifact store, once the registry module is running on the anchor.
*
* By digest, because the lab's registry drops tags when it stocks: `registry:2` is not there and
* asking for it fails with "not found", which reads like a missing image rather than a naming
* convention. A digest is also what a declaration pins, so this is the reference a module would
* really carry.
* Not a registry the lab raised — there is no longer any such thing. A build publishes into the
* store the mesh itself runs, which is the only registry that exists outside this repository.
*/
function pinned(repository: string): string {
const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository);
assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`);
return found;
const registry = "127.0.0.1:5000";
/**
* The registry module's image, pinned upstream, pulled by the machine over its uplink.
*
* The digest mesh-catalog's `registry` module pins, so the store the mesh runs here is the store
* the mesh runs anywhere.
*/
const ARTIFACT_STORE =
"registry@sha256:a3d8aaa63ed8681a604f1dea0aa03f100d5895b6a58ace528858a7b332415373";
/** The mesh's own images, as the machines hold them. */
let held: HeldImage[] = [];
/** What a manifest's image reference becomes on the machine — ours by ID, everything else as written. */
function pinned(reference: string): string {
return onTheMachine(reference, held);
}
function quote(s: string): string {
@@ -179,23 +183,14 @@ async function settled(node: string, withinMs = 480_000): Promise<void> {
}
/**
* The bundle, with every image reference pointed at this scenario's registry.
* The bundle, as a machine should receive it.
*
* Matched by repository rather than by the whole reference, because the address and the digest
* both differ from whatever the committed bundle names — and a bundle that names the wrong
* registry is not wrong, it is built for a different target.
* The committed example was written for a target that had a registry the lab raised. Its two
* third-party images become upstream references the machine pulls itself; mesh-control, which
* exists in no registry, becomes the ID this machine was handed.
*/
function bundleFor(images: string[]): string {
let text = readFileSync(bundle, "utf8");
for (const pinned of images) {
const repository = pinned.slice(pinned.indexOf("/") + 1, pinned.indexOf("@"));
const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\.");
text = text.replaceAll(
new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"),
pinned,
);
}
return text;
function bundleFor(images: HeldImage[]): string {
return substrateBundle(bundle, images);
}
/** Take a token out of what `token issue` printed. It is the one base64url blob on its own line. */
@@ -219,7 +214,7 @@ before(async () => {
if (said.use === "restore") {
instanceId = said.instanceId;
const seconds = await returnTo(instanceId);
stocked = warmStock(instanceId).images;
held = warmStock(instanceId).images;
// **A snapshot captures disk, not memory.** Restoring reboots the machine, so everything
// this suite started by hand is gone — the host most of all. Without it the mesh looks
@@ -255,13 +250,11 @@ before(async () => {
instanceId = raised.instanceId;
// The first node raises everything from a file rather than from a bundle built into the binary,
// because the digests are this registry's and are not known until it is up.
// because the control plane's image is named by the ID this machine holds it under, which is not
// knowable until it has been handed over.
held = raised.images;
await must("anchor", `cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`);
await must("anchor", `${HOST_PATH} apply /tmp/substrate.lock`);
stocked = raised.images;
const first = raised.images[0];
assert.ok(first, "the scenario stocked no images, so nothing can be mirrored");
registry = first.slice(0, first.indexOf("/"));
// A build machine, so anything here can ask the mesh to build something. Placed rather than
// assumed: nothing else in this scenario would start one.
@@ -279,7 +272,7 @@ before(async () => {
if (warming) {
// Snapshotted only now, with everything up: a state worth returning to is the one after the
// part nobody wants to repeat.
await rememberStock(instanceId, stocked);
await rememberStock(instanceId, held);
const warm = await keep(SCENARIO, instanceId);
console.log(`warm: ${warm.instanceId} kept, against ` +
Object.entries(warm.against).map(([n, c]) => `${n} ${c}`).join(", "));
@@ -608,13 +601,13 @@ test("a machine that fell behind catches up without being named", { skip, timeou
});
test("the mesh runs its own artifact store", { skip, timeout: 900_000 }, async () => {
// Artifacts go to a registry, and the only registries that existed were raised by the lab or by
// the bootstrap bundle. A mesh had no way to run its own.
// Artifacts go to a registry, and a mesh had no way to run its own — the only one that existed
// was raised by the lab, which is to say it existed nowhere but here.
//
// **Named, not mirrored** (novox/hq 04-ISSUES/029). Mirroring publishes to the artifact store,
// and the builder will not start without one — so a module that provides the store and builds
// its own image asks the mesh to put an artifact into the thing that artifact is needed to
// create. It worked here only because the scenario's registry was already standing to receive
// create. It used to pass here only because the LAB's registry was already standing to receive
// the push, which is exactly why a real first mesh would have found this and the lab did not.
//
// So the image is named by digest, the way the bundle names the three a first node starts from.
@@ -626,7 +619,7 @@ test("the mesh runs its own artifact store", { skip, timeout: 900_000 }, async (
`"serves":{"artifact-store":{"port":5000}},` +
`"resources":[` +
`{"id":"state","type":"directory","path":"/var/lib/mesh/registry","mode":"0700"},` +
`{"id":"store","type":"container","name":"mesh-registry","image":"${pinned("registry")}",` +
`{"id":"store","type":"container","name":"mesh-registry","image":"${ARTIFACT_STORE}",` +
`"ports":["5000:5000"],"volumes":["mesh-registry-data:/var/lib/registry"]}]}' ` +
`> /root/registry/module.json`);
// **Added, not built** — and this is the half that proves the fix. Building needs a builder,
@@ -677,7 +670,7 @@ test("a machine serves its internal name with a certificate the mesh issued", {
// The name it was issued for is the one the mesh gave this machine.
const named = await must("anchor",
`openssl x509 -in /etc/mesh/serving.crt -noout -ext subjectAltName 2>/dev/null || ` +
`docker run --rm -v /etc/mesh:/m ${pinned("registry")} sh -c ` +
`docker run --rm -v /etc/mesh:/m ${ARTIFACT_STORE} sh -c ` +
`"apk add --no-cache openssl >/dev/null 2>&1; openssl x509 -in /m/serving.crt -noout -text" | grep -A1 'Alternative'`);
assert.match(named, /anchor\.internal/, `the certificate is not for this machine's name:\n${named}`);
@@ -1086,7 +1079,7 @@ test("a route is a grant: a workload is reached by the name it asked for", {
`"listens":[{"port":8088,"from":"mesh","why":"the proxy reaches it here"}],` +
`"resources":[{"id":"dir","type":"directory","path":"/etc/storefront","mode":"0755"},` +
`{"id":"app","type":"container","name":"storefront",` +
`"image":"${pinned("registry")}","ports":["8088:5000"]}]}' > /tmp/storefront.json`);
`"image":"${ARTIFACT_STORE}","ports":["8088:5000"]}]}' > /tmp/storefront.json`);
for (const f of ["frontdoor", "storefront"]) {
await must("anchor", `docker cp /tmp/${f}.json mesh-control:/${f}.json`);
await mesh(`module add /${f}.json`);
@@ -1456,7 +1449,7 @@ test("a container reaches another machine by the name the mesh gave it", {
await must("anchor", `printf %s '{"module":"resolves","version":"1",` +
`"capabilities":["container-runtime"],` +
`"resources":[{"id":"idle","type":"container","name":"resolves",` +
`"image":"${pinned("registry")}"}]}' > /tmp/resolves.json`);
`"image":"${ARTIFACT_STORE}"}]}' > /tmp/resolves.json`);
await must("anchor", `docker cp /tmp/resolves.json mesh-control:/resolves.json`);
await mesh("module add /resolves.json");
await mesh("assign laptop resolves");
@@ -1810,7 +1803,7 @@ test("the real modules resolve together, and compose a declaration a host accept
// until it is built — so the file legitimately carries a placeholder, and composing a
// declaration from it is refused (novox/hq 04-ISSUES/025). Planning what could never run is
// what this test used to do.
const pinned = pinnedInto(raw, stocked);
const pinned = pinnedInto(raw, held);
// What this scenario does not serve cannot be redirected, and a module still naming a
// placeholder cannot be planned — the refusal is the point (novox/hq 04-ISSUES/025). Skipped
// and said, rather than silently dropped: a planning test quietly covering four modules
@@ -1934,8 +1927,8 @@ test("the forge runs, on a database the mesh gave it", { skip, timeout: 900_000
for (const name of ["postgres", "gitea"]) {
const raw = readFileSync(`${process.env["MESH_LAB_MODULES"]}/${name}.json`, "utf8");
// An image the mesh builds has no digest until it is built, and one it does not build belongs
// to whichever registry served it. Both are answered by this scenario's own registry.
const pinned = pinnedInto(raw, stocked);
// to whichever registry served it. Only the first is rewritten; the second is pulled.
const pinned = pinnedInto(raw, held);
assert.deepEqual(stillUnpinned(pinned), [],
`${name} still names an image nothing serves, so it could not start`);
await must("anchor", `printf %s ${quote(pinned)} > /run-${name}.json`);
@@ -2021,7 +2014,7 @@ test("a consumer's cache grant means exactly its own keys", { skip, timeout: 600
// keyspace, so the grant is a pattern — and the test is that the pattern means what the
// manifest said, in both directions.
const raw = readFileSync(`${process.env["MESH_LAB_MODULES"]}/redis.json`, "utf8");
const pinned = pinnedInto(raw, stocked);
const pinned = pinnedInto(raw, held);
assert.deepEqual(stillUnpinned(pinned), [],
"redis still names an image nothing serves, so it could not start");
await must("anchor", `printf %s ${quote(pinned)} > /run-redis.json`);