Adopt a third-party workload, and keep a mesh between runs
**The adoption.** Software nobody here wrote, taking its credentials the way such software does — from its environment — and needing two containers that reach each other by name. The first module that could not have been declared this morning: it needs the network shape and it needs a sealed value to reach a container's environment. Its password is accepted rather than generated, which is the whole shape of an adoption: a service that already exists keeps the credential it already has. Asserted properly — a wrong password is refused by the same database, so the passing case means something. **The warm scenario.** A mesh kept between runs and returned to, which turned twelve minutes of bootstrap into thirty seconds of restore. Off unless asked for: a run that is meant to mean something raises from nothing. Its guard fired for real during this work, unprompted — a mesh-host commit landed and it refused the stale base, naming both commits, rather than passing tests against yesterday's binary. That is 04-ISSUES/005's rule one level down. Three things the guard learned the hard way and now handles: a snapshot captures disk and not memory, so the host is restarted after a restore and asserted to have come back; the stocked image digests are worked out while raising and a restored instance never raises, so they are kept; and comparing only the repositories this run can see clears the ones it cannot, so both directions are compared. The one real bug behind five failed attempts was in mesh-host and it reported itself precisely: a network shape the language had and no host implemented. Everything else was scaffolding of mine.
This commit is contained in:
@@ -28,6 +28,10 @@ images:
|
|||||||
# what the mesh's registry is built from — the same chicken-and-egg the bootstrap has, resolved
|
# what the mesh's registry is built from — the same chicken-and-egg the bootstrap has, resolved
|
||||||
# the same way.
|
# the same way.
|
||||||
- registry:2
|
- registry:2
|
||||||
|
# A real third-party workload, for adopting one the way the conversion will. Its database is
|
||||||
|
# the substrate's postgres image rather than its own: what is under test is the mesh delivering
|
||||||
|
# a module, not which postgres it delivers.
|
||||||
|
- ghcr.io/umami-software/umami:postgresql-latest
|
||||||
# And the builder, because it is a module the mesh assigns rather than a program somebody
|
# And the builder, because it is a module the mesh assigns rather than a program somebody
|
||||||
# starts by hand — which is the only way its credential can be one the mesh delivered.
|
# starts by hand — which is the only way its credential can be one the mesh delivered.
|
||||||
- mesh-builder:development
|
- mesh-builder:development
|
||||||
|
|||||||
+35
@@ -30,6 +30,8 @@ const USAGE = `mesh-lab — raise a disposable mesh on one machine
|
|||||||
diagram <scenario.yml> [out.drawio] draw what a scenario asks for
|
diagram <scenario.yml> [out.drawio] draw what a scenario asks for
|
||||||
diagram --live <instance> [out.drawio] draw what is actually raised
|
diagram --live <instance> [out.drawio] draw what is actually raised
|
||||||
|
|
||||||
|
warm the scenario kept between runs, and whether it still counts
|
||||||
|
warm cool destroy it and forget it
|
||||||
suite [paths...] [--no-build] rebuild the artifacts, run the end-to-end tests, leave a receipt
|
suite [paths...] [--no-build] rebuild the artifacts, run the end-to-end tests, leave a receipt
|
||||||
last-run whether the last run still counts; non-zero when it does not
|
last-run whether the last run still counts; non-zero when it does not
|
||||||
|
|
||||||
@@ -118,6 +120,39 @@ async function main(): Promise<void> {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A base state many tests start from, rather than each raising its own mesh.
|
||||||
|
//
|
||||||
|
// **The speed is the lesser half.** Tests that share one long-lived mesh accumulate each
|
||||||
|
// other's state, and a test that reads what the previous one left is a test that passes for
|
||||||
|
// the wrong reason — which has already happened here once. Returning to a named state between
|
||||||
|
// tests makes each of them independent.
|
||||||
|
case "warm": {
|
||||||
|
const { remembered, ready, cool } = await import("./warm.ts");
|
||||||
|
const what = rest[0] ?? "status";
|
||||||
|
if (what === "cool") {
|
||||||
|
const gone = await cool();
|
||||||
|
console.log(gone ? `destroyed ${gone}, and forgot it` : "nothing was being kept warm");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const held = remembered();
|
||||||
|
if (!held) {
|
||||||
|
console.log("nothing is being kept warm.");
|
||||||
|
console.log(" a scenario is warmed by whatever brought it to a state worth keeping;");
|
||||||
|
console.log(" the integration suite does it when MESH_LAB_WARM is set.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
console.log(`${held.instanceId} — ${held.scenario}, warmed ${held.at}`);
|
||||||
|
for (const [name, commit] of Object.entries(held.against)) {
|
||||||
|
console.log(` ${name.padEnd(14)} ${commit}`);
|
||||||
|
}
|
||||||
|
const said = await ready(held.scenario);
|
||||||
|
console.log(said.use === "restore"
|
||||||
|
? "\n usable: it can be returned to"
|
||||||
|
: `\n NOT usable: ${said.why}`);
|
||||||
|
if (said.use !== "restore") process.exitCode = 1;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
case "base": {
|
case "base": {
|
||||||
// `base build` exists because a sealed scenario cannot install a container runtime, and
|
// `base build` exists because a sealed scenario cannot install a container runtime, and
|
||||||
// the runtime has to come from somewhere with a network (novox/hq ADR 0006).
|
// the runtime has to come from somewhere with a network (novox/hq ADR 0006).
|
||||||
|
|||||||
+210
@@ -0,0 +1,210 @@
|
|||||||
|
/**
|
||||||
|
* A scenario kept between runs, already brought to a state worth starting from.
|
||||||
|
*
|
||||||
|
* **Bootstrapping a mesh takes minutes and proves the same thing every time.** The tests worth
|
||||||
|
* iterating on are the ones after it — assigning a module, adopting a workload, watching something
|
||||||
|
* fail. A warm instance is raised once, brought to that state, snapshotted, and restored on every
|
||||||
|
* later run in seconds.
|
||||||
|
*
|
||||||
|
* **The danger is precisely the one 04-ISSUES/005 is about**, one level down: a mesh snapshotted
|
||||||
|
* against yesterday's binaries will pass today's tests and report green, and nothing about the
|
||||||
|
* result would say what it was actually run against. So a warm instance records the commits it was
|
||||||
|
* built from, and is refused — not silently rebuilt, refused — when they have moved.
|
||||||
|
*
|
||||||
|
* **Fresh stays the default.** This is for iterating. A run that is meant to mean something raises
|
||||||
|
* from nothing, because "it passes" must not quietly come to mean "it passes against a mesh
|
||||||
|
* somebody bootstrapped last week".
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { readFileSync, writeFileSync, mkdirSync, rmSync } from "node:fs";
|
||||||
|
import { dirname, join } from "node:path";
|
||||||
|
import { homedir } from "node:os";
|
||||||
|
|
||||||
|
import { list, restore, snapshot, snapshots, destroy } from "./lifecycle/operate.ts";
|
||||||
|
import type { Against } from "./lastrun.ts";
|
||||||
|
import { whatWasTested } from "./lastrun.ts";
|
||||||
|
|
||||||
|
/** The state a warm instance is kept at. One label, because a second is a state nobody named. */
|
||||||
|
export const label = "warm";
|
||||||
|
|
||||||
|
export interface Warm {
|
||||||
|
scenario: string;
|
||||||
|
instanceId: string;
|
||||||
|
/**
|
||||||
|
* The image references the scenario's registry serves, pinned by digest.
|
||||||
|
*
|
||||||
|
* Kept because they are worked out while raising and a restored instance never raises. Without
|
||||||
|
* them a warm run knows nothing about what it can pull, and every test naming an image fails
|
||||||
|
* for a reason that has nothing to do with what it was testing.
|
||||||
|
*/
|
||||||
|
images: string[];
|
||||||
|
/** The commit each repository was at when this was brought to its state. */
|
||||||
|
against: Against;
|
||||||
|
at: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Where the record lives: XDG state, beside the run receipt, for the same reason. */
|
||||||
|
export function recordPath(): string {
|
||||||
|
const state = process.env["XDG_STATE_HOME"] ?? join(homedir(), ".local", "state");
|
||||||
|
return join(state, "mesh-lab", "warm.json");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function remember(warm: Warm): void {
|
||||||
|
const path = recordPath();
|
||||||
|
mkdirSync(dirname(path), { recursive: true });
|
||||||
|
writeFileSync(path, JSON.stringify(warm, null, 2) + "\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function remembered(): Warm | null {
|
||||||
|
try {
|
||||||
|
return JSON.parse(readFileSync(recordPath(), "utf8")) as Warm;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function forget(): void {
|
||||||
|
rmSync(recordPath(), { force: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
export type Verdict =
|
||||||
|
| { use: "restore"; instanceId: string }
|
||||||
|
| { use: "raise"; why: string };
|
||||||
|
|
||||||
|
/**
|
||||||
|
* judge decides whether a remembered instance may be restored.
|
||||||
|
*
|
||||||
|
* **Every reason to refuse is a reason a test would otherwise pass while meaning nothing**, so
|
||||||
|
* each is named rather than collapsed into "not usable".
|
||||||
|
*/
|
||||||
|
export function judge(
|
||||||
|
warm: Warm | null,
|
||||||
|
scenario: string,
|
||||||
|
standing: string[],
|
||||||
|
hasSnapshot: boolean,
|
||||||
|
against: Against,
|
||||||
|
): Verdict {
|
||||||
|
if (!warm) return { use: "raise", why: "nothing is being kept warm" };
|
||||||
|
if (warm.scenario !== scenario) {
|
||||||
|
return { use: "raise", why: `what is kept warm is ${warm.scenario}, and this is ${scenario}` };
|
||||||
|
}
|
||||||
|
if (!standing.includes(warm.instanceId)) {
|
||||||
|
return { use: "raise", why: `${warm.instanceId} is no longer standing` };
|
||||||
|
}
|
||||||
|
if (!hasSnapshot) {
|
||||||
|
return { use: "raise", why: `${warm.instanceId} has no ${label} snapshot to return to` };
|
||||||
|
}
|
||||||
|
// The check that keeps this honest. A mesh built from code that has since moved would pass
|
||||||
|
// today's tests against yesterday's binaries, and say nothing about it.
|
||||||
|
//
|
||||||
|
// **Both directions, because comparing only what is in front of you clears what is not.** The
|
||||||
|
// first version walked the current repositories alone, so running without the environment that
|
||||||
|
// names where they are compared nothing and reported the mesh usable — a warm instance built
|
||||||
|
// from code that had since moved, cleared by a check that had looked at neither. That is
|
||||||
|
// 04-ISSUES/005's rule again: a record that says nothing about something is not a record that
|
||||||
|
// clears it.
|
||||||
|
for (const name of new Set([...Object.keys(warm.against), ...Object.keys(against)])) {
|
||||||
|
const then = warm.against[name];
|
||||||
|
const now = against[name];
|
||||||
|
if (then === now) continue;
|
||||||
|
if (!now) {
|
||||||
|
return {
|
||||||
|
use: "raise",
|
||||||
|
why: `${name} was at ${then} when this was warmed, and nothing says where it is now — ` +
|
||||||
|
`so nothing can say whether it moved`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
use: "raise",
|
||||||
|
why: `${name} was at ${then ?? "nothing recorded"} when this was warmed, ` +
|
||||||
|
`and is now at ${now}`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return { use: "restore", instanceId: warm.instanceId };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What is standing right now, by instance. */
|
||||||
|
export async function standingNow(): Promise<string[]> {
|
||||||
|
return (await list()).map((i) => i.instanceId);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* ready returns an instance already at its warm state, or says why one must be raised.
|
||||||
|
*
|
||||||
|
* It never raises: raising needs a scenario, images and a bootstrap, and all of that belongs to
|
||||||
|
* whoever is using this rather than here.
|
||||||
|
*/
|
||||||
|
export async function ready(
|
||||||
|
scenario: string,
|
||||||
|
env: NodeJS.ProcessEnv = process.env,
|
||||||
|
): Promise<Verdict> {
|
||||||
|
const warm = remembered();
|
||||||
|
const standing = await standingNow();
|
||||||
|
const has = warm ? (await snapshots(warm.instanceId)).includes(label) : false;
|
||||||
|
return judge(warm, scenario, standing, has, whatWasTested(env));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** returnTo puts a warm instance back to its state, and says how long it took. */
|
||||||
|
export async function returnTo(
|
||||||
|
instanceId: string,
|
||||||
|
log: (message: string) => void = () => {},
|
||||||
|
): Promise<number> {
|
||||||
|
const { usableSeconds } = await restore(instanceId, label, 180, log);
|
||||||
|
return usableSeconds;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* keep snapshots an instance as the state to come back to, and records what it was built from.
|
||||||
|
*
|
||||||
|
* Called once the caller has brought the scenario to whatever "ready to work" means for it.
|
||||||
|
*/
|
||||||
|
export async function keep(
|
||||||
|
scenario: string,
|
||||||
|
instanceId: string,
|
||||||
|
env: NodeJS.ProcessEnv = process.env,
|
||||||
|
): Promise<Warm> {
|
||||||
|
await snapshot(instanceId, label);
|
||||||
|
const warm: Warm = {
|
||||||
|
scenario,
|
||||||
|
instanceId,
|
||||||
|
images: stockOf(instanceId),
|
||||||
|
against: whatWasTested(env),
|
||||||
|
at: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
remember(warm);
|
||||||
|
return warm;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** cool destroys what is being kept and forgets it. */
|
||||||
|
export async function cool(): Promise<string | null> {
|
||||||
|
const warm = remembered();
|
||||||
|
forget();
|
||||||
|
if (!warm) return null;
|
||||||
|
if ((await standingNow()).includes(warm.instanceId)) {
|
||||||
|
await destroy(warm.instanceId);
|
||||||
|
}
|
||||||
|
return warm.instanceId;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What a raised scenario stocked, held until it is kept.
|
||||||
|
*
|
||||||
|
* Raising works the images out and snapshotting happens later, so this carries them between the
|
||||||
|
* two without the caller having to hold them.
|
||||||
|
*/
|
||||||
|
const stock = new Map<string, string[]>();
|
||||||
|
|
||||||
|
export function rememberStock(instanceId: string, images: string[]): void {
|
||||||
|
stock.set(instanceId, images);
|
||||||
|
}
|
||||||
|
|
||||||
|
function stockOf(instanceId: string): string[] {
|
||||||
|
return stock.get(instanceId) ?? [];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** What a restored instance's registry serves, from when it was warmed. */
|
||||||
|
export function warmStock(instanceId: string): { images: string[] } {
|
||||||
|
const warm = remembered();
|
||||||
|
if (!warm || warm.instanceId !== instanceId) return { images: [] };
|
||||||
|
return { images: warm.images };
|
||||||
|
}
|
||||||
@@ -26,6 +26,10 @@ import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts";
|
|||||||
import { labIsUsable, destroyAll } from "./harness.ts";
|
import { labIsUsable, destroyAll } from "./harness.ts";
|
||||||
import { incus } from "../../src/incus/client.ts";
|
import { incus } from "../../src/incus/client.ts";
|
||||||
import { machineName } from "../../src/lifecycle/names.ts";
|
import { machineName } from "../../src/lifecycle/names.ts";
|
||||||
|
import { ready, returnTo, keep, rememberStock, warmStock } from "../../src/warm.ts";
|
||||||
|
|
||||||
|
/** Whether this run keeps its mesh for the next one. Off unless asked for. */
|
||||||
|
const warming = process.env["MESH_LAB_WARM"] === "1";
|
||||||
|
|
||||||
const capability = await labIsUsable();
|
const capability = await labIsUsable();
|
||||||
const binary = hostBinaryPath();
|
const binary = hostBinaryPath();
|
||||||
@@ -128,6 +132,44 @@ function tokenFrom(said: string): string {
|
|||||||
|
|
||||||
before(async () => {
|
before(async () => {
|
||||||
if (skip) return;
|
if (skip) return;
|
||||||
|
|
||||||
|
// A mesh kept between runs, when one is being kept and still counts.
|
||||||
|
//
|
||||||
|
// **Bootstrapping proves the same thing every time**, and the tests worth iterating on are the
|
||||||
|
// ones after it. Off by default: a run that is meant to mean something raises from nothing,
|
||||||
|
// because "it passes" must not come to mean "it passes against a mesh somebody bootstrapped
|
||||||
|
// last week".
|
||||||
|
if (warming) {
|
||||||
|
const said = await ready(SCENARIO);
|
||||||
|
if (said.use === "restore") {
|
||||||
|
instanceId = said.instanceId;
|
||||||
|
const seconds = await returnTo(instanceId);
|
||||||
|
stocked = warmStock(instanceId).images;
|
||||||
|
|
||||||
|
// **A snapshot captures disk, not memory.** Restoring reboots the machine, so everything
|
||||||
|
// this suite started by hand is gone — the host most of all. Without it the mesh looks
|
||||||
|
// perfectly healthy from the control plane's side: a module is assigned, a declaration is
|
||||||
|
// sent and recorded, and nothing on the machine is listening to apply it. That is exactly
|
||||||
|
// how this was first met, and it cost an hour to see.
|
||||||
|
//
|
||||||
|
// The real answer is a host started by init, which is what the design says it is anyway
|
||||||
|
// (novox/hq 05-the-node-host: a root service, installed as a package). Until the lab places
|
||||||
|
// it that way, the warm path restarts what it knows it started.
|
||||||
|
for (const machine of ["anchor", "laptop"]) {
|
||||||
|
await must(machine, `pgrep -x mesh-host >/dev/null || ` +
|
||||||
|
`(nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3)`);
|
||||||
|
}
|
||||||
|
const running = await on("anchor", `pgrep -x mesh-host >/dev/null && echo yes || echo no`);
|
||||||
|
assert.equal(running.out.trim(), "yes",
|
||||||
|
"the host did not come back after a restore, so nothing would apply anything");
|
||||||
|
|
||||||
|
console.log(`warm: returned ${instanceId} to its state in ${seconds.toFixed(1)}s, ` +
|
||||||
|
`and started the host again`);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
console.log(`warm: raising fresh — ${said.why}`);
|
||||||
|
}
|
||||||
|
|
||||||
const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), {});
|
const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), {});
|
||||||
instanceId = raised.instanceId;
|
instanceId = raised.instanceId;
|
||||||
|
|
||||||
@@ -153,9 +195,20 @@ before(async () => {
|
|||||||
`MESH_WORKSPACE=/var/lib/mesh-builder ` +
|
`MESH_WORKSPACE=/var/lib/mesh-builder ` +
|
||||||
`nohup /usr/local/bin/mesh-builder > /var/log/mesh-builder.log 2>&1 & sleep 3`);
|
`nohup /usr/local/bin/mesh-builder > /var/log/mesh-builder.log 2>&1 & sleep 3`);
|
||||||
}
|
}
|
||||||
|
if (warming) {
|
||||||
|
// Snapshotted only now, with everything up: a state worth returning to is the one after the
|
||||||
|
// part nobody wants to repeat.
|
||||||
|
await rememberStock(instanceId, stocked);
|
||||||
|
const warm = await keep(SCENARIO, instanceId);
|
||||||
|
console.log(`warm: ${warm.instanceId} kept, against ` +
|
||||||
|
Object.entries(warm.against).map(([n, c]) => `${n} ${c}`).join(", "));
|
||||||
|
}
|
||||||
}, { timeout: 1_800_000 });
|
}, { timeout: 1_800_000 });
|
||||||
|
|
||||||
after(async () => {
|
after(async () => {
|
||||||
|
// A kept instance survives on purpose, and `mesh-lab warm cool` is how it goes away. Everything
|
||||||
|
// else is destroyed, because an instance nobody meant to keep is one nobody will remember.
|
||||||
|
if (warming) return;
|
||||||
if (instanceId) await destroy(instanceId);
|
if (instanceId) await destroy(instanceId);
|
||||||
await destroyAll(`${SCENARIO}-`);
|
await destroyAll(`${SCENARIO}-`);
|
||||||
}, { timeout: 600_000 });
|
}, { timeout: 600_000 });
|
||||||
@@ -774,16 +827,16 @@ test("rotating a credential moves both ends, and the old one stops working", {
|
|||||||
// holding a matching string proves they agree; only an authentication proves they are right.
|
// holding a matching string proves they agree; only an authentication proves they are right.
|
||||||
const store = "/var/lib/mesh/postgres";
|
const store = "/var/lib/mesh/postgres";
|
||||||
await must("anchor", `printf %s '{"module":"realstore","version":"1",` +
|
await must("anchor", `printf %s '{"module":"realstore","version":"1",` +
|
||||||
`"provides":[{"name":"realpostgres-database","scope":"mesh"}],` +
|
`"provides":[{"name":"real-postgres-database","scope":"mesh"}],` +
|
||||||
`"capabilities":["container-runtime"],` +
|
`"capabilities":["container-runtime"],` +
|
||||||
`"serves":{"realpostgres-database":{"port":5433}},` +
|
`"serves":{"real-postgres-database":{"port":5433}},` +
|
||||||
`"own-secrets":{"superuser":"${store}/superuser"},` +
|
`"own-secrets":{"superuser":"${store}/superuser"},` +
|
||||||
`"grants":{"realpostgres-database":"${store}/grants"},` +
|
`"grants":{"real-postgres-database":"${store}/grants"},` +
|
||||||
// Both halves. `grants` is where each consumer's sealed password lands; `receives` is the
|
// Both halves. `grants` is where each consumer's sealed password lands; `receives` is the
|
||||||
// manifest saying who asked and for what. Without the second the provisioner finds a
|
// manifest saying who asked and for what. Without the second the provisioner finds a
|
||||||
// directory of unexplained secrets and says nothing has been granted — which is true, and
|
// directory of unexplained secrets and says nothing has been granted — which is true, and
|
||||||
// reads exactly like a credential that was never delivered.
|
// reads exactly like a credential that was never delivered.
|
||||||
`"receives":{"realpostgres-database":"${store}/grants/mesh.json"},` +
|
`"receives":{"real-postgres-database":"${store}/grants/mesh.json"},` +
|
||||||
`"listens":[{"port":5433,"from":"mesh","why":"a database the mesh provisions"}],` +
|
`"listens":[{"port":5433,"from":"mesh","why":"a database the mesh provisions"}],` +
|
||||||
`"resources":[` +
|
`"resources":[` +
|
||||||
`{"id":"state","type":"directory","path":"${store}","mode":"0755"},` +
|
`{"id":"state","type":"directory","path":"${store}","mode":"0755"},` +
|
||||||
@@ -801,9 +854,9 @@ test("rotating a credential moves both ends, and the old one stops working", {
|
|||||||
`"MESH_PROVISION_POSTGRES":"postgres://postgres@127.0.0.1:5433/postgres?sslmode=disable"}}]}' ` +
|
`"MESH_PROVISION_POSTGRES":"postgres://postgres@127.0.0.1:5433/postgres?sslmode=disable"}}]}' ` +
|
||||||
`> /tmp/realstore.json`);
|
`> /tmp/realstore.json`);
|
||||||
await must("anchor", `printf %s '{"module":"realapp","version":"1",` +
|
await must("anchor", `printf %s '{"module":"realapp","version":"1",` +
|
||||||
`"requires":["realpostgres-database"],"contributes":{"realpostgres-database":{"name":"realapp"}},` +
|
`"requires":["real-postgres-database"],"contributes":{"real-postgres-database":{"name":"realapp"}},` +
|
||||||
`"binds":{"realpostgres-database":"/etc/realapp/where.json"},` +
|
`"binds":{"real-postgres-database":"/etc/realapp/where.json"},` +
|
||||||
`"secrets":{"realpostgres-database":"/etc/realapp/password"},` +
|
`"secrets":{"real-postgres-database":"/etc/realapp/password"},` +
|
||||||
`"resources":[{"id":"dir","type":"directory","path":"/etc/realapp","mode":"0755"}]}' ` +
|
`"resources":[{"id":"dir","type":"directory","path":"/etc/realapp","mode":"0755"}]}' ` +
|
||||||
`> /tmp/realapp.json`);
|
`> /tmp/realapp.json`);
|
||||||
for (const f of ["realstore", "realapp"]) {
|
for (const f of ["realstore", "realapp"]) {
|
||||||
@@ -856,7 +909,7 @@ test("rotating a credential moves both ends, and the old one stops working", {
|
|||||||
|
|
||||||
// Now rotate. One command: the record changes AND both ends are sent, because leaving the
|
// Now rotate. One command: the record changes AND both ends are sent, because leaving the
|
||||||
// sending to a later command is the fault above, exactly.
|
// sending to a later command is the fault above, exactly.
|
||||||
const said = await mesh("rotate realdatabase", 180_000);
|
const said = await mesh("rotate real-postgres-database", 180_000);
|
||||||
assert.match(said, /anchor/, `rotation did not touch the provider:\n${said}`);
|
assert.match(said, /anchor/, `rotation did not touch the provider:\n${said}`);
|
||||||
assert.match(said, /laptop/, `rotation did not touch the consumer:\n${said}`);
|
assert.match(said, /laptop/, `rotation did not touch the consumer:\n${said}`);
|
||||||
await new Promise((r) => setTimeout(r, 25_000));
|
await new Promise((r) => setTimeout(r, 25_000));
|
||||||
@@ -1486,3 +1539,122 @@ test("a service is reached by a name under the machine it runs on", {
|
|||||||
}
|
}
|
||||||
await mesh("push");
|
await mesh("push");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// A real third-party workload, adopted the way the conversion will adopt one.
|
||||||
|
//
|
||||||
|
// **Everything before this used modules written to exercise the mesh.** This one is software
|
||||||
|
// nobody here wrote, taking its credentials the way such software does — from its environment —
|
||||||
|
// and needing two containers that reach each other by name. It is the first module that could not
|
||||||
|
// have been declared before today: it needs the `network` shape, and it needs a sealed value to
|
||||||
|
// reach a container's environment.
|
||||||
|
//
|
||||||
|
// Its database password is **accepted rather than generated**, which is the whole shape of an
|
||||||
|
// adoption: a service that already exists keeps the credential it already has, because minting a
|
||||||
|
// new one is how a running application stops being able to reach its own database.
|
||||||
|
test("a third-party workload is adopted, with the credential it already had", {
|
||||||
|
skip, timeout: 900_000,
|
||||||
|
}, async () => {
|
||||||
|
const password = "the-password-it-already-had";
|
||||||
|
|
||||||
|
await must("anchor", `printf %s ${quote(JSON.stringify({
|
||||||
|
module: "umami",
|
||||||
|
version: "1",
|
||||||
|
capabilities: ["container-runtime"],
|
||||||
|
"own-secrets": {
|
||||||
|
database: "/var/lib/umami/database.env",
|
||||||
|
app: "/var/lib/umami/app.env",
|
||||||
|
},
|
||||||
|
listens: [{ port: 1212, protocol: "tcp", from: "mesh", why: "the analytics page" }],
|
||||||
|
resources: [
|
||||||
|
{ id: "state", type: "directory", path: "/var/lib/umami", mode: "0700" },
|
||||||
|
// The two containers must reach each other by name, which is what this shape is for.
|
||||||
|
{ id: "net", type: "network", name: "umami" },
|
||||||
|
{
|
||||||
|
id: "db", type: "container", name: "umami-db",
|
||||||
|
image: pinned("postgres"),
|
||||||
|
network: "umami",
|
||||||
|
env: { POSTGRES_DB: "umami", POSTGRES_USER: "umami" },
|
||||||
|
"env-file": ["/var/lib/umami/database.env"],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: "app", type: "container", name: "umami",
|
||||||
|
image: pinned("ghcr.io/umami-software/umami"),
|
||||||
|
network: "umami",
|
||||||
|
env: { DATABASE_TYPE: "postgresql" },
|
||||||
|
"env-file": ["/var/lib/umami/app.env"],
|
||||||
|
ports: ["1212:3000"],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}))} > /umami.json`);
|
||||||
|
await must("anchor", `docker cp /umami.json mesh-control:/umami.json`);
|
||||||
|
await mesh("module add /umami.json");
|
||||||
|
|
||||||
|
// **Accepted, not generated.** The value is what the database already answers to; the mesh
|
||||||
|
// seals it and cannot read it again. Given whole, as the environment lines the containers read.
|
||||||
|
await must("anchor",
|
||||||
|
`printf %s ${quote(`POSTGRES_PASSWORD=${password}`)} | ` +
|
||||||
|
`docker exec -i mesh-control /mesh-control secret accept anchor umami database --from -`);
|
||||||
|
await must("anchor",
|
||||||
|
`printf %s ${quote(
|
||||||
|
`DATABASE_URL=postgresql://umami:${password}@umami-db:5432/umami`)} | ` +
|
||||||
|
`docker exec -i mesh-control /mesh-control secret accept anchor umami app --from -`);
|
||||||
|
|
||||||
|
await mesh("assign anchor umami");
|
||||||
|
await mesh("push anchor", 300_000);
|
||||||
|
|
||||||
|
// Both containers, and the network they share.
|
||||||
|
let up = false;
|
||||||
|
for (let i = 0; i < 60 && !up; i++) {
|
||||||
|
const running = await on("anchor", `docker ps --format '{{.Names}}'`);
|
||||||
|
up = running.out.includes("umami-db") && running.out.includes("umami");
|
||||||
|
if (!up) await new Promise((r) => setTimeout(r, 5000));
|
||||||
|
}
|
||||||
|
if (!up) {
|
||||||
|
// Everything that could say why, gathered before asserting. "It did not start" is the one
|
||||||
|
// thing already known; what is wanted is whether the mesh sent it, whether the host refused
|
||||||
|
// it, and what the runtime said when it tried.
|
||||||
|
const said = await mesh("status");
|
||||||
|
const containers = await on("anchor", `docker ps -a --format '{{.Names}} {{.Status}}'`);
|
||||||
|
const applied = await on("anchor",
|
||||||
|
`${HOST_PATH} owned 2>&1 | head -30 || echo "the host could not say what it owns"`);
|
||||||
|
const files = await on("anchor", `ls -la /var/lib/umami/ 2>&1; ` +
|
||||||
|
`for f in /var/lib/umami/*.env; do echo "-- $f"; wc -c "$f"; done 2>&1`);
|
||||||
|
const tried = await on("anchor",
|
||||||
|
`docker inspect umami-db --format '{{.State.Status}} {{.State.Error}}' 2>&1; ` +
|
||||||
|
`docker logs umami-db 2>&1 | tail -15`);
|
||||||
|
assert.fail(
|
||||||
|
`the workload never started.\n\n` +
|
||||||
|
`── what the mesh thinks:\n${said}\n` +
|
||||||
|
`── containers:\n${containers.out}\n` +
|
||||||
|
`── what the host owns:\n${applied.out}\n` +
|
||||||
|
`── what the mesh wrote:\n${files.out}\n` +
|
||||||
|
`── the database container:\n${tried.out}\n`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The environment file the mesh sealed is on the machine and readable only by root.
|
||||||
|
const mode = await must("anchor", `stat -c %a /var/lib/umami/database.env`);
|
||||||
|
assert.equal(mode.trim(), "600", "a file holding a credential is readable by more than root");
|
||||||
|
|
||||||
|
// **The assertion that matters: the credential works.** Not that a file arrived — that the
|
||||||
|
// database the mesh started answers to the password the mesh was given rather than one it made.
|
||||||
|
let connected = { out: "", ok: false };
|
||||||
|
for (let i = 0; i < 40 && !connected.ok; i++) {
|
||||||
|
connected = await on("anchor",
|
||||||
|
`docker exec umami-db psql -U umami -d umami -qAt -c 'select 1'`);
|
||||||
|
if (!connected.ok) await new Promise((r) => setTimeout(r, 3000));
|
||||||
|
}
|
||||||
|
assert.ok(connected.ok, `the database never came up:\n${connected.out}`);
|
||||||
|
|
||||||
|
const wrong = await on("anchor",
|
||||||
|
`docker run --rm --network umami -e PGPASSWORD=not-the-password ${pinned("postgres")} ` +
|
||||||
|
`psql -h umami-db -U umami -d umami -qAt -c 'select 1'`);
|
||||||
|
assert.ok(!wrong.ok,
|
||||||
|
"the database accepted a password nobody gave it, so this proves nothing about the one that was");
|
||||||
|
|
||||||
|
// And the two containers reach each other by name over the module's own network.
|
||||||
|
const reached = await must("anchor",
|
||||||
|
`docker run --rm --network umami ${pinned("postgres")} ` +
|
||||||
|
`sh -c 'getent hosts umami-db || echo unreachable'`);
|
||||||
|
assert.doesNotMatch(reached, /unreachable/,
|
||||||
|
"a container could not reach the other by name, so the module's network did nothing");
|
||||||
|
});
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
import { test } from "node:test";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { judge, type Warm } from "../src/warm.ts";
|
||||||
|
|
||||||
|
const at = "2026-08-31T20:00:00Z";
|
||||||
|
const built = { "mesh-lab": "aaa", "mesh-host": "bbb", "mesh-control": "ccc" };
|
||||||
|
const warm = (over: Partial<Warm> = {}): Warm =>
|
||||||
|
({ scenario: "two-nodes", instanceId: "mlab-two-nodes-1", images: [], against: built, at, ...over });
|
||||||
|
|
||||||
|
// The check this exists for: a mesh warmed against code that has since moved would pass today's
|
||||||
|
// tests against yesterday's binaries, and the result would say nothing about it.
|
||||||
|
//
|
||||||
|
// Same fault as novox/hq 04-ISSUES/005, one level down — a green result standing for a run
|
||||||
|
// against something other than what is in front of you.
|
||||||
|
test("a warm mesh built from code that has moved is refused", () => {
|
||||||
|
const said = judge(warm(), "two-nodes", ["mlab-two-nodes-1"], true,
|
||||||
|
{ ...built, "mesh-host": "moved" });
|
||||||
|
assert.equal(said.use, "raise");
|
||||||
|
assert.match(said.use === "raise" ? said.why : "", /mesh-host was at bbb.*now at moved/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a warm mesh built from the same code is used", () => {
|
||||||
|
const said = judge(warm(), "two-nodes", ["mlab-two-nodes-1"], true, built);
|
||||||
|
assert.equal(said.use, "restore");
|
||||||
|
});
|
||||||
|
|
||||||
|
// Each refusal is named, because each is a different thing being wrong.
|
||||||
|
test("every reason to raise instead says which reason it was", () => {
|
||||||
|
const cases: [string, ReturnType<typeof judge>][] = [
|
||||||
|
["nothing kept", judge(null, "two-nodes", [], true, built)],
|
||||||
|
["another scenario", judge(warm({ scenario: "first-node" }), "two-nodes",
|
||||||
|
["mlab-two-nodes-1"], true, built)],
|
||||||
|
["not standing", judge(warm(), "two-nodes", [], true, built)],
|
||||||
|
["no snapshot", judge(warm(), "two-nodes", ["mlab-two-nodes-1"], false, built)],
|
||||||
|
];
|
||||||
|
for (const [what, said] of cases) {
|
||||||
|
assert.equal(said.use, "raise", what);
|
||||||
|
assert.ok(said.use === "raise" && said.why.length > 10,
|
||||||
|
`${what} was refused without saying why: ${JSON.stringify(said)}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// A repository the warm record never accounted for is a difference, not a match.
|
||||||
|
test("a repository that was not recorded when it was warmed is refused", () => {
|
||||||
|
const said = judge(warm({ against: { "mesh-lab": "aaa" } }), "two-nodes",
|
||||||
|
["mlab-two-nodes-1"], true, built);
|
||||||
|
assert.equal(said.use, "raise");
|
||||||
|
});
|
||||||
|
|
||||||
|
// A repository the current run cannot see is not a repository that agrees.
|
||||||
|
//
|
||||||
|
// **Found by testing the guard rather than trusting it.** The first version walked only the
|
||||||
|
// repositories the current environment names, so running without that environment compared
|
||||||
|
// nothing and called a stale mesh usable. The mesh had genuinely moved; the check had looked at
|
||||||
|
// neither side.
|
||||||
|
test("a repository this run cannot locate is refused, not passed over", () => {
|
||||||
|
const said = judge(warm(), "two-nodes", ["mlab-two-nodes-1"], true, { "mesh-lab": "aaa" });
|
||||||
|
assert.equal(said.use, "raise");
|
||||||
|
assert.match(said.use === "raise" ? said.why : "",
|
||||||
|
/nothing says where it is now|so nothing can say whether it moved/);
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user