From e89379fef8fba1547b9bdf3e50aca244ee73b085 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 30 Aug 2026 01:29:46 +0200 Subject: [PATCH] Prove a mesh credential becomes a login, against a real database MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mesh generates a password, seals it to the machine that must accept it, and discards the plaintext — so it cannot tell PostgreSQL to start accepting it. Something on that machine reads what the host wrote and makes it true. Everything up to that step is proven elsewhere; this is where a password either becomes a login or does not. A scenario with one machine and a database, and six assertions: the password works, running again reaches the same state and says nothing, rotation makes the new one work and the old one stop, a departed consumer loses its login, a role nobody here made is left alone, and a manifest naming a credential that was never written is refused rather than creating a login with no password. Each was confirmed to fail — and only it to fail — with the behaviour removed from the provisioner: only-creates breaks rotation, no-revoke breaks revocation, revoking everything breaks the bystander role, and ignoring a missing credential breaks the refusal. Two faults in the test itself, both worth recording: - it checked logins from inside the database's own container over 127.0.0.1, which PostgreSQL's default pg_hba trusts. No password was ever verified. Demonstrated directly: over loopback a deliberately wrong password still returns a row. Only the rotation assertion noticed, because it is the one that requires a password to STOP working — which is an argument for writing that assertion every time. - the fix then read .NetworkSettings.IPAddress, which docker 29 no longer populates. It templates to empty, psql falls back to a unix socket that is not there, and every login looks impossible rather than misconfigured. --- provisioners/README.md | 47 +++++ scenarios/a-provider.yml | 22 +++ test/integration/provisioner.test.ts | 258 +++++++++++++++++++++++++++ 3 files changed, 327 insertions(+) create mode 100644 provisioners/README.md create mode 100644 scenarios/a-provider.yml create mode 100644 test/integration/provisioner.test.ts diff --git a/provisioners/README.md b/provisioners/README.md new file mode 100644 index 0000000..504cefa --- /dev/null +++ b/provisioners/README.md @@ -0,0 +1,47 @@ +# Provisioners + +The half that makes a credential real. + +The mesh generates a password, seals it to the machine that must accept it, and never holds the +value — so it cannot tell PostgreSQL, or MinIO, or a broker, to start accepting it. Something on +that machine reads what arrived and makes it true. That something is a provisioner, and it belongs +to the module that ships the software, not to the mesh. + +**What the mesh owns is the contract.** A provider module declares: + +```json +{ + "provides": [{"name": "database", "scope": "mesh"}], + "receives": {"database": "/var/lib/postgres/grants/mesh.json"}, + "grants": {"database": "/var/lib/postgres/grants"} +} +``` + +and is then given, by the host, from an ordinary declaration: + +| | | +|---|---| +| `mesh.json` | every consumer, what it asked for, and where its credential is | +| `.secret` | one consumer's password, alone in the file, sealed in transit and written in plain by the host | + +Two files rather than one because the mesh discarded the plaintext and cannot compose a document +containing it. The consequence is a good one: the readable half stays readable, and the secret +half changes only when the secret does. + +**A provisioner reconciles; it is not told what changed.** It runs after every declaration and +must reach the same state from wherever it starts. That means, in order: + +1. every consumer in the manifest has what it asked for, with the password it was given — set + every time, not only on creation, or a rotation reports success and changes nothing +2. **everything this provisioner made that is no longer asked for is removed.** A consumer that + goes away otherwise leaves a working login behind for ever, and nothing says so + +Step 2 is the half usually missing, and it is the same rule the host follows about removing what +it declared and no longer declares. + +The reference implementation lives in `mesh-control/examples/postgres-provisioner`, because that +is where the contract is defined and where the language is already set up to read it. The lab's +job is the other half: raising a real PostgreSQL and proving that what the mesh delivered becomes +a login that works, a rotation that takes effect, and a revocation that bites. + +Set `MESH_LAB_PROVISIONER` to a built one to run those. diff --git a/scenarios/a-provider.yml b/scenarios/a-provider.yml new file mode 100644 index 0000000..f221c17 --- /dev/null +++ b/scenarios/a-provider.yml @@ -0,0 +1,22 @@ +# One machine running a database that other machines use. +# +# It exists to prove the last step of a credential: the mesh generated a password, sealed it to +# this machine, and cannot tell PostgreSQL to accept it. Something here has to, and this is where +# that something is run against a real database rather than described. +scenario: a-provider + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + +images: + - postgres:17-alpine + +place: + all: [runtime] diff --git a/test/integration/provisioner.test.ts b/test/integration/provisioner.test.ts new file mode 100644 index 0000000..0ffa6c3 --- /dev/null +++ b/test/integration/provisioner.test.ts @@ -0,0 +1,258 @@ +/** + * The last step of a credential, against a real database. + * + * The mesh generates a password, seals it to the machine that must accept it, and discards the + * plaintext — so it cannot tell PostgreSQL to start accepting it. Something on that machine reads + * what the host wrote and makes it true. Everything up to that point is proven elsewhere; this is + * the step where a password either becomes a login or does not. + * + * Against a real PostgreSQL because there is no version of this worth asserting against a fake: + * what is under test is whether `create role ... password` and a connection agree, which is + * exactly what a fake would be told to agree about (novox/hq ADR 0017). + */ + +import { test, after, before } from "node:test"; +import assert from "node:assert/strict"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; +import { incus } from "../../src/incus/client.ts"; +import { machineName } from "../../src/lifecycle/names.ts"; + +const capability = await labIsUsable(); +const provisioner = process.env["MESH_LAB_PROVISIONER"] ?? ""; +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !provisioner + ? "set MESH_LAB_PROVISIONER to a built provisioner (mesh-control: go build ./examples/postgres-provisioner)" + : false; + +const SCENARIO = "a-provider"; +const MACHINE = "anchor"; +const GRANTS = "/var/lib/postgres/grants"; +const SUPER = "postgres://postgres:super@127.0.0.1:5432/postgres?sslmode=disable"; + +let instanceId = ""; +/** The postgres image, by digest, from the registry the scenario raised. */ +let image = ""; + +function shellQuote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +/** Run something on the machine and return what it said, with its exit status. */ +async function on(command: string): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `${command} 2>&1; echo "__exit=$?"`, + ]); + const marker = stdout.lastIndexOf("__exit="); + const status = Number(stdout.slice(marker + 7).trim()); + return { out: stdout.slice(0, marker), ok: status === 0 }; +} + +/** The same, refusing to continue past a failure nobody would otherwise see. */ +async function must(command: string): Promise { + const { out, ok } = await on(command); + if (!ok) throw new Error(`${command}\n${out}`); + return out; +} + +/** psql as the superuser, inside the database container. */ +async function sql(query: string): Promise { + return (await must(`docker exec mesh-db psql -U postgres -qAt -c ${shellQuote(query)}`)).trim(); +} + +/** + * Write what the host would have written from a declaration: the manifest of who asked, and one + * file per consumer holding its password alone. + * + * Written here rather than by running the host, because what is under test is the step *after* + * the host — and that the host writes these exact shapes is asserted in its own suite. + */ +async function meshWrote( + consumers: { node: string; module: string; name: string; password: string }[], +): Promise { + const manifest = { + contributions: 1, + requirement: "database", + generated: "by the mesh", + given: consumers.map((c) => ({ + from: c.module, + node: c.node, + secret: `${GRANTS}/${c.node}.secret`, + values: { name: c.name }, + })), + }; + await must(`mkdir -p ${GRANTS}`); + await must(`printf %s ${shellQuote(JSON.stringify(manifest))} > ${GRANTS}/mesh.json`); + // Every credential file rewritten from nothing, so a removed consumer's does not linger and + // make the revocation test pass for a reason that is not the one being tested. + await must(`find ${GRANTS} -name '*.secret' -delete`); + for (const c of consumers) { + await must(`printf %s ${shellQuote(c.password)} > ${GRANTS}/${c.node}.secret`); + await must(`chmod 600 ${GRANTS}/${c.node}.secret`); + } +} + +/** The provisioner, as the module shipping PostgreSQL would run it. */ +async function provision(): Promise<{ out: string; ok: boolean }> { + return on( + `GRANTS=${GRANTS} MESH_PROVISION_POSTGRES=${shellQuote(SUPER)} /usr/local/bin/mesh-provision-postgres`, + ); +} + +/** + * Can this role log in with this password? + * + * Over the bridge, from a container of its own. `--network container:mesh-db` would share the + * database's namespace and put us back on its loopback, which is the very thing being avoided. + * + * The address comes from `.NetworkSettings.Networks.bridge.IPAddress` rather than the top-level + * `.NetworkSettings.IPAddress`, which docker 29 no longer populates — it templates to empty, psql + * silently falls back to a unix socket that is not there, and every login looks impossible. + * + * From a separate container, reaching the database over the bridge — **not** from inside it over + * loopback. PostgreSQL's default `pg_hba.conf` trusts `127.0.0.1`, so a check made from inside + * the container authenticates nothing and returns true for any password at all. Which is what the + * first version of this did: two tests passed without ever verifying a password, and only the + * rotation test noticed, by asserting that an old password had *stopped* working. + */ +async function canLogIn(role: string, password: string, database: string): Promise { + return (await tryLogIn(role, password, database)).ok; +} + +/** The same, keeping what the database said — so a failure says why rather than only that. */ +async function tryLogIn( + role: string, + password: string, + database: string, +): Promise<{ ok: boolean; out: string }> { + const { out } = await on( + `docker run --rm -e PGPASSWORD=${shellQuote(password)} ${image} ` + + `psql -h "$(docker inspect -f '{{.NetworkSettings.Networks.bridge.IPAddress}}' mesh-db)" ` + + `-U ${role} -d ${database} -qAt -c 'select 1'`, + ); + return { ok: out.trim() === "1", out }; +} + +before(async () => { + if (skip) return; + const scenario = loadScenario(`scenarios/${SCENARIO}.yml`); + const instance = await raise(scenario, {}); + instanceId = instance.instanceId; + + // From the registry the scenario raised, by digest. There is no route to a public registry from + // a documentation range, which is the point of the lab having its own. + const stocked = instance.images.find((r) => r.includes("postgres")); + assert.ok(stocked, `the scenario stocked no postgres image: ${instance.images.join(", ")}`); + image = stocked; + + await must( + `docker run -d --name mesh-db -e POSTGRES_PASSWORD=super ` + + `-p 127.0.0.1:5432:5432 ${image}`, + ); + let ready = false; + for (let i = 0; i < 90 && !ready; i++) { + ({ ok: ready } = await on(`docker exec mesh-db pg_isready -U postgres`)); + if (!ready) await new Promise((r) => setTimeout(r, 1000)); + } + assert.ok(ready, "the database never became ready"); + + await incus([ + "file", "push", provisioner, + `${machineName(instanceId, MACHINE)}/usr/local/bin/mesh-provision-postgres`, + "--mode", "0755", + ], 180_000); +}, { timeout: 1_200_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("a password the mesh generated becomes a login that works", { skip, timeout: 300_000 }, async () => { + await meshWrote([ + { node: "workstation", module: "meshboard", name: "meshboard", password: "first-password-aaa" }, + ]); + const { out, ok } = await provision(); + assert.ok(ok, out); + + assert.equal(await sql(`select rolcanlogin from pg_roles where rolname = 'mesh_workstation'`), "t"); + assert.equal(await sql(`select 1 from pg_database where datname = 'meshboard'`), "1"); + const attempt = await tryLogIn("mesh_workstation", "first-password-aaa", "meshboard"); + assert.ok(attempt.ok, `the consumer cannot log in with the password the mesh gave it:\n${attempt.out}`); +}); + +test("running it again reaches the same state and says nothing", { skip, timeout: 300_000 }, async () => { + // It runs after every declaration and is never told what changed, so arriving at an already + // correct state is the ordinary case rather than an edge one. + const { out, ok } = await provision(); + assert.ok(ok, out); + assert.equal(out.trim(), "", `it did work on a second run: ${out}`); + assert.ok(await canLogIn("mesh_workstation", "first-password-aaa", "meshboard")); +}); + +test("rotating the password makes the new one work and the old one stop", { skip, timeout: 300_000 }, async () => { + // The failure this guards is a provisioner that only ever creates: the mesh replaces the file, + // the role exists, nothing happens, and a rotation reports success while changing nothing. + await meshWrote([ + { node: "workstation", module: "meshboard", name: "meshboard", password: "second-password-bbb" }, + ]); + const { out, ok } = await provision(); + assert.ok(ok, out); + + assert.ok( + await canLogIn("mesh_workstation", "second-password-bbb", "meshboard"), + "the rotated password does not work", + ); + assert.equal( + await canLogIn("mesh_workstation", "first-password-aaa", "meshboard"), + false, + "the old password still works, so the rotation changed nothing", + ); +}); + +test("a consumer that goes away loses its login", { skip, timeout: 300_000 }, async () => { + // The half usually missing. A consumer removed from the mesh otherwise keeps a working login + // for ever and nothing says so — the same rule the host follows about removing what it declared + // and no longer declares. + await meshWrote([]); + const { out, ok } = await provision(); + assert.ok(ok, out); + assert.match(out, /revoked mesh_workstation/); + + assert.equal(await sql(`select rolcanlogin from pg_roles where rolname = 'mesh_workstation'`), "f"); + assert.equal( + await canLogIn("mesh_workstation", "second-password-bbb", "meshboard"), + false, + "a consumer nobody asks for any more can still log in", + ); +}); + +test("a role nobody here made is left alone", { skip, timeout: 300_000 }, async () => { + // A provisioner that removed every role it did not recognise could not safely be run on a + // database that predates it — which is every database anybody would want to adopt. + await sql(`create role someone_elses with login password 'theirs'`); + await sql(`create database theirs owner someone_elses`); + await meshWrote([]); + const { ok } = await provision(); + assert.ok(ok); + assert.equal(await sql(`select rolcanlogin from pg_roles where rolname = 'someone_elses'`), "t"); + assert.ok(await canLogIn("someone_elses", "theirs", "theirs")); +}); + +test("a manifest naming a credential that was never written is refused", { skip, timeout: 300_000 }, async () => { + // Rather than creating a role with no password — a login nothing can use, which nothing would + // report until something tried to connect. + await meshWrote([]); + await must( + `printf %s '{"contributions":1,"requirement":"database","given":[` + + `{"from":"meshboard","node":"ghost","secret":"${GRANTS}/ghost.secret","values":{"name":"ghost"}}` + + `]}' > ${GRANTS}/mesh.json`, + ); + const { out, ok } = await provision(); + assert.equal(ok, false, "it carried on past a missing credential"); + assert.match(out, /should be at .*ghost\.secret/); + assert.equal(await sql(`select count(*) from pg_roles where rolname = 'mesh_ghost'`), "0"); +});