diff --git a/README.md b/README.md index 9ed9167..39a6014 100644 --- a/README.md +++ b/README.md @@ -131,6 +131,74 @@ Placing needs a built host binary — set `MESH_LAB_HOST_BINARY` to one. It is a rather than a search on purpose: the declaration design leaves *where `place:` gets its artifacts from* open, and guessing would harden into the answer by accident. +## Pointing a run at the repositories + +**Every variable is an explicit path, and none of them has a default.** A test whose artifact was +not pointed at *skips* — it does not fail — so an unset variable is a green run that proved +nothing. That is `novox/hq` 04-ISSUES/005 exactly, and it has now been rediscovered twice, so it +is written down here rather than reconstructed a third time. + +```sh +export MESH_LAB_HOST_BINARY=/mesh-host +export MESH_LAB_BUNDLE=/examples/substrate-first-node.lock +export MESH_LAB_MODULES=/examples/modules +export MESH_LAB_BUILDER=/build/mesh-builder # build/, which is git-ignored + +# Built with `go build -o ./examples/` in mesh-control. +export MESH_LAB_PROVISIONER=/postgres-provisioner +export MESH_LAB_OBJECTSTORE_PROVISIONER=/objectstore-provisioner +export MESH_LAB_ROUTE_PROXY=/route-proxy +``` + +`MESH_LAB_HOST_BINARY` and `MESH_LAB_MODULES` do double duty: the repository each sits in is what +`suite` rebuilds and what the receipt claims. Point the run at a repository and it is built and +claimed; leave it out and it is neither. + +Check before running a long suite — it says which of these are missing rather than skipping +quietly: + +```sh +node --experimental-strip-types src/cli.ts check +``` + +If it says the daemon is not reachable, the group grant postdates the shell. `newgrp` fixes it, +but a heredoc into `newgrp` runs the suite as a child of a shell that then exits — start it with +`setsid nohup … &` inside the heredoc, or the run dies with the shell that launched it. + +## When something takes too long + +```sh +export MESH_LAB_LOG=info # or debug, or trace +export MESH_LAB_LOG_FILE=/tmp/lab.log # unset writes to stderr +``` + +| level | what it adds | +|---|---| +| `info` | each step of a raise, with how long the previous one took; anything that failed; **and a line every 15s naming whatever is still running** | +| `debug` | every command the lab runs — incus, docker, and anything local — with its duration, and the stderr of anything that failed | +| `trace` | what those commands printed | + +**The heartbeat is the point.** A stall is a command that started and has not finished, and the +only thing separating it from ordinary work is how long it has been going — which nothing can tell +you unless something is still counting. At `info` a run says `… docker load -i /tmp/image.tar — +still running after 45s` while it happens, rather than nothing until it gives up. + +Written with `appendFileSync`, so unlike a redirected stdout it cannot lag behind the run. It is +off unless asked for. + +**A redirected log lags, so do not diagnose a stall from it.** Node block-buffers stdout when it +is a file rather than a terminal, so `> run.log` can sit unchanged for minutes while the run is +working normally. On 2026-09-01 that was read as a stall twice, once after a real stall had just +been fixed — the most expensive kind of false signal, because it argues the fix did not work. Ask +the machines instead: + +```sh +incus list -c ns +incus exec -registry -- systemctl is-active docker +``` + +*"I cannot see progress" is not evidence of no progress.* + ## Measured on a workstation | | one machine | two machines | two machines + a router | @@ -140,7 +208,7 @@ artifacts from* open, and guessing would harden into the answer by accident. | restore, to usable again | 10.5 s | 11.6 s | — | A router adds seconds, not a boot: it is a container, because it is scenery rather than -something under test (`novox/hq` ADR 0033). +something under test (`novox/hq` ADR 0016). **Verified by running**, not asserted — a machine at `192.168.1.135` behind a household gateway, reached from a machine on a routable address: @@ -235,28 +303,55 @@ This repository carries implementation. It does not carry decisions. No build step — Node strips the types. ``` -npm test the declaration layer and the diagram, offline, 49 tests -npm run test:integration real scenarios against a real hypervisor, 14 tests +npm test the declaration layer and the diagram, offline +npm run test:integration real scenarios against a real hypervisor npm run typecheck source and tests both — a test that does not compile is a test that silently never ran npm run check typecheck + both suites — this is the gate +npm run last-run when this machine last ran the suite, and whether that still counts ``` -**A test names the decision it defends** (`novox/hq` ADR 0034). A decision with no test is one +**The integration suite rebuilds what it tests, and leaves a receipt saying it ran.** + +It needs a hypervisor, so it cannot run on every push — which means it runs when somebody +remembers, and *remembering is not a mechanism*. The harness this replaces had not built for two +and a half months and nothing said so (`novox/hq` 04-ISSUES/005). So: + +- **Before the run**, the host binary, the control-plane image and the builder are rebuilt from + source. The last two both parse manifests; building one and not the other is how a rename gets + tested against an eleven-hour-old binary. +- **After the run**, a receipt is written to XDG state — outside the repository, because the + question is *has this machine run it*, and a receipt in git would be a claim about everybody's + machine made by whoever committed last. +- `last-run` judges it and exits non-zero when it no longer counts: old, failed, taken against + commits the repositories have moved past, taken against a tree with uncommitted work, or a run + that never included the end-to-end file. + **A receipt that says nothing about something is not a receipt that clears it.** + +`suite ` runs something narrower, and the receipt records that it did — a green run of the +unit tests must not be readable as coverage of the pipeline. `--no-build` skips the rebuild, for +iterating on a test rather than on the code under it. + +**A test names the decision it defends** (`novox/hq` ADR 0017). A decision with no test is one that will quietly stop being true, and nobody learns that from a document: | Test | Defends | |---|---| -| the lab provides the underlay and nothing of the overlay | ADR 0031 | -| the workstation has no route into the scenario | ADR 0032 | -| a router is a container while machines are virtual machines | ADR 0033 | +| the lab provides the underlay and nothing of the overlay | ADR 0016 | +| the workstation has no route into the scenario | ADR 0016 | +| a router is a container while machines are virtual machines | ADR 0016 | | raise waits for *usable*, not for the call to return | the lifecycle design | | snapshots are whole-scenario | the lifecycle design | | a public range that is not documentation space is refused | the declaration design | | a scenario declaring what cannot be materialised is refused | the declaration design | -| the live diagram distinguishes scenery from a node | ADR 0033 | +| the live diagram distinguishes scenery from a node | ADR 0016 | | the live diagram draws what exists, never what was asked for | the diagram design | | a picture nobody can open is not a picture | the diagram design | +| a run that raised no machines is not end-to-end coverage | 04-ISSUES/005 | +| a run whose result could not be read writes nothing | 04-ISSUES/005 | +| the control plane's image and builder are always built together | 04-ISSUES/005 | +| every repository the receipt claims was built by the run | 04-ISSUES/005 | +| a run against uncommitted work does not cover the commit | 04-ISSUES/005 | **Mocking the hypervisor is forbidden.** A fake would assert that the fake behaves as expected, which is the shape of test this project exists to stop shipping. Integration tests skip with a diff --git a/package-lock.json b/package-lock.json index b8e8eba..7cd189e 100644 --- a/package-lock.json +++ b/package-lock.json @@ -11,7 +11,7 @@ "yaml": "^2.6.0" }, "bin": { - "mesh-lab": "dist/cli.js" + "mesh-lab": "src/cli.ts" }, "devDependencies": { "@types/node": "^22.0.0", diff --git a/package.json b/package.json index 26b5ed2..553c488 100644 --- a/package.json +++ b/package.json @@ -9,8 +9,9 @@ "scripts": { "typecheck": "tsc --noEmit && tsc --noEmit -p tsconfig.test.json", "test": "node --test --experimental-strip-types 'test/*.test.ts'", - "test:integration": "node --test --test-concurrency=1 --experimental-strip-types 'test/integration/*.test.ts'", - "check": "npm run typecheck && npm test && npm run test:integration" + "test:integration": "node --experimental-strip-types src/cli.ts suite", + "check": "npm run typecheck && npm test && npm run test:integration", + "last-run": "node --experimental-strip-types src/cli.ts last-run" }, "dependencies": { "yaml": "^2.6.0" diff --git a/provisioners/README.md b/provisioners/README.md new file mode 100644 index 0000000..13f5acc --- /dev/null +++ b/provisioners/README.md @@ -0,0 +1,47 @@ +# Provisioners + +The half that makes a credential real. + +The mesh generates a password, seals it to the machine that must accept it, and never holds the +value — so it cannot tell PostgreSQL, or MinIO, or a broker, to start accepting it. Something on +that machine reads what arrived and makes it true. That something is a provisioner, and it belongs +to the module that ships the software, not to the mesh. + +**What the mesh owns is the contract.** A provider module declares: + +```json +{ + "provides": [{"name": "database", "scope": "mesh"}], + "receives": {"database": "/var/lib/postgres/grants/mesh.json"}, + "grants": {"database": "/var/lib/postgres/grants"} +} +``` + +and is then given, by the host, from an ordinary declaration: + +| | | +|---|---| +| `mesh.json` | every consumer, what it asked for, and where its credential is | +| `..secret` | one consumer's password, alone in the file, sealed in transit and written in plain by the host. Named after both, because a consumer is a module on a machine and a node routinely runs several (`novox/hq` 04-ISSUES/022) | + +Two files rather than one because the mesh discarded the plaintext and cannot compose a document +containing it. The consequence is a good one: the readable half stays readable, and the secret +half changes only when the secret does. + +**A provisioner reconciles; it is not told what changed.** It runs after every declaration and +must reach the same state from wherever it starts. That means, in order: + +1. every consumer in the manifest has what it asked for, with the password it was given — set + every time, not only on creation, or a rotation reports success and changes nothing +2. **everything this provisioner made that is no longer asked for is removed.** A consumer that + goes away otherwise leaves a working login behind for ever, and nothing says so + +Step 2 is the half usually missing, and it is the same rule the host follows about removing what +it declared and no longer declares. + +The reference implementation lives in `mesh-control/examples/postgres-provisioner`, because that +is where the contract is defined and where the language is already set up to read it. The lab's +job is the other half: raising a real PostgreSQL and proving that what the mesh delivered becomes +a login that works, a rotation that takes effect, and a revocation that bites. + +Set `MESH_LAB_PROVISIONER` to a built one to run those. diff --git a/scenarios/a-provider.yml b/scenarios/a-provider.yml new file mode 100644 index 0000000..f221c17 --- /dev/null +++ b/scenarios/a-provider.yml @@ -0,0 +1,22 @@ +# One machine running a database that other machines use. +# +# It exists to prove the last step of a credential: the mesh generated a password, sealed it to +# this machine, and cannot tell PostgreSQL to accept it. Something here has to, and this is where +# that something is run against a real database rather than described. +scenario: a-provider + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + +images: + - postgres:17-alpine + +place: + all: [runtime] diff --git a/scenarios/a-public-name.yml b/scenarios/a-public-name.yml new file mode 100644 index 0000000..ecb24e8 --- /dev/null +++ b/scenarios/a-public-name.yml @@ -0,0 +1,26 @@ +# One machine serving a public name with a certificate from an authority it did not run itself. +# +# The lab keeps production's two-authority split rather than collapsing it (01-RESEARCH/004): the +# mesh's own authority certifies `.internal` names, and a name reachable from outside is certified +# by ACME. A single-authority lab would hide any fault living in that split, so this raises a real +# ACME server and makes the proxy actually order from it. +# +# Pebble rather than a stub, for the reason the lab exists at all: what is under test is whether an +# HTTP-01 challenge is answered at the name being certified, and a fake would be told to agree. +scenario: a-public-name + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + +images: + - ghcr.io/letsencrypt/pebble:2.5.0 + +place: + all: [runtime] diff --git a/scenarios/an-object-store.yml b/scenarios/an-object-store.yml new file mode 100644 index 0000000..7145f21 --- /dev/null +++ b/scenarios/an-object-store.yml @@ -0,0 +1,31 @@ +# One machine running an object store that other machines use. +# +# The same shape as `a-provider`, against a different kind of provision, and that is the whole +# reason it exists: novox/hq 04-ISSUES and the work breakdown's Phase 1.1 ask whether a module can +# be given a bucket the way it is given a database. The provisioning model is name-agnostic — the +# control plane special-cases neither — so what is unproven is not the mesh's half but the last +# step, where something on the machine turns a delivered secret into a key that works. +# +# It also proves the half a database does not: **a consumer must not be able to reach another +# consumer's bucket.** One store holds everybody's, where one PostgreSQL server holds separate +# databases, so isolation here is a policy somebody wrote rather than a boundary the product has. +scenario: an-object-store + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + +images: + - minio/minio:RELEASE.2025-09-07T16-13-09Z + # The vendor's client, stocked so the provisioner has the thing it drives without reaching a + # public registry from a documentation range. + - minio/mc:RELEASE.2025-08-13T08-35-41Z + +place: + all: [runtime] diff --git a/scenarios/audit-node.yml b/scenarios/audit-node.yml new file mode 100644 index 0000000..958c505 --- /dev/null +++ b/scenarios/audit-node.yml @@ -0,0 +1,30 @@ +# One machine that becomes a mesh and then assigns itself the audit logger. +# +# The substrate is first-node's — a store, a broker, the control plane — and one module image on +# top: the tool runtime carrying the audit-logger (mesh-catalog). The node enrols itself and the +# mesh assigns it the audit logger, so its events account is one the mesh delivered, not the +# broker's own (novox/hq ADR 0048). +scenario: audit-node + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + memory: 3GiB + cpus: 2 + +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + # The tool runtime with the audit-logger, built by scripts/build-runtime-image.sh into the local + # daemon and stocked into the scenario's own registry, which is where the host pulls it from. + - mesh-runtime-audit:development + +place: + all: [host, runtime] diff --git a/scenarios/bootstrap-with-registry.yml b/scenarios/bootstrap-with-registry.yml new file mode 100644 index 0000000..83a92f0 --- /dev/null +++ b/scenarios/bootstrap-with-registry.yml @@ -0,0 +1,23 @@ +# One machine and a registry, which is the smallest scenario that can exercise a container. +# +# A sealed machine cannot reach a registry and an image placed from an archive cannot keep its +# digest (novox/hq 04-ISSUES/009), so the lab raises one inside the scenario and serves the +# images below from it. What a declaration pins is reported when this is raised — the digest +# belongs to this registry, not to the one the image came from. +scenario: bootstrap-with-registry + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + +images: + - alpine:3.20 + +place: + all: [host, runtime] diff --git a/scenarios/first-node.yml b/scenarios/first-node.yml new file mode 100644 index 0000000..0566555 --- /dev/null +++ b/scenarios/first-node.yml @@ -0,0 +1,29 @@ +# One machine, raising a substrate from the bundle its host carries. +# +# This is the bootstrap class (novox/hq ADR 0009): no forge, no control plane to talk to, no +# delivery. It exists to develop the steps of raising a mesh on a machine with no route out — +# a container runtime, a store, the control plane's schema in it, and the broker. +# +# It stops before the control plane *runs*, because there is nothing for it to serve yet. +scenario: first-node + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + +# Placed into a registry the scenario raises, which is what a real node pulls from anyway. The +# digests below are the ones that registry assigns, and that satisfies pinning: what is required +# is a reference that is exact and cannot move (novox/hq ADR 0006). +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + +place: + all: [host, runtime] diff --git a/scenarios/grafana-node.yml b/scenarios/grafana-node.yml new file mode 100644 index 0000000..dac3864 --- /dev/null +++ b/scenarios/grafana-node.yml @@ -0,0 +1,28 @@ +# One machine that becomes a mesh and assigns itself grafana's tool runtime, configured by settings. +# +# plex/sonarr prove a runtime that self-detects its credential; this proves the other half of the +# config story (novox/hq ADR 0051 + 0052): the operator states grafana's URL and token as the +# assignment's settings, the mesh merges them into the config file the runtime reads, and the +# runtime registers and serves grafana's tools from that — no credential baked into the manifest. +scenario: grafana-node + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + memory: 3GiB + cpus: 2 + +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + - mesh-runtime-grafana:development + +place: + all: [host, runtime] diff --git a/scenarios/growing-mesh.yml b/scenarios/growing-mesh.yml new file mode 100644 index 0000000..41fcd55 --- /dev/null +++ b/scenarios/growing-mesh.yml @@ -0,0 +1,33 @@ +# Three machines, joined one at a time. +# +# The point is not the third machine. It is that adding one changes **every other node's** peer +# list: each existing node has to be told again, or the newcomer is on a network nobody else can +# see. A mesh that only configures the arriving node looks like it worked and is half a network. +# +# So this scenario exists to be raised, then grown — enrol two, push, check; enrol the third, +# push, and check that the first two changed. +scenario: growing-mesh + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + laptop: + at: { segment: hosting, address: [192.0.2.20] } + inbound: allow + workstation: + at: { segment: hosting, address: [192.0.2.30] } + inbound: allow + +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + +place: + all: [host, runtime] diff --git a/scenarios/minio-node.yml b/scenarios/minio-node.yml new file mode 100644 index 0000000..764ad88 --- /dev/null +++ b/scenarios/minio-node.yml @@ -0,0 +1,32 @@ +# One machine that becomes a mesh and grants a consumer an S3 bucket from an assigned minio provider. +# +# The postgres bed proves the provider/consumer contract for a database; this proves it for object +# storage (novox/hq ADR 0052/0053), on a provider whose code drives the `mc` CLI (so the runtime image +# carries it): minio is assigned, a consumer that requires s3-bucket is assigned, and the mesh mints +# one secret key; minio's provisioner creates a bucket and a service account under the access key the +# mesh derived with the secret it minted, and the consumer reaches its bucket with only that. +scenario: minio-node + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + memory: 3GiB + cpus: 2 + +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + - minio/minio:latest + # minio's runtime, built by scripts/build-module-runtime.sh minio (it carries mc), stocked into the + # scenario's own registry. + - mesh-runtime-minio:development + +place: + all: [host, runtime] diff --git a/scenarios/plex-node.yml b/scenarios/plex-node.yml new file mode 100644 index 0000000..67724b9 --- /dev/null +++ b/scenarios/plex-node.yml @@ -0,0 +1,31 @@ +# One machine that becomes a mesh and then assigns itself plex's tool runtime. +# +# The audit-node bed proved an assigned *consumer* (novox/hq ADR 0048). This proves an assigned +# module that *serves tools* (ADR 0052): the same first-node substrate, plus plex's tool runtime on +# top. The node enrols itself, the mesh issues plex a broker account scoped to serve.plex.* and +# assigns it, the host runs the runtime container, and a caller invokes plex.plex_reachable over the +# mesh — proof the module runs its own code as its own process under its own scoped account. +scenario: plex-node + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + memory: 3GiB + cpus: 2 + +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + # Plex's tool runtime, built by scripts/build-module-runtime.sh plex into the local daemon and + # stocked into the scenario's own registry, which is where the host pulls it from. + - mesh-runtime-plex:development + +place: + all: [host, runtime] diff --git a/scenarios/postgres-node.yml b/scenarios/postgres-node.yml new file mode 100644 index 0000000..36d7ad0 --- /dev/null +++ b/scenarios/postgres-node.yml @@ -0,0 +1,30 @@ +# One machine that becomes a mesh and grants a consumer a database from an assigned postgres provider. +# +# The redis mesh-grant bed proves the whole provider/consumer contract for a cache; this proves it for +# a database (novox/hq ADR 0052/0053): postgres's runtime carries psql, its provisioner creates a role +# and database under the login the mesh derived with the password the mesh minted, and a consumer +# connects to its own database with only what the mesh delivered. +scenario: postgres-node + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + memory: 3GiB + cpus: 2 + +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + # postgres's runtime, built by scripts/build-module-runtime.sh postgres (it carries psql), stocked + # into the scenario's own registry. + - mesh-runtime-postgres:development + +place: + all: [host, runtime] diff --git a/scenarios/redis-node.yml b/scenarios/redis-node.yml new file mode 100644 index 0000000..9c44d97 --- /dev/null +++ b/scenarios/redis-node.yml @@ -0,0 +1,31 @@ +# One machine that becomes a mesh and then assigns itself redis — a *provider* module. +# +# plex-node proves an assigned module that serves tools (novox/hq ADR 0052). This proves the same +# for a provider: redis's runtime runs its provisioner AND its tools as one process under one scoped +# broker account. The provisioner emitting a lifecycle event is the thing 0052 fixes — before it, +# the provisioner ran in a container with no broker and its emit could not fire. +scenario: redis-node + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + memory: 3GiB + cpus: 2 + +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + - redis:7-alpine + # Redis's tool+provisioner runtime, built by scripts/build-module-runtime.sh redis into the local + # daemon and stocked into the scenario's own registry, which is where the host pulls it from. + - mesh-runtime-redis:development + +place: + all: [host, runtime] diff --git a/scenarios/sonarr-node.yml b/scenarios/sonarr-node.yml new file mode 100644 index 0000000..85dcaf9 --- /dev/null +++ b/scenarios/sonarr-node.yml @@ -0,0 +1,28 @@ +# One machine that becomes a mesh and assigns itself sonarr's tool runtime. +# +# plex-node proved a tools+events module that self-detects its token from a mounted config dir; this +# proves the same self-configuring pattern generalises to the Servarr family (novox/hq ADR 0052): +# sonarr's runtime detects its API key from the server's config.xml and serves sonarr's tools over a +# mesh-issued scoped account, with no live Sonarr to reach. +scenario: sonarr-node + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + memory: 3GiB + cpus: 2 + +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + - mesh-runtime-sonarr:development + +place: + all: [host, runtime] diff --git a/scenarios/two-nodes.yml b/scenarios/two-nodes.yml new file mode 100644 index 0000000..bbfeb6f --- /dev/null +++ b/scenarios/two-nodes.yml @@ -0,0 +1,63 @@ +# Two machines, one mesh. +# +# The first raises everything from the bundle its host carries and joins the mesh it made. The +# second is an ordinary node: it has a host and nothing else, and a person carries it a token. +# +# This is the first scenario where the mesh is a mesh. Everything before it proved a machine could +# talk to a control plane on its own loopback, which proves less than it looks. +scenario: two-nodes + +segments: + hosting: + kind: public + cidr: [192.0.2.0/24] + +machines: + anchor: + at: { segment: hosting, address: [192.0.2.10] } + inbound: allow + # The whole substrate, the registry, the builder, an adopted workload and the modules under + # test all land here — eleven containers before the forge arrives. At the 1GiB default this + # machine thrashes, and it presents as "the mesh hangs": every exec slows from 15s to 105s + # and the forge test fails on a status poll that is merely queued behind page-outs. + memory: 4GiB + cpus: 4 + laptop: + at: { segment: hosting, address: [192.0.2.20] } + inbound: allow + memory: 2GiB + +images: + - postgres:17-alpine + - cloudamqp/lavinmq:latest + - mesh-control:development + # So a module can mirror one into a registry of the mesh's own. The scenario's registry serves + # what the mesh's registry is built from — the same chicken-and-egg the bootstrap has, resolved + # the same way. + - registry:2 + # A real third-party workload, for adopting one the way the conversion will. Its database is + # the substrate's postgres image rather than its own: what is under test is the mesh delivering + # a module, not which postgres it delivers. + - ghcr.io/umami-software/umami:postgresql-latest + # And the builder, because it is a module the mesh assigns rather than a program somebody + # starts by hand — which is the only way its credential can be one the mesh delivered. + - mesh-builder:development + # And the provisioner, which is what makes a sealed credential true on a machine — the mesh + # discarded the plaintext and cannot tell a database to start accepting it. + - mesh-provision-postgres:development + # And the proxy, which is what turns a route grant into traffic actually arriving. + - mesh-route-proxy:development + # And the cache, with its provisioner — the third provision after a database and a bucket, + # and the first whose tenancy is a keyspace rather than a namespace something else enforces. + - redis:7-alpine + - mesh-provision-redis:development + # And the object store's provisioner, so the module describing it can be planned. Without it + # that module still names an image nothing serves, and planning it is refused — correctly. + - mesh-provision-objectstore:development + # And a forge, so one of the real module descriptions can be started rather than only planned. + # It is the first of them to run: it needs a database from another module, a credential it did + # not choose, and a connection string it could not have written itself. + - gitea/gitea:1.22 + +place: + all: [host, runtime] diff --git a/scripts/build-module-runtime.sh b/scripts/build-module-runtime.sh new file mode 100755 index 0000000..dd6b340 --- /dev/null +++ b/scripts/build-module-runtime.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Build a per-module runtime image (novox/hq ADR 0052): the tool runtime carrying ONE module's +# compiled code, which serves that module's tools and runs its events/provisioner under the module's +# own scoped broker account. Generalises build-runtime-image.sh from the audit-logger to any module. +# +# build-module-runtime.sh +# -> tags mesh-runtime-:development and saves it to +set -euo pipefail + +MODULE="${1:?usage: build-module-runtime.sh }" +OUT="${2:?usage: build-module-runtime.sh }" +HERE="$(cd "$(dirname "$0")/.." && pwd)"; ROOT="$(cd "$HERE/.." && pwd)" +MESH_TOOLS="${MESH_TOOLS:-$ROOT/mesh-tools}" +MESH_SDK="${MESH_SDK:-$ROOT/mesh-sdk}" +MESH_CATALOG="${MESH_CATALOG:-$ROOT/mesh-catalog}" +MOD="$MESH_CATALOG/modules/$MODULE" +TAG="${RUNTIME_TAG:-mesh-runtime-$MODULE:development}" +BASE="${RUNTIME_BASE:-node:22-bookworm-slim}" +[ -d "$MOD" ] || { echo "no module $MODULE at $MOD" >&2; exit 1; } + +( cd "$MESH_SDK" && npm run build >/dev/null ) +( cd "$MESH_TOOLS" && npm run build >/dev/null ) +# Compile whichever of the module's entrypoints exist. +SRCS=(); for f in client.ts index.ts tools/index.ts provisioner/index.ts; do [ -f "$MOD/$f" ] && SRCS+=("$f"); done +TSC="$MESH_SDK/node_modules/.bin/tsc"; ( cd "$MOD" && "$TSC" "${SRCS[@]}" --module NodeNext --moduleResolution NodeNext --target ES2022 --outDir dist >/dev/null ) + +STAGE="$(mktemp -d)"; trap 'rm -rf "$STAGE"' EXIT +cp -r "$MESH_TOOLS/dist" "$STAGE/dist" +cp -rL "$MESH_TOOLS/node_modules" "$STAGE/node_modules" +mkdir -p "$STAGE/modules/$MODULE"; cp -r "$MOD/dist" "$STAGE/modules/$MODULE/dist" +cp "$MESH_TOOLS/package.json" "$STAGE/package.json" + +# The entrypoints the runtime loads: tools, events and (a provider's) provisioner, whichever exist. +ENTRIES=""; for e in tools/index.js index.js provisioner/index.js; do + [ -f "$STAGE/modules/$MODULE/dist/$e" ] && ENTRIES="${ENTRIES:+$ENTRIES,}/app/modules/$MODULE/dist/$e" +done + +# A module whose code drives a CLI needs that CLI in the image — postgres shells out to `psql`, minio +# to `mc`. Everything else speaks a wire protocol or HTTP and needs nothing added. +EXTRA="" +case "$MODULE" in + postgres) EXTRA='RUN apt-get update && apt-get install -y --no-install-recommends postgresql-client && rm -rf /var/lib/apt/lists/*' ;; + minio) EXTRA='COPY --from=minio/mc:latest /usr/bin/mc /usr/bin/mc' ;; +esac + +cat > "$STAGE/Dockerfile" < $OUT" diff --git a/scripts/build-runtime-image.sh b/scripts/build-runtime-image.sh new file mode 100755 index 0000000..41cabec --- /dev/null +++ b/scripts/build-runtime-image.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# Build the runtime+audit-logger image the events test runs, and save it to a tar. +# +# The image is the tier-3 tool runtime (mesh-tools) carrying one tier-4 module (audit-logger) and +# the sdk it imports. It is what MESH_LAB_RUNTIME points at: +# +# scripts/build-runtime-image.sh /tmp/mesh-runtime-audit.tar +# MESH_LAB_RUNTIME=/tmp/mesh-runtime-audit.tar node --test test/integration/events.test.ts +# +# The sdk is vendored (dereferenced), not npm-installed: the sdk is not published, and the lab +# machine has no route out anyway — the image must be self-contained. Sibling repositories are +# assumed alongside this one; override with MESH_TOOLS / MESH_SDK / MESH_CATALOG. +set -euo pipefail + +OUT="${1:?usage: build-runtime-image.sh }" +HERE="$(cd "$(dirname "$0")/.." && pwd)" +ROOT="$(cd "$HERE/.." && pwd)" +MESH_TOOLS="${MESH_TOOLS:-$ROOT/mesh-tools}" +MESH_SDK="${MESH_SDK:-$ROOT/mesh-sdk}" +MESH_CATALOG="${MESH_CATALOG:-$ROOT/mesh-catalog}" +AUDIT="$MESH_CATALOG/modules/audit-logger" +TAG="${RUNTIME_TAG:-mesh-runtime-audit:development}" +BASE="${RUNTIME_BASE:-node:22-bookworm-slim}" + +echo "building $TAG from:" +echo " runtime $MESH_TOOLS" +echo " sdk $MESH_SDK" +echo " module $AUDIT" + +# Compile the three, so the image carries current dist. The sdk first — the others import it. +( cd "$MESH_SDK" && npm run build >/dev/null ) +( cd "$MESH_TOOLS" && npm run build >/dev/null ) +( cd "$AUDIT" && npx tsc audit.ts index.ts --module NodeNext --moduleResolution NodeNext \ + --target ES2022 --outDir dist >/dev/null ) + +STAGE="$(mktemp -d)" +trap 'rm -rf "$STAGE"' EXIT +cp -r "$MESH_TOOLS/dist" "$STAGE/dist" +cp -rL "$MESH_TOOLS/node_modules" "$STAGE/node_modules" # -L materialises the @novox/mesh-sdk symlink +mkdir -p "$STAGE/modules/audit-logger" +cp -r "$AUDIT/dist" "$STAGE/modules/audit-logger/dist" +cp "$MESH_TOOLS/package.json" "$STAGE/package.json" + +cat > "$STAGE/Dockerfile" < $OUT" diff --git a/src/cli.ts b/src/cli.ts index 2cb9eaa..e353173 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -30,6 +30,19 @@ const USAGE = `mesh-lab — raise a disposable mesh on one machine diagram [out.drawio] draw what a scenario asks for diagram --live [out.drawio] draw what is actually raised + warm the scenario kept between runs, and whether it still counts + warm cool destroy it and forget it + suite [paths...] [--no-build] rebuild the artifacts, run the end-to-end tests, leave a receipt + last-run whether the last run still counts; non-zero when it does not + + connect [instance] reach the standing scenario from this workstation, by name + disconnect give the address back and stop answering those names + connected what is reachable right now + +A scenario is a closed address space, so only one can be reachable at a time: connect refuses +rather than guessing which you meant. It needs root for an address and a resolver rule, and +disconnect puts both back. + Set MESH_LAB_INCUS if the daemon needs a different invocation, e.g. "sudo -n incus". `; @@ -91,6 +104,73 @@ async function main(): Promise { case "check": return check(); + // Rebuilding, running, and recording that it ran are one act. Separate commands would mean a + // run against a stale artifact, or a run nobody recorded — and both are the state + // novox/hq 04-ISSUES/005 is about. + case "suite": { + const { runSuite } = await import("./suite.ts"); + process.exitCode = await runSuite(rest); + return; + } + + // **What 04-ISSUES/005 says nobody was ever told.** The suite needs a machine with a + // hypervisor, so it cannot run on every push — which means it runs when somebody remembers, + // and remembering is not a mechanism. This asks whether the last run still means anything, + // and exits non-zero when it does not, so a timer or a person can act on it. + case "last-run": { + const { read, judge, whatWasTested } = await import("./lastrun.ts"); + const said = judge(read(), new Date(), whatWasTested()); + for (const line of said.lines) console.log(line); + if (!said.current) { + console.log("\n `npm run check` runs it."); + process.exitCode = 1; + } + return; + } + + // A base state many tests start from, rather than each raising its own mesh. + // + // **The speed is the lesser half.** Tests that share one long-lived mesh accumulate each + // other's state, and a test that reads what the previous one left is a test that passes for + // the wrong reason — which has already happened here once. Returning to a named state between + // tests makes each of them independent. + case "warm": { + const { remembered, ready, cool } = await import("./warm.ts"); + const what = rest[0] ?? "status"; + if (what === "cool") { + const gone = await cool(); + console.log(gone ? `destroyed ${gone}, and forgot it` : "nothing was being kept warm"); + return; + } + const held = remembered(); + if (!held) { + console.log("nothing is being kept warm."); + console.log(" a scenario is warmed by whatever brought it to a state worth keeping;"); + console.log(" the integration suite does it when MESH_LAB_WARM is set."); + return; + } + console.log(`${held.instanceId} — ${held.scenario}, warmed ${held.at}`); + for (const [name, commit] of Object.entries(held.against)) { + console.log(` ${name.padEnd(14)} ${commit}`); + } + const said = await ready(held.scenario); + console.log(said.use === "restore" + ? "\n usable: it can be returned to" + : `\n NOT usable: ${said.why}`); + if (said.use !== "restore") process.exitCode = 1; + return; + } + + case "base": { + // `base build` exists because a sealed scenario cannot install a container runtime, and + // the runtime has to come from somewhere with a network (novox/hq ADR 0006). + if (rest[0] !== "build") fail("base needs a subcommand: build"); + const { buildBaseImage } = await import("./lifecycle/base.ts"); + const built = await buildBaseImage((line) => console.log(line)); + console.log(`${built.alias}: built, with docker ${built.runtime}`); + return; + } + case "validate": { const path = rest[0] ?? fail("validate needs a scenario file"); const scenario = loadScenario(path); @@ -110,6 +190,38 @@ async function main(): Promise { return; } + case "connect": { + const { connect } = await import("./lifecycle/connect.ts"); + const reached = await connect(rest[0]); + console.log(`connected to ${reached.instanceId} as ${reached.address} on ${reached.bridge}\n`); + console.log("these answer here now:"); + for (const [machine, address] of Object.entries(reached.machines).sort()) { + console.log(` anything.${machine}.internal → ${address}`); + } + console.log(`\ntry: curl -sI http://${Object.keys(reached.machines)[0]}.internal`); + console.log("run `mesh-lab disconnect` when finished — these names are only true while"); + console.log("that scenario is standing."); + return; + } + + case "disconnect": { + const { disconnect } = await import("./lifecycle/connect.ts"); + for (const line of await disconnect()) console.log(line); + return; + } + + case "connected": { + const { connection } = await import("./lifecycle/connect.ts"); + const now = await connection(); + if (now.length === 0) { + console.log("nothing is connected"); + process.exitCode = 1; + return; + } + for (const line of now) console.log(` ${line}`); + return; + } + case "raise": { const path = rest[0] ?? fail("raise needs a scenario file"); const scenario = loadScenario(path); @@ -120,6 +232,12 @@ async function main(): Promise { }); const seconds = ((Date.now() - started) / 1000).toFixed(1); console.log(`\nraised ${raised.instanceId} in ${seconds}s — ${raised.machines.length} machines usable`); + if (raised.images.length > 0) { + // Printed because this is what a declaration pins, and it is not knowable until the + // scenario has been raised — the digest belongs to this registry. + console.log(`\nimages served, pinned by digest:`); + for (const image of raised.images) console.log(` ${image}`); + } if (scenario.snapshot) { const took = await snapshot(raised.instanceId, scenario.snapshot); console.log(`snapshot '${scenario.snapshot}' in ${took.toFixed(2)}s`); diff --git a/src/declaration/parse.ts b/src/declaration/parse.ts index 26714b7..0911ae0 100644 --- a/src/declaration/parse.ts +++ b/src/declaration/parse.ts @@ -37,6 +37,9 @@ function normaliseMachine(raw: unknown): Machine { } const inbound = machine["inbound"]; if (inbound === "allow" || inbound === "deny") result.inbound = inbound; + if (machine["egress"] === true) result.egress = true; + if (machine["memory"] !== undefined) result.memory = String(machine["memory"]); + if (machine["cpus"] !== undefined) result.cpus = Number(machine["cpus"]); return result; } @@ -96,6 +99,11 @@ export function parseScenario(text: string): Scenario { }); } + const images = raw["images"]; + if (Array.isArray(images)) { + scenario.images = images.map((i) => String(i)); + } + const place = raw["place"]; if (place && typeof place === "object") { const normalised: Record = {}; diff --git a/src/declaration/types.ts b/src/declaration/types.ts index 2721a2d..8b70f04 100644 --- a/src/declaration/types.ts +++ b/src/declaration/types.ts @@ -85,6 +85,31 @@ export interface Machine { * v6-addressed machine. Without this, v6 addressing would imply reachability. */ inbound?: "allow" | "deny"; + /** + * Whether this machine can reach the world outside the scenario. + * + * **Off unless asked for.** A scenario is a closed address space, and a machine that could + * reach anything would make every test's result depend on what else was reachable that day. + * It is declared for the same reason an address is: so what a run proves is what the scenario + * says, and not what the workstation happened to have. + * + * What it is for is the one thing a mesh genuinely cannot do without an outside: a first node + * fetching the images it starts from, before there is any mesh to serve them + * (novox/hq 04-ISSUES/029). + */ + egress?: boolean; + /** + * How big the machine is. Absent means the lab's default, which suits a machine running a host + * and a handful of containers. + * + * Declared, because it is a fact about the machine the scenario describes — the node that runs + * the whole substrate is bigger than the laptop that joins it, and a test that starves its + * anchor at the default answers questions about memory pressure, not about the mesh. The forge + * test failed three times as "status hangs" before anyone counted the containers in 1GiB + * (novox/hq 04-ISSUES/024 is the same lesson about a different resource). + */ + memory?: string; + cpus?: number; } /** Reachability between segments, as a segmented router enforces it. Asymmetric by design. */ @@ -109,6 +134,14 @@ export interface Scenario { machines: Record; policy?: Policy[]; place?: Placement; + /** + * Container images this scenario needs inside it. + * + * A sealed machine cannot reach a registry, so the lab raises one on a public segment and + * serves these from it. Written as tags — the digest a declaration pins is the one THIS + * registry assigns, and it is reported when the scenario is raised. + */ + images?: string[]; /** Name the state once placement finishes, so a run can return to it. */ snapshot?: string; } diff --git a/src/declaration/validate.ts b/src/declaration/validate.ts index f705a99..ee6f2da 100644 --- a/src/declaration/validate.ts +++ b/src/declaration/validate.ts @@ -199,11 +199,27 @@ export function validate(scenario: Scenario): void { } } + // "uplink" is the one network name the lab itself claims, for the NAT bridge behind + // `egress: true`. A segment wearing it would be created first, as an isolated bridge — and the + // uplink code, finding a network by that name, would attach egress machines to it. No error + // anywhere, no route anywhere: a scenario key silently ignored, which is the fault this file + // exists to refuse. + if (scenario.segments["uplink"]) { + problems.push(`segment 'uplink': the name is reserved for the lab's own NAT bridge — ` + + `an egress machine would be silently attached to this segment instead of the world`); + } + for (const [name, machine] of Object.entries(scenario.machines)) { if (machine.at === "detached") { if (machine.published?.length) { problems.push(`machine '${name}' is detached but declares published ports`); } + // The same fault as publishing from nowhere: raising it would drop the key on the floor, + // and a scenario key the runtime silently ignores is the thing this lab exists to catch. + if (machine.egress) { + problems.push(`machine '${name}' is detached but declares egress — a machine on no ` + + `segment reaches nothing, the world included`); + } continue; } @@ -299,5 +315,29 @@ export function validate(scenario: Scenario): void { } } + for (const image of scenario.images ?? []) { + if (!image.trim()) { + problems.push("images: an empty entry names nothing"); + } else if (image.includes("@sha256:")) { + // The digest a declaration pins is the one the LAB's registry assigns, which is not + // knowable before the scenario is raised. Naming an upstream digest here would pin + // something this registry will never serve. + problems.push( + `images: '${image}' is pinned by digest. Name it by tag — the lab's registry assigns ` + + `its own digest and reports it when the scenario is raised`, + ); + } + } + if ((scenario.images ?? []).length > 0) { + const hasPublicV4 = Object.values(scenario.segments) + .some((s) => s.kind === "public" && s.cidr.some((c) => !c.includes(":"))); + if (!hasPublicV4) { + problems.push( + "images: this scenario declares images and has no public IPv4 segment to serve them " + + "from. The registry stands in for the outside world, so it sits on a public segment", + ); + } + } + if (problems.length > 0) throw new DeclarationError(problems); } diff --git a/src/diagram/from-live.ts b/src/diagram/from-live.ts index 2529ca0..8befac7 100644 --- a/src/diagram/from-live.ts +++ b/src/diagram/from-live.ts @@ -8,7 +8,7 @@ * by the running machine now — nothing is inferred from a file on disk. */ -import { incusOk, taggedNetworks } from "../incus/client.ts"; +import { incus, incusOk, taggedNetworks } from "../incus/client.ts"; import { depthOf, type Diagram, type DiagramMachine, type DiagramSegment } from "./model.ts"; interface RawInstance { @@ -28,7 +28,10 @@ interface RawInstance { export async function diagramFromLive(instanceId: string): Promise { const networks = (await taggedNetworks()).filter((n) => n.instanceId === instanceId); - const json = (await incusOk(["list", "--format", "json"], 30_000)) ?? "[]"; + // Not `incusOk(...) ?? "[]"`. A picture is read from what runs (novox/hq ADR 0018), and a read + // that failed and became an empty list would draw an empty scenario rather than fail — a + // diagram that is confidently wrong, which is worse than no diagram. + const json = (await incus(["list", "--format", "json"], 30_000)).stdout.trim() || "[]"; const parsed = JSON.parse(json) as RawInstance[]; const mine = parsed.filter((i) => i.config?.["user.mesh-lab.instance"] === instanceId); if (mine.length === 0 && networks.length === 0) throw new Error(`no scenario instance '${instanceId}'`); diff --git a/src/incus/client.ts b/src/incus/client.ts index eab22d6..31d9e66 100644 --- a/src/incus/client.ts +++ b/src/incus/client.ts @@ -13,6 +13,8 @@ import { spawn } from "node:child_process"; +import { around, log, shorten } from "../log.ts"; + /** * How to invoke incus. Overridable because the socket is group-owned and a session that * predates the group grant cannot reach it — which is a real thing that happens on the @@ -52,6 +54,27 @@ export class IncusError extends Error { * that worked perfectly when typed. */ export async function incus(args: string[], timeoutMs = 60_000): Promise { + // **Every command through here is recorded** (novox/hq 04-ISSUES/024). This is one of three + // places the lab runs an external program, and ninety-odd call sites reach a hypervisor through + // it — so logging here covers all of them and none of them has to remember to. + return invoke(args, timeoutMs, false); +} + +/** + * `expectedToFail` is not about this command; it is about the caller. + * + * `incus` rejects and the caller is expected to care. `incusOk` and `succeeds` turn a failure into + * an answer — *does this network exist*, *is the agent up yet* — and those are asked constantly + * while a scenario comes up. Recording them as faults fills a healthy run with ✗. + */ +function invoke(args: string[], timeoutMs: number, expectedToFail: boolean): Promise { + return around(`incus ${shorten(args)}`, () => run(args, timeoutMs), { + heartbeatMs: 15_000, + expectedToFail, + }); +} + +function run(args: string[], timeoutMs: number): Promise { const [command, ...prefix] = INCUS; if (!command) throw new Error("MESH_LAB_INCUS is empty"); @@ -80,10 +103,15 @@ export async function incus(args: string[], timeoutMs = 60_000): Promise { clearTimeout(timer); if (timedOut) { + // Said explicitly. A SIGKILL leaves an empty stderr, so without this the failure arrives + // with no explanation at all — which is how the first raise reported "(no output)". + log.info(`incus ${shorten(args)} was killed after ${timeoutMs}ms`); reject(new IncusError(args, `timed out after ${timeoutMs}ms`, null)); } else if (code === 0) { + if (stdout.trim()) log.trace(` stdout: ${shorten([stdout.trim()], 400)}`); resolve({ stdout, stderr }); } else { + log.debug(` exit ${code}: ${shorten([stderr.trim() || "(nothing on stderr)"], 400)}`); reject(new IncusError(args, stderr, code)); } }); @@ -105,12 +133,28 @@ export async function incus(args: string[], timeoutMs = 60_000): Promise { try { - return (await incus(args, timeoutMs)).stdout; + return (await invoke(args, timeoutMs, true)).stdout; } catch { return null; } } +/** + * Ask incus what exists, where answering "none" without having looked would be a lie. + * + * The comment on `incusOk` above warns that absence and success must not be made + * indistinguishable. Three of its callers then wrote `?? "[]"` and did exactly that, and it cost + * a session: `mesh-lab list` reported *no scenario instances standing* while two were standing, + * because this shell had no permission to reach the daemon. The lab was not wrong about the + * instances — it had never managed to ask. + * + * So anything enumerating what exists comes through here and throws. `incusOk` remains right for + * questions where failure genuinely means no, like `instanceExists`. + */ +async function enumerate(args: string[], timeoutMs = 30_000): Promise { + return (await incus(args, timeoutMs)).stdout; +} + /** Did the command work? For commands whose output is not the point. */ export async function succeeds(args: string[], timeoutMs = 60_000): Promise { return (await incusOk(args, timeoutMs)) !== null; @@ -168,14 +212,18 @@ export interface TaggedInstance { * nothing. Metadata is what the instance actually knows about itself. */ export async function taggedInstances(): Promise { - const json = (await incusOk(["list", "--format", "json"], 30_000)) ?? "[]"; + const json = (await enumerate(["list", "--format", "json"])).trim() || "[]"; let parsed: unknown; try { parsed = JSON.parse(json); } catch { - return []; + throw new IncusError(["list", "--format", "json"], + `incus answered something that is not JSON: ${json.slice(0, 200)}`, null); + } + if (!Array.isArray(parsed)) { + throw new IncusError(["list", "--format", "json"], + "incus answered JSON that is not a list of instances", null); } - if (!Array.isArray(parsed)) return []; const tagged: TaggedInstance[] = []; for (const entry of parsed) { @@ -199,14 +247,17 @@ export interface TaggedNetwork { } export async function taggedNetworks(): Promise { - const json = (await incusOk(["network", "list", "--format", "json"], 30_000)) ?? "[]"; + const args = ["network", "list", "--format", "json"]; + const json = (await enumerate(args)).trim() || "[]"; let parsed: unknown; try { parsed = JSON.parse(json); } catch { - return []; + throw new IncusError(args, `incus answered something that is not JSON: ${json.slice(0, 200)}`, null); + } + if (!Array.isArray(parsed)) { + throw new IncusError(args, "incus answered JSON that is not a list of networks", null); } - if (!Array.isArray(parsed)) return []; const tagged: TaggedNetwork[] = []; for (const entry of parsed) { diff --git a/src/lastrun.ts b/src/lastrun.ts new file mode 100644 index 0000000..cfcebb4 --- /dev/null +++ b/src/lastrun.ts @@ -0,0 +1,213 @@ +/** + * When the end-to-end suite last ran, and against what. + * + * **The fault this exists for is not that the suite breaks — it is that nobody notices it stopped + * running** (novox/hq 04-ISSUES/005). The harness it replaces had not built for two and a half + * months, and nothing said so; the coverage was assumed rather than checked, and several of the + * pipeline's most expensive defects landed inside that window. + * + * This suite is in a better position and the same danger: it needs a machine with a hypervisor, so + * it cannot run on every push, which means it runs when somebody remembers. Remembering is not a + * mechanism. + * + * So a run leaves a receipt, and something can be asked whether the receipt still means anything. + * A receipt that is old, or taken against code the repositories have since moved past, is the + * thing 005 says nobody was ever told. + * + * **Kept outside the repository**, because the question is *has this machine run it* rather than + * *what is committed* — and a receipt in git would be a claim about everyone's machine made by + * whoever committed last. + */ + +import { execFileSync } from "node:child_process"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { dirname, join } from "node:path"; + +import { repositories } from "./repos.ts"; + +/** What a run was taken against, per repository. */ +export type Against = Record; + +export interface Receipt { + /** When it finished, ISO 8601. */ + at: string; + passed: number; + failed: number; + /** The commit each repository was at. Absent for anything that was not a git checkout. */ + against: Against; + /** The test files this run was pointed at. See {@link endToEnd}. */ + ran: string[]; +} + +/** + * endToEnd is the file that raises real machines. A run that did not include it proved nothing + * about the pipeline, however green it was. + * + * The suite takes paths, so it can be pointed at one quick file — and the receipt from that would + * otherwise be indistinguishable from a receipt for the real thing. That is 04-ISSUES/005 again: + * not a suite that fails, a record that says more than the run behind it. + */ +export const endToEnd = "test/integration/mesh.test.ts"; + +/** Where the receipt lives: XDG state, which is for exactly this — data a tool keeps between runs. */ +export function receiptPath(): string { + const state = process.env["XDG_STATE_HOME"] ?? join(homedir(), ".local", "state"); + return join(state, "mesh-lab", "last-run.json"); +} + +/** + * headOf is the commit a directory's repository is at, or "" if it is not one. + * + * **A tree with uncommitted changes is marked, and never equal to the clean commit it sits on.** + * The run tested what was on disk, and that is not what the commit contains — so a receipt naming + * the bare hash would claim coverage of code nobody can check out. Nothing else could tell: the + * hash is identical either way. That is 04-ISSUES/005's overclaim in its quietest form. + */ +export function headOf(directory: string): string { + const git = (args: string[]) => + execFileSync("git", ["-C", directory, ...args], { + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + }); + try { + const head = git(["rev-parse", "--short", "HEAD"]).trim(); + const dirty = git(["status", "--porcelain"]).trim() !== ""; + return dirty ? `${head}+uncommitted` : head; + } catch { + // Not a checkout, or no git. Absent rather than guessed: a receipt claiming a commit it did + // not read is worse than one that says it could not tell. + return ""; + } +} + +/** + * whatWasTested is the repositories this run exercised, by the paths it was given. + * + * From the environment rather than a fixed list, because the paths are how the suite is told what + * to run — so anything it was pointed at is something the receipt should account for, and anything + * it was not pointed at was not tested. + */ +export function whatWasTested(env: NodeJS.ProcessEnv = process.env): Against { + const against: Against = {}; + for (const [name, directory] of Object.entries(repositories(env))) { + const head = headOf(directory); + if (head) against[name] = head; + } + return against; +} + +/** record writes the receipt. Failures are recorded too: a run that failed still ran. */ +export function record( + passed: number, + failed: number, + ran: string[], + env = process.env, + builtAgainst?: Against, +): Receipt { + const receipt: Receipt = { + at: new Date().toISOString(), + passed, + failed, + // What was BUILT, when the caller says — not what the repositories are at when the run ends. + // A receipt read at record time names whatever was committed during the twenty minutes the + // suite took, and it did: one run's receipt claimed a commit that landed mid-run and was + // never in the binaries. A verdict is only worth something attributed to one exact state. + against: builtAgainst ?? whatWasTested(env), + ran, + }; + const path = receiptPath(); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, JSON.stringify(receipt, null, 2) + "\n"); + return receipt; +} + +/** read returns the receipt, or null when this machine has never run the suite. */ +export function read(): Receipt | null { + try { + return JSON.parse(readFileSync(receiptPath(), "utf8")) as Receipt; + } catch { + return null; + } +} + +export interface Verdict { + /** True when the receipt still says something about the code as it stands. */ + current: boolean; + lines: string[]; +} + +/** + * judge says whether the last run still means anything. + * + * Two ways it can stop meaning something, and they read differently: it was long ago, or the code + * has moved since. The second is the one that matters — a suite that passed against code nobody + * runs any more is coverage in name. + */ +export function judge( + receipt: Receipt | null, + now: Date, + against: Against, + staleAfterDays = 7, +): Verdict { + if (!receipt) { + return { + current: false, + lines: [ + "this machine has never run the end-to-end suite.", + " Nothing here has been checked end to end, which is not the same as nothing being wrong.", + ], + }; + } + + // A receipt written before this field existed says nothing about what it ran, and the honest + // reading of "nothing said" is not "everything". + const endToEndRan = (receipt.ran ?? []).some((path) => path.endsWith(endToEnd)); + + const days = (now.getTime() - Date.parse(receipt.at)) / 86_400_000; + const lines: string[] = []; + const outcome = receipt.failed > 0 + ? `last ran ${ago(days)} and ${receipt.failed} test(s) failed` + : `last passed ${ago(days)}, ${receipt.passed} test(s)`; + lines.push(`the end-to-end suite ${outcome}`); + + let moved = false; + for (const name of Object.keys(against).sort()) { + const then = receipt.against[name]; + const now = against[name]; + if (!then) { + lines.push(` ${name.padEnd(14)} was not accounted for in that run`); + moved = true; + continue; + } + if (then === now) { + lines.push(` ${name.padEnd(14)} at ${then} (unchanged)`); + continue; + } + lines.push(` ${name.padEnd(14)} at ${then}, now at ${now}`); + moved = true; + } + + if (!endToEndRan) { + lines.push(` That run did not include ${endToEnd}, so it raised no machines.`); + } + + if (receipt.failed > 0) { + lines.push(" Nothing has been proven end to end since."); + } else if (moved) { + lines.push(" What it proved was proven about code that has since changed."); + } else if (days > staleAfterDays) { + lines.push(` Nothing has changed since, but that was more than ${staleAfterDays} days ago.`); + } + + return { + current: endToEndRan && receipt.failed === 0 && !moved && days <= staleAfterDays, + lines, + }; +} + +function ago(days: number): string { + if (days < 1 / 24) return "less than an hour ago"; + if (days < 1) return `${Math.round(days * 24)} hour(s) ago`; + return `${Math.round(days)} day(s) ago`; +} diff --git a/src/lifecycle/address.ts b/src/lifecycle/address.ts index 21bc5a3..6d6ed48 100644 --- a/src/lifecycle/address.ts +++ b/src/lifecycle/address.ts @@ -23,6 +23,15 @@ export interface Wire { mac: string; addresses: string[]; mtu: number | undefined; + /** + * Ask the host for an address rather than declaring one. + * + * **Only the uplink**, and it is the one address a scenario has no business choosing: it is on + * the machine's side of the host's own network, and what is routable there is the host's fact, + * not the declaration's. Every segment a scenario describes is still static, for the reason + * below. + */ + dhcp?: boolean; } /** @@ -31,6 +40,27 @@ export interface Wire { * the declaration is supposed to own. */ function networkUnit(wire: Wire): string { + if (wire.dhcp) { + return [ + "[Match]", + `MACAddress=${wire.mac}`, + "", + "[Network]", + "DHCP=ipv4", + "IPv6AcceptRA=no", + "", + "[DHCPv4]", + // UseDNS matters only where systemd-resolved runs; on a machine whose mesh modules own + // /etc/resolv.conf it is inert, and that inertness is load-bearing — the uplink must not + // outvote the resolver a scenario is testing. + // **Worse than any route the scenario states.** A machine behind a declared gateway must + // keep using it: the uplink is a way out of the scenario, not a better way around inside + // it. On-link segments win regardless, being connected routes; this only settles which + // default route is preferred when a scenario declares one of its own. + "RouteMetric=4096", + "UseDNS=yes", + ].join("\n") + "\n"; + } const lines = [ "[Match]", `MACAddress=${wire.mac}`, @@ -47,6 +77,36 @@ function networkUnit(wire: Wire): string { return lines.join("\n") + "\n"; } +/** + * Give one link a static address through systemd-networkd, and wait until networkd says it is + * configured. + * + * **Not `ip addr add`**, which is what this replaced and what cost a lab that could not finish. + * An address set by hand leaves the link `configuring` for ever, because networkd is still + * waiting to configure something it was never told about. `systemd-networkd-wait-online` then + * never returns — its timeout is `infinity` — so `network-online.target` is never reached, and + * **anything ordered after it never starts**. On these machines that is Docker, which meant + * `docker load` blocked on a socket whose daemon was queued behind a target that would never + * come. The lab stalled for thirty-five minutes with nothing to say. + * + * Every machine already did it this way. The registry did not, and it was the only one that + * needed Docker before anything else ran. + */ +export async function addressLink( + instanceName: string, + wire: Wire, + index = 0, +): Promise { + const unit = networkUnit(wire); + await incus( + ["exec", instanceName, "--", "sh", "-c", + `mkdir -p /etc/systemd/network && cat > /etc/systemd/network/10-mlab-${index}.network <<'MLAB'\n${unit}MLAB`], + 30_000, + ); + await incus(["exec", instanceName, "--", "systemctl", "enable", "--now", "systemd-networkd"], 60_000); + await incus(["exec", instanceName, "--", "systemctl", "restart", "systemd-networkd"], 60_000); +} + export async function applyAddresses( scenario: Scenario, instanceId: string, @@ -68,6 +128,16 @@ export async function applyAddresses( }); } + if (spec.egress) { + wires.push({ + device: `eth${spec.at.length}`, + mac: macFor(instanceId, machine, spec.at.length), + addresses: [], + mtu: undefined, + dhcp: true, + }); + } + for (const [index, wire] of wires.entries()) { const unit = networkUnit(wire); await incus( @@ -79,7 +149,7 @@ export async function applyAddresses( await incus(["exec", name, "--", "systemctl", "enable", "--now", "systemd-networkd"], 60_000); await incus(["exec", name, "--", "systemctl", "restart", "systemd-networkd"], 60_000); - log(` addressed ${machine} (${wires.map((w) => w.addresses.join(",")).join(" | ")})`); + log(` addressed ${machine} (${wires.map((w) => w.dhcp ? "uplink:dhcp" : w.addresses.join(",")).join(" | ")})`); } } diff --git a/src/lifecycle/base.ts b/src/lifecycle/base.ts new file mode 100644 index 0000000..b4f169d --- /dev/null +++ b/src/lifecycle/base.ts @@ -0,0 +1,218 @@ +/** + * Building the base image a scenario's machines are raised from. + * + * A sealed scenario cannot install a container runtime: its segments use documentation ranges + * and there is no route out (novox/hq ADR 0016). ADR 0006 records the consequence — *the lab + * needs a way to place images, and the machine it places them into needs a container runtime, + * which a sealed scenario cannot install either.* + * + * This is that, and it is research 012's reframing applied literally: **fetch at build time on + * a machine that has a network, apply on a target that then needs nothing.** The build happens + * here, once per lab, on a machine with a network. What a scenario raises afterwards needs + * neither. + * + * Measured while writing it: installing the runtime takes about 30 seconds, publishing about a + * minute, and the result is roughly 700 MiB. + */ + +import { incus, incusOk, succeeds } from "../incus/client.ts"; +import { BASE_IMAGE_ALIAS } from "./place.ts"; + +/** The stock image the base is built FROM. */ +export const UPSTREAM_IMAGE = "images:archlinux/current"; + +const BUILDER = "mesh-lab-base-builder"; + +export class BaseImageError extends Error { + constructor(message: string) { + super(message); + this.name = "BaseImageError"; + } +} + +/** + * Build the base image, replacing any previous one. + * + * Every step is read back. A published image that turns out not to have a working runtime is + * worse than no image, because every scenario raised from it fails somewhere else. + */ +export async function buildBaseImage( + log: (message: string) => void = () => {}, +): Promise<{ alias: string; runtime: string }> { + await succeeds(["delete", "-f", BUILDER], 120_000); + + log(` launching ${BUILDER} from ${UPSTREAM_IMAGE}, with a network`); + await incus([ + "launch", UPSTREAM_IMAGE, BUILDER, "--vm", + "-c", "security.secureboot=false", + "-c", "limits.memory=2GiB", + "-c", "limits.cpu=2", + ], 300_000); + + try { + await waitForAgent(BUILDER); + + log(" installing a container runtime"); + await incus(["exec", BUILDER, "--", "pacman", "-Sy", "--noconfirm", "docker"], 600_000); + + // And the tools for the private network, for the same reason as the runtime: a sealed + // scenario cannot install them, so a lab that omits them cannot test connectivity at all — + // which is most of what the mesh does between machines. + // + // Installed here and NOT started. What a node runs is the mesh's decision, delivered as a + // declaration; a lab that brought the interface up itself would be testing its own setup + // rather than the mesh's. + log(" installing the tools for the private network"); + await incus(["exec", BUILDER, "--", "pacman", "-S", "--noconfirm", "wireguard-tools"], 600_000); + + // And git, for the same reason again: a machine that builds modules clones them, and a sealed + // scenario cannot install it. On a real build machine the mesh installs it as a package like + // anything else — the lab is the special case, because its machines reach no mirror. + log(" installing git, so a machine can build modules"); + await incus(["exec", BUILDER, "--", "pacman", "-S", "--noconfirm", "git"], 600_000); + + // And nftables, because the mesh computes a machine's filtering and delivers it as a file + // that a service reflects — and neither the file nor the service can install what loads it. + // Installed and NOT enabled: whether a machine filters is the mesh's decision, and a lab that + // turned it on itself would be testing its own setup. + log(" installing nftables, so a machine can enforce what the mesh computed"); + await incus(["exec", BUILDER, "--", "pacman", "-S", "--noconfirm", "nftables"], 600_000); + + // And dnsmasq, because a service is named under the machine it runs on — postgres.novox.internal + // — and only a resolver can answer a name the mesh was never told about. Installed and NOT + // started: whether a machine resolves for the mesh is the mesh's decision, and a lab that + // turned it on itself would be testing its own setup. + log(" installing dnsmasq, so a machine can answer names under another machine"); + await incus(["exec", BUILDER, "--", "pacman", "-S", "--noconfirm", "dnsmasq"], 600_000); + + // Trust the documentation ranges as plain-HTTP registries. + // + // A scenario's registry is scenery inside the scenario, serving over HTTP, and a runtime + // will not pull from one without being told. Scoped to RFC 5737 and RFC 3849 ranges rather + // than a specific address, because those never route on the real internet — so this cannot + // make a real machine trust a real registry, whatever it is copied onto. + await incus([ + "exec", BUILDER, "--", "sh", "-c", + `mkdir -p /etc/docker && printf '%s' '${JSON.stringify({ + "insecure-registries": ["192.0.2.0/24", "198.51.100.0/24", "203.0.113.0/24"], + })}' > /etc/docker/daemon.json`, + ], 60_000); + await incus(["exec", BUILDER, "--", "systemctl", "enable", "docker"], 60_000); + await incus(["exec", BUILDER, "--", "systemctl", "start", "docker"], 120_000); + + // Read back from the tool, not from the package manager (novox/hq 04-ISSUES/007). + const wg = await incusOk(["exec", BUILDER, "--", "wg", "--version"], 60_000); + if (!wg?.trim()) { + throw new BaseImageError( + `wireguard-tools was installed in ${BUILDER} and \`wg\` does not answer. Publishing ` + + `this would give every scenario a machine that cannot join a private network.`, + ); + } + log(` ${wg.trim()}`); + + // The same, for git. An installed package is not a capability, and this is the one place to + // catch it — a build machine whose clone fails does so three minutes into a scenario, with + // the failure reported as a build problem rather than a lab one. + const git = await incusOk(["exec", BUILDER, "--", "git", "--version"], 60_000); + if (!git?.trim()) { + throw new BaseImageError( + `git was installed in ${BUILDER} and \`git --version\` does not answer. Publishing ` + + `this would give every scenario a machine that cannot build a module.`, + ); + } + log(` ${git.trim()}`); + + // The same again, for nftables. A machine that cannot load a rule set applies the mesh's + // filtering, reports success, and filters nothing — which is precisely the fault the whole + // derivation exists to remove, reappearing in the lab. + const nft = await incusOk(["exec", BUILDER, "--", "nft", "--version"], 60_000); + if (!nft?.trim()) { + throw new BaseImageError( + `nftables was installed in ${BUILDER} and \`nft\` does not answer. Publishing this would ` + + `give every scenario a machine that cannot enforce what the mesh computed for it.`, + ); + } + log(` ${nft.trim()}`); + + // The same again for dnsmasq. A machine that cannot answer names applies the mesh's resolver + // data, reports success, and resolves nothing — the shape of fault this lab exists to catch. + const dns = await incusOk(["exec", BUILDER, "--", "dnsmasq", "--version"], 60_000); + if (!dns?.trim()) { + throw new BaseImageError( + `dnsmasq was installed in ${BUILDER} and does not answer. Publishing this would give ` + + `every scenario a machine that cannot resolve a name under another machine.`, + ); + } + log(` ${dns.trim().split("\n")[0]}`); + + // Read back from the runtime, not from the package manager. An installed package is not a + // capability (novox/hq 04-ISSUES/007), and this is the one place to catch that — after + // publishing, every scenario pays for it instead. + const runtime = (await incusOk( + ["exec", BUILDER, "--", "docker", "info", "--format", "{{.ServerVersion}}"], + 120_000, + ))?.trim(); + if (!runtime) { + throw new BaseImageError( + `the runtime was installed in ${BUILDER} and does not answer. Publishing this would ` + + `give every scenario an image that looks right and is not.`, + ); + } + log(` runtime works (docker ${runtime})`); + + // Read back that the runtime will actually pull over plain HTTP from a documentation + // range. Writing the file is not the same as the daemon honouring it, and a base image + // that looks right here fails much later — in a sealed scenario, as a container that + // cannot fetch its image, which is a long way from the cause. + const trusted = await incusOk( + ["exec", BUILDER, "--", "docker", "info", "--format", "{{.RegistryConfig.InsecureRegistryCIDRs}}"], + 60_000, + ); + if (!trusted?.includes("192.0.2.0/24")) { + throw new BaseImageError( + `the runtime in ${BUILDER} does not trust the documentation ranges as plain-HTTP ` + + `registries. It reported: ${trusted?.trim() || "nothing"}\n` + + ` Every scenario raised from this image would fail to pull from its own registry.`, + ); + } + log(" trusts the documentation ranges as registries"); + + log(" publishing"); + await incus(["stop", BUILDER, "--timeout", "120"], 300_000); + await incus(["publish", BUILDER, "--alias", BASE_IMAGE_ALIAS, "--reuse"], 900_000); + + const listed = await incusOk(["image", "list", BASE_IMAGE_ALIAS, "--format", "csv", "-c", "l"], 60_000); + if (!listed?.includes(BASE_IMAGE_ALIAS)) { + throw new BaseImageError( + `publishing reported success and '${BASE_IMAGE_ALIAS}' is not in the image list.`, + ); + } + + log(` published ${BASE_IMAGE_ALIAS}`); + return { alias: BASE_IMAGE_ALIAS, runtime }; + } finally { + // The builder is scaffolding. Leaving it standing would be a machine with a network in a + // lab whose whole point is that scenarios do not have one. + await succeeds(["delete", "-f", BUILDER], 120_000); + } +} + +/** Whether the base image exists, for a scenario to check before it raises. */ +export async function baseImageExists(): Promise { + const listed = await incusOk( + ["image", "list", BASE_IMAGE_ALIAS, "--format", "csv", "-c", "l"], 30_000, + ); + return Boolean(listed?.includes(BASE_IMAGE_ALIAS)); +} + +/** + * Wait for the guest agent, because `launch` returning means the VM started, not that anything + * inside it will answer. + */ +async function waitForAgent(name: string): Promise { + for (let i = 0; i < 90; i++) { + if (await succeeds(["exec", name, "--", "true"], 10_000)) return; + await new Promise((r) => setTimeout(r, 2_000)); + } + throw new BaseImageError(`${name} started and its agent never answered.`); +} diff --git a/src/lifecycle/connect.ts b/src/lifecycle/connect.ts new file mode 100644 index 0000000..e27d616 --- /dev/null +++ b/src/lifecycle/connect.ts @@ -0,0 +1,196 @@ +/** + * Letting the workstation reach one scenario by name, on purpose and temporarily. + * + * **A scenario is a closed address space** ([ADR 0016](../../02-DECISIONS/0016-the-lab.md)): two + * raised from the same declaration hold the same addresses and never meet, because nothing joins + * their links. That is what lets two identical scenarios run at once, and it is why the lab talks + * to machines through the hypervisor's own channel rather than over IP. + * + * Reaching in from the workstation breaks that, so it is **opt-in, one scenario at a time, and + * reversible**. It refuses when more than one is standing rather than guessing which was meant — + * the failure it exists to avoid is not an error but one scenario's traffic arriving in another. + * + * **Names resolve to the segment address, not the overlay one.** Inside the mesh a name answers + * with a machine's private-network address; from here that would need the workstation on the + * overlay, which is a much larger door to open. The segment address reaches the same machine and + * the same ports, which is what somebody opening a board in a browser actually needs. + */ + +import { writeFileSync, unlinkSync, existsSync, readFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; + +import { incus, incusOk, taggedInstances, taggedNetworks } from "../incus/client.ts"; +import { log } from "../log.ts"; + +/** Where the rule goes. Its own file — the one beside it belongs to something else. */ +export const RULE = "/etc/dnsmasq.d/mesh-lab.conf"; + +export interface Reached { + instanceId: string; + bridge: string; + /** The address the workstation took on that link. */ + address: string; + /** Machine name to the address its names now answer with. */ + machines: Record; +} + +export class ConnectError extends Error {} + +/** Run something as root, and say plainly when that is what failed. */ +function asRoot(argv: string[]): { ok: boolean; said: string } { + const ran = spawnSync("sudo", ["-n", ...argv], { encoding: "utf8" }); + const said = `${ran.stdout ?? ""}${ran.stderr ?? ""}`.trim(); + if (ran.status !== 0 && /password|not allowed|no tty/i.test(said)) { + throw new ConnectError( + `this needs root and sudo asked for a password, which there is nowhere to type here.\n` + + ` Run it yourself: sudo ${argv.join(" ")}`); + } + return { ok: ran.status === 0, said }; +} + +/** The one instance standing, or a refusal naming what it found instead. */ +export async function theOnlyInstance(): Promise { + const ids = [...new Set((await taggedInstances()).map((i) => i.instanceId))].sort(); + if (ids.length === 1) return ids[0]!; + if (ids.length === 0) { + throw new ConnectError("no scenario is standing, so there is nothing to reach."); + } + throw new ConnectError( + `${ids.length} scenarios are standing and they may hold the same addresses, so there is no ` + + `answer to which one you meant: ${ids.join(", ")}.\n` + + ` Take the others down, or name one — but only one can be reachable at a time.`); +} + +/** Every machine in an instance, with the address it has on the given link. */ +async function addressesOn(instanceId: string, segment: string): Promise> { + const out: Record = {}; + for (const machine of (await taggedInstances()).filter((i) => i.instanceId === instanceId)) { + const said = await incusOk( + ["exec", machine.name, "--", "sh", "-c", + `ip -4 -o addr show | awk '{print $4}' | cut -d/ -f1`], 30_000); + for (const address of (said ?? "").split("\n").map((l) => l.trim()).filter(Boolean)) { + if (address.startsWith("127.")) continue; + // The first non-loopback address on the segment. A machine on two links has the one that + // matches this segment's range, which is what the caller asked about. + if (!out[machine.machine]) out[machine.machine] = address; + } + } + void segment; + return out; +} + +/** Names this workstation already answers for, so a scenario cannot quietly shadow one. */ +function alreadyServed(): string[] { + const beside = "/etc/dnsmasq.d/hal-dns.conf"; + if (!existsSync(beside)) return []; + return [...readFileSync(beside, "utf8").matchAll(/^address=\/([^/]+)\//gm)].map((m) => m[1]!); +} + +export async function connect(instanceId?: string): Promise { + const id = instanceId ?? (await theOnlyInstance()); + + const link = (await taggedNetworks()).find( + (n) => n.instanceId === id && n.kind === "public" && n.cidr.some((c) => !c.includes(":"))); + if (!link) { + throw new ConnectError( + `${id} has no public IPv4 segment, so there is nothing for this workstation to join.`); + } + const range = link.cidr.find((c) => !c.includes(":"))!; + const prefix = range.slice(range.lastIndexOf("/") + 1); + + const machines = await addressesOn(id, link.segment); + if (Object.keys(machines).length === 0) { + throw new ConnectError(`no machine in ${id} has an address yet — is it still coming up?`); + } + + // **Refused rather than shadowed.** This workstation already answers for the mesh it really + // runs; a scenario machine sharing one of those names would silently take it over, and the + // damage would land on the real thing rather than the lab. + const clash = Object.keys(machines) + .map((m) => `${m}.internal`) + .filter((n) => alreadyServed().includes(n)); + if (clash.length > 0) { + throw new ConnectError( + `${clash.join(", ")} is already answered on this workstation for something real. ` + + `Connecting would point it at the lab instead, which is the wrong thing to break.`); + } + + // An address on the link, high in the range so it does not meet what a scenario declares. + const base = Object.values(machines)[0]!.split(".").slice(0, 3).join("."); + const mine = `${base}.254`; + if (Object.values(machines).includes(mine)) { + throw new ConnectError(`${mine} is taken by a machine, and that is the address this uses.`); + } + + const added = asRoot(["ip", "addr", "add", `${mine}/${prefix}`, "dev", link.name]); + if (!added.ok && !/File exists/i.test(added.said)) { + throw new ConnectError(`could not take an address on ${link.name}: ${added.said}`); + } + + // Everything under a machine's name, answered with that machine. The same shape the mesh's own + // resolver writes, because it is answering the same question. + const rule = [ + "# Written by mesh-lab connect. Removed by mesh-lab disconnect.", + "# One scenario at a time: these names are only true while that scenario is standing.", + ...Object.entries(machines).sort() + .map(([machine, address]) => `address=/${machine}.internal/${address}`), + "", + ].join("\n"); + writeFileSync("/tmp/mesh-lab-dns.conf", rule); + const placed = asRoot(["cp", "/tmp/mesh-lab-dns.conf", RULE]); + if (!placed.ok) throw new ConnectError(`could not write ${RULE}: ${placed.said}`); + // **Restart, not reload.** A reload is SIGHUP, and dnsmasq answers that by re-reading its hosts + // file and clearing its cache — not its configuration. The rule was written, the reload + // reported success, and nothing resolved. Measured: the daemon's start time was nine days old + // after a "successful" reload. + // + // It costs a moment of no name resolution on this workstation, which is the honest price and is + // paid again by disconnect. + const reloaded = asRoot(["systemctl", "restart", "dnsmasq"]); + if (!reloaded.ok) throw new ConnectError(`could not restart dnsmasq: ${reloaded.said}`); + + log.info(`connected to ${id} as ${mine} on ${link.name}`); + return { instanceId: id, bridge: link.name, address: mine, machines }; +} + +export async function disconnect(): Promise { + const undone: string[] = []; + + if (existsSync(RULE)) { + const removed = asRoot(["rm", "-f", RULE]); + if (!removed.ok) throw new ConnectError(`could not remove ${RULE}: ${removed.said}`); + asRoot(["systemctl", "restart", "dnsmasq"]); + undone.push(`removed ${RULE} and reloaded dnsmasq`); + } + + // Any address this took, on any lab link still present. Done by looking rather than by + // remembering: a workstation that was rebooted, or a scenario destroyed under it, must still + // be able to tidy up. + for (const link of await taggedNetworks()) { + const shown = await incusOk(["network", "info", link.name], 15_000); + if (shown === null) continue; + const ran = spawnSync("ip", ["-4", "-o", "addr", "show", "dev", link.name], { encoding: "utf8" }); + for (const line of (ran.stdout ?? "").split("\n")) { + const found = line.match(/inet (\d+\.\d+\.\d+\.254\/\d+)/); + if (!found) continue; + const dropped = asRoot(["ip", "addr", "del", found[1]!, "dev", link.name]); + if (dropped.ok) undone.push(`gave up ${found[1]} on ${link.name}`); + } + } + + if (undone.length === 0) undone.push("nothing was connected"); + return undone; +} + +/** What is connected now, for a person who cannot remember. */ +export async function connection(): Promise { + if (!existsSync(RULE)) return []; + return readFileSync(RULE, "utf8").split("\n") + .filter((l) => l.startsWith("address=/")) + .map((l) => { + const [, name, address] = l.match(/^address=\/([^/]+)\/(.+)$/) ?? []; + return `${name} → ${address}`; + }); +} + +void incus; diff --git a/src/lifecycle/operate.ts b/src/lifecycle/operate.ts index f2f2180..9dc8165 100644 --- a/src/lifecycle/operate.ts +++ b/src/lifecycle/operate.ts @@ -54,12 +54,18 @@ export async function exec( instanceId: string, machine: string, command: string[], + /** + * How long to wait. Two minutes suits a command; a build, or anything that waits on another + * machine, needs longer — and a caller that cannot say so has to split the work up to fit, + * which is a test shaped by its harness rather than by what it is testing. + */ + timeoutMs = 120_000, ): Promise<{ stdout: string; stderr: string }> { const found = (await taggedInstances()).find( (i) => i.instanceId === instanceId && i.machine === machine, ); const name = found?.name ?? machineName(instanceId, machine); - return incus(["exec", name, "--", ...command], 120_000); + return incus(["exec", name, "--", ...command], timeoutMs); } /** diff --git a/src/lifecycle/place.ts b/src/lifecycle/place.ts index 7d5d321..eb245a6 100644 --- a/src/lifecycle/place.ts +++ b/src/lifecycle/place.ts @@ -12,12 +12,32 @@ */ import type { Scenario } from "../declaration/types.ts"; -import { incus, incusOk } from "../incus/client.ts"; +import { spawn } from "node:child_process"; +import { unlink } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; -/** What this stage can put inside a machine. */ -export const PLACEABLE = ["host"] as const; +import { incus, incusOk, succeeds } from "../incus/client.ts"; +import { around, log, shorten } from "../log.ts"; + +/** + * What this stage can put inside a machine. + * + * `image:` is the one that takes an argument — `image:alpine@sha256:...` — because + * WHICH image is the point of placing one. + */ +export const PLACEABLE = ["host", "runtime"] as const; export type Placeable = (typeof PLACEABLE)[number]; +/** The prefix that marks a container image, and what follows it is passed through unchanged. */ +export const IMAGE_PREFIX = "image:"; + +/** Whether an artifact names something this stage can place. */ +export function isPlaceable(artifact: string): boolean { + return (PLACEABLE as readonly string[]).includes(artifact) || + (artifact.startsWith(IMAGE_PREFIX) && artifact.slice(IMAGE_PREFIX.length).trim() !== ""); +} + /** Where the host binary lives on a machine once placed. */ export const HOST_PATH = "/usr/local/bin/mesh-host"; @@ -78,7 +98,7 @@ export interface PlacedHost { * Put the host on a machine and ask it what the machine is. * * The result is read back from the running binary, never assumed from the fact that the copy - * succeeded (novox/hq ADR 0035). A file arriving is not a host working, which is the same + * succeeded (novox/hq ADR 0018). A file arriving is not a host working, which is the same * distinction the host itself makes about installed packages. */ export async function placeHost( @@ -134,8 +154,12 @@ export async function applyPlacements( const placements = planPlacements(scenario); if (placements.length === 0) return []; + // Only demanded when something actually needs it. A scenario placing a runtime and an image + // and no host is a legitimate thing to want, and asking it for a host binary would be + // refusing a scenario for not doing something it never said it would. + const wantsHost = placements.some((p) => p.artifacts.includes("host")); const binary = hostBinaryPath(); - if (!binary) { + if (wantsHost && !binary) { throw new Error( `this scenario places the host, and no host binary was given. Set ` + `MESH_LAB_HOST_BINARY to a built mesh-host. Guessing at a path would place ` + @@ -147,10 +171,229 @@ export async function applyPlacements( for (const { machine, artifacts } of placements) { const name = machineNames.get(machine); if (!name) continue; + + // Order within a machine is the order declared, because a scenario placing an image + // before a runtime means something different from the reverse and the lab should not + // silently reorder it into working. for (const artifact of artifacts) { - if (artifact !== "host") continue; // refused earlier; belt and braces - placed.push(await placeHost(name, machine, binary, log)); + if (artifact === "host") { + placed.push(await placeHost(name, machine, binary as string, log)); + } else if (artifact === "runtime") { + await placeRuntime(name, machine, log); + } else if (artifact.startsWith(IMAGE_PREFIX)) { + await placeImage(name, machine, artifact.slice(IMAGE_PREFIX.length).trim(), log); + } + // Anything else was refused by the validator; reaching here would be a validator bug. } } return placed; } + +/** + * The base image a scenario's machines are built from, when they need a container runtime. + * + * A sealed scenario cannot install one: its segments use documentation ranges and there is no + * route out (novox/hq ADR 0016). So the runtime arrives the way research 012 says everything + * awkward should — **fetched at build time on a machine that has a network, applied on a target + * that then needs nothing.** Building this image is that build time. + * + * Measured before this was written: an Arch VM installs docker in about 30 seconds, and + * publishing the result takes about a minute and yields roughly 700 MiB. That is a one-time + * cost paid once per lab, not per scenario. + */ +export const BASE_IMAGE_ALIAS = "mesh-lab/base"; + +/** How to build it, said in one place so an error can point at it. */ +export const BASE_IMAGE_HOWTO = + `Build it with: mesh-lab base build\n` + + ` It launches a machine WITH a network, installs a container runtime, and publishes the\n` + + ` result as '${BASE_IMAGE_ALIAS}'. A scenario is sealed and cannot do that for itself.`; + +/** + * Confirm a machine has a working container runtime. + * + * Asks the runtime, never the filesystem. A binary being present is not a runtime working — + * that is 04-ISSUES/007, and it is the reason this runs `info` rather than checking a path. + * + * It does not INSTALL one. A sealed machine cannot fetch, so an absent runtime is a scenario + * built on the wrong image, and saying that is more useful than failing inside a package + * manager with no network. + */ +export async function placeRuntime( + instanceName: string, + machine: string, + log: (message: string) => void = () => {}, +): Promise { + const version = (await incusOk( + ["exec", instanceName, "--", "docker", "info", "--format", "{{.ServerVersion}}"], + 60_000, + ))?.trim(); + + if (!version) { + throw new PlacementError( + machine, + `${machine} has no working container runtime, and a sealed scenario cannot install one.\n` + + ` This machine needs to be built from '${BASE_IMAGE_ALIAS}'.\n` + + BASE_IMAGE_HOWTO, + ); + } + log(` ${machine} has a container runtime (docker ${version})`); + return version; +} + +/** + * Put a container image inside a machine. + * + * Exported from this workstation and loaded in the machine, because the machine cannot reach a + * registry. The reference is passed through unchanged — including its digest — so what runs in + * the lab is the image the declaration names rather than whatever a tag pointed at. + */ +export async function placeImage( + instanceName: string, + machine: string, + reference: string, + log: (message: string) => void = () => {}, +): Promise { + // A digest reference cannot be placed this way, and finding that out here is much better + // than finding it out when a container fails to start. + // + // `docker save alpine@sha256:...` produces an archive with NO repo tag — only an image ID — + // because a repo digest exists only for an image a registry served. Loading it gives a + // dangling image, so `docker run ` falls through to the registry, which a + // sealed machine cannot reach. Measured, not assumed: the load says `Loaded image ID:` + // instead of `Loaded image:`, and `docker images` then lists nothing. + // + // This collides with novox/hq ADR 0006, which pins bundle images BY DIGEST and has the host + // refuse anything else. Reconciling the two needs a registry inside the scenario, which is + // real design work — see 04-ISSUES/009. + if (reference.includes("@sha256:")) { + throw new PlacementError( + machine, + `${reference} is pinned by digest, and an image placed from an archive cannot keep its ` + + `digest — a repo digest only exists for an image a registry served.\n` + + ` Placing it would load an image with no name, and a container declaring that digest ` + + `would try to reach a registry the machine cannot see.\n` + + ` Place it by tag, or give the scenario a registry (novox/hq 04-ISSUES/009).`, + ); + } + + const tar = join(tmpdir(), `mesh-lab-image-${process.pid}-${Date.now()}.tar`); + + // Exported from whatever this workstation has. The lab places images; it does not fetch + // them, so an image nobody pulled here is an error rather than a download. + const saved = await local("docker", ["save", reference, "-o", tar], 600_000); + if (!saved.ok) { + throw new PlacementError( + machine, + `cannot export ${reference} from this workstation: ${saved.stderr.trim()}\n` + + ` The lab places images it already has, and never fetches on a scenario's behalf.\n` + + ` Pull it here first, then raise.`, + ); + } + + try { + await waitForRuntime(instanceName, machine); + await incus(["file", "push", tar, `${instanceName}/tmp/image.tar`], 900_000); + + // Read back what the runtime says, not that the push returned. A file arriving is not an + // image present, and `docker load` names what it loaded — so that is what is checked. + const loaded = await incusOk( + ["exec", instanceName, "--", "docker", "load", "-i", "/tmp/image.tar"], + 600_000, + ); + // `Loaded image:` and `Loaded image ID:` are different outcomes and only one is useful. + // Matching the shorter string accepted both, so an archive that loaded as a dangling + // image reported success — a check that passes on the wrong thing. + if (!loaded?.includes("Loaded image:")) { + throw new PlacementError( + machine, + `${reference} was pushed to ${machine} and did not arrive as a usable image.\n` + + ` The runtime said: ${loaded?.trim() || "nothing"}\n` + + ` 'Loaded image ID:' means it loaded with no name, which nothing can run by name.`, + ); + } + log(` placed ${reference} on ${machine}`); + } finally { + await unlink(tar).catch(() => {}); + } +} + +/** Run something on THIS workstation. The incus client only speaks to incus. */ +function local( + command: string, + args: string[], + timeoutMs: number, +): Promise<{ ok: boolean; stdout: string; stderr: string }> { + // The third and last place the lab runs an external program (novox/hq 04-ISSUES/024). + // `docker save` of a large image is the slowest single thing a raise does. + return around(`${command} ${shorten(args)}`, () => runLocal(command, args, timeoutMs), { + heartbeatMs: 15_000, + }); +} + +function runLocal( + command: string, + args: string[], + timeoutMs: number, +): Promise<{ ok: boolean; stdout: string; stderr: string }> { + return new Promise((resolve) => { + const child = spawn(command, args, { stdio: ["ignore", "pipe", "pipe"] }); + let stdout = ""; + let stderr = ""; + const timer = setTimeout(() => child.kill("SIGKILL"), timeoutMs); + child.stdout.on("data", (d) => (stdout += d)); + child.stderr.on("data", (d) => (stderr += d)); + child.on("error", (err) => { + clearTimeout(timer); + resolve({ ok: false, stdout, stderr: err.message }); + }); + child.on("close", (code) => { + clearTimeout(timer); + if (code !== 0) { + log.debug(` exit ${code}: ${shorten([stderr.trim() || "(nothing on stderr)"], 400)}`); + } + resolve({ ok: code === 0, stdout, stderr }); + }); + }); +} + +/** + * Wait until the machine's container runtime will answer, and say why if it will not. + * + * **A stall must become a failure with a reason.** `docker load` against a daemon that is not + * running does not fail — the socket exists and is socket-activated, so the client blocks while + * systemd tries to start a service that may be queued behind something. That turned a + * misconfigured link into a thirty-five minute silence and then a timeout naming the wrong + * thing entirely (novox/hq 04-ISSUES/024). + * + * The one cause seen in practice is named in the message, because it is not guessable from + * "docker did not start": these machines have no DHCP by design, so a link that networkd was + * never told how to configure leaves `network-online.target` unreachable for ever, and Docker + * is ordered after it. + */ +async function waitForRuntime(instanceName: string, machine: string): Promise { + const deadline = Date.now() + 120_000; + while (Date.now() < deadline) { + if (await succeeds(["exec", instanceName, "--", "docker", "info"], 20_000)) return; + await new Promise((r) => setTimeout(r, 2_000)); + } + + // What it was waiting for, asked once, so the report names the cause rather than the symptom. + const jobs = (await incusOk( + ["exec", instanceName, "--", "systemctl", "list-jobs", "--no-pager"], 20_000, + )) ?? ""; + const blocked = jobs.includes("network-online.target") || jobs.includes("wait-online"); + + throw new PlacementError( + machine, + `the container runtime on ${machine} did not answer within 120s, so an image cannot be ` + + `placed into it.\n` + + (blocked + ? ` Docker is queued behind network-online.target, which is waiting for a link that ` + + `systemd-networkd was never told how to configure. These machines have no DHCP by ` + + `design, so that wait never ends — give the link a .network unit rather than an ` + + `address set by hand.\n` + : "") + + ` systemd is waiting on:\n${jobs.trim() || " (it said nothing)"}`, + ); +} diff --git a/src/lifecycle/raise.ts b/src/lifecycle/raise.ts index 5f8d517..9ebc0ee 100644 --- a/src/lifecycle/raise.ts +++ b/src/lifecycle/raise.ts @@ -22,7 +22,10 @@ import { applyAddresses, applyDefaultRoutes } from "./address.ts"; import { assertSupported } from "./supported.ts"; import { planRouters, raiseRouters, raiseTransit } from "./router.ts"; import { applyHostFirewalls } from "./firewall.ts"; -import { applyPlacements } from "./place.ts"; +import { IMAGE_PREFIX, BASE_IMAGE_ALIAS, BASE_IMAGE_HOWTO, planPlacements, applyPlacements } from "./place.ts"; +import { baseImageExists, UPSTREAM_IMAGE } from "./base.ts"; +import { discardStock, raiseRegistry, stockRegistry } from "./registry.ts"; +import { log as record } from "../log.ts"; /** Drivers whose snapshots are copy-on-write. On `dir` a snapshot is a full copy. */ const COW_DRIVERS = ["btrfs", "zfs"]; @@ -43,6 +46,13 @@ export interface RaisedScenario { machines: string[]; networks: string[]; pool: string; + /** + * Images the scenario's registry serves, as references a declaration can pin. + * + * Reported rather than declared, because the digest is the one this registry assigned and + * is not knowable before it was raised. + */ + images: string[]; } export class RaiseError extends Error { @@ -121,12 +131,42 @@ async function createNetwork( return name; } +/** + * The one network the lab supplies rather than the declaration. + * + * Every segment a scenario describes is an isolated bridge with no addresses, no DHCP and no NAT, + * because the declaration owns addressing. This is the opposite of that on purpose: it is not part + * of the scenario, it carries no scenario traffic, and what is routable on it is the host's fact. + * + * It exists so a machine can fetch what it starts from. A first node pulls three images before + * there is any mesh, and the module that gives a mesh its own store pulls one more + * (novox/hq 04-ISSUES/029) — none of which anything inside a scenario can serve. + * + * Tagged like everything else, so tearing the scenario down takes it too. + */ +async function createUplink(instanceId: string): Promise { + const name = networkName(instanceId, "uplink"); + if (await succeeds(["network", "show", name], 15_000)) return name; + await incus([ + "network", "create", name, + "ipv4.address=auto", + "ipv4.nat=true", + "ipv6.address=none", + `user.mesh-lab.instance=${instanceId}`, + "user.mesh-lab.segment=uplink", + ]); + return name; +} + async function createMachine( instanceId: string, machine: string, attachments: { segment: string }[], image: string, pool: string, + egress = false, + memory = "1GiB", + cpus = 2, ): Promise { const name = machineName(instanceId, machine); if (await succeeds(["config", "show", name], 15_000)) return name; @@ -138,8 +178,8 @@ async function createMachine( // Arch images refuse to boot under secureboot with the shipped keys. Discovered by // the first launch failing with exactly that message. "-c", "security.secureboot=false", - "-c", "limits.memory=1GiB", - "-c", "limits.cpu=2", + "-c", `limits.memory=${memory}`, + "-c", `limits.cpu=${cpus}`, "-c", `user.mesh-lab.instance=${instanceId}`, "-c", `user.mesh-lab.machine=${machine}`, ]; @@ -158,6 +198,17 @@ async function createMachine( `hwaddr=${macFor(instanceId, machine, index)}`, ]); } + // After every declared attachment, so eth0..ethN keep meaning what the scenario said and the + // uplink is whatever comes next. A machine that never asked for one has no such interface at + // all, which is the difference between a closed scenario and an open one. + if (egress) { + await incus([ + "config", "device", "add", name, `eth${attachments.length}`, "nic", + "nictype=bridged", + `parent=${await createUplink(instanceId)}`, + `hwaddr=${macFor(instanceId, machine, attachments.length)}`, + ]); + } return name; } @@ -165,8 +216,34 @@ export async function raise( scenario: Scenario, options: RaiseOptions = {}, ): Promise { - const log = options.onProgress ?? (() => {}); - const image = options.image ?? "images:archlinux/current"; + // **Progress always reaches the file, whether or not anyone asked to see it.** + // + // This used to be the caller's callback or nothing, and every function below takes its `log` + // from here — so a caller that passed none silenced the whole lifecycle. That is exactly what + // happened: the end-to-end test called `raise` with no callback, so the one run that mattered + // reported not a single step (novox/hq 04-ISSUES/024). + // + // Teeing rather than replacing: the caller still gets what it asked for, and the record is kept + // regardless. A record nobody switched on is the one you want after the thing goes wrong. + const log = (message: string): void => { + record.info(message); + options.onProgress?.(message); + }; + + // A scenario that places a runtime or an image needs machines built from the base image, + // because a sealed machine cannot install one (novox/hq ADR 0006). Chosen here rather than + // declared, so a scenario says WHAT it needs and not which image provides it. + const needsRuntime = (scenario.images ?? []).length > 0 || + planPlacements(scenario).some(({ artifacts }) => + artifacts.some((a) => a === "runtime" || a.startsWith(IMAGE_PREFIX)) + ); + if (needsRuntime && !options.image && !(await baseImageExists())) { + throw new Error( + `this scenario needs a container runtime inside its machines, and '${BASE_IMAGE_ALIAS}' ` + + `does not exist.\n${BASE_IMAGE_HOWTO}`, + ); + } + const image = options.image ?? (needsRuntime ? BASE_IMAGE_ALIAS : UPSTREAM_IMAGE); const readyTimeout = options.readyTimeoutSeconds ?? 180; const instanceId = options.instanceId ?? newInstanceId(scenario.scenario, new Date()); @@ -175,65 +252,100 @@ export async function raise( // has been created, and there is no wreckage to leave standing. assertSupported(scenario); - let step = "choosing a storage pool"; + // **The step is the log.** Setting it and recording it are one act, so a step added later + // cannot be a step that goes unrecorded — which is the drift that made a thirty-five minute + // stall untraceable (novox/hq 04-ISSUES/024). Each entry closes the previous one with its + // duration, so the log says where a raise spends its time as well as where it stopped. + let step = ""; + let stepFrom = Date.now(); + const enter = (next: string): string => { + if (step) record.info(` ${step} — ${((Date.now() - stepFrom) / 1000).toFixed(1)}s`); + record.info(`▶ ${next}`); + stepFrom = Date.now(); + step = next; + return next; + }; + + enter("choosing a storage pool"); try { const pool = await choosePool(log); log(`instance ${instanceId} pool ${pool}`); - step = "creating segments"; + enter("creating segments"); const networks: string[] = []; for (const [segment, spec] of Object.entries(scenario.segments)) { networks.push(await createNetwork(instanceId, segment, spec)); log(` segment ${segment}`); } - step = "creating machines"; + enter("creating machines"); const created: string[] = []; const byMachine = new Map(); for (const [machine, spec] of Object.entries(scenario.machines)) { const attachments = spec.at === "detached" ? [] : spec.at; - const name = await createMachine(instanceId, machine, attachments, image, pool); + const name = await createMachine( + instanceId, machine, attachments, image, pool, + spec.at !== "detached" && spec.egress === true, spec.memory, spec.cpus); created.push(name); byMachine.set(machine, name); log(` machine ${machine}${spec.at === "detached" ? " (detached)" : ""}`); } - step = "starting machines"; + enter("starting machines"); for (const name of created) { await succeeds(["start", name], 60_000); } - step = "waiting for machines to become usable"; + enter("waiting for machines to become usable"); await waitUntilAllUsable(created, readyTimeout, log); - step = "applying declared addresses"; + enter("applying declared addresses"); await applyAddresses(scenario, instanceId, byMachine, log); // Transit first: a gateway's default route points at it, so it has to exist. - step = "wiring the public networks together"; + enter("wiring the public networks together"); const transit = await raiseTransit(scenario, instanceId, log); - step = "raising routers"; + enter("raising routers"); const routers = await raiseRouters(scenario, instanceId, planRouters(scenario, instanceId), log); if (transit) routers.push(transit); - step = "routing machines through their gateways"; + // Stocked on this workstation, where there is a network, and served from inside the + // scenario, where there is not (novox/hq 04-ISSUES/009). + enter("stocking the registry"); + const stock = await stockRegistry(scenario.images ?? [], log); + let registry: Awaited> = null; + try { + enter("raising the registry"); + registry = await raiseRegistry(scenario, instanceId, stock, log); + } finally { + // Cleaning up scratch must not fail a raise that succeeded. The scenario is standing + // and usable; a directory left behind is untidy, and saying so is the honest report. + try { + await discardStock(stock); + } catch (err) { + log(` (could not remove the registry's scratch directory: ${(err as Error).message})`); + } + } + + enter("routing machines through their gateways"); await applyDefaultRoutes(scenario, byMachine, log); // Last: a machine that refuses inbound must still have been reachable while the lab // was configuring it. - step = "applying host firewalls"; + enter("applying host firewalls"); await applyHostFirewalls(scenario, byMachine, log); // Last, and only once the underlay is real. Placing before the machines can reach each // other would test the host against a network the scenario does not describe. - step = "placing"; + enter("placing"); await applyPlacements(scenario, byMachine, log); return { instanceId, scenario: scenario.scenario, - machines: [...created, ...routers], + images: registry?.pinned ?? [], + machines: [...created, ...routers, ...(registry ? [registry.machine] : [])], networks, pool, }; diff --git a/src/lifecycle/registry.ts b/src/lifecycle/registry.ts new file mode 100644 index 0000000..4daefd8 --- /dev/null +++ b/src/lifecycle/registry.ts @@ -0,0 +1,405 @@ +/** + * A registry inside the scenario. + * + * A sealed machine cannot reach a registry, and an image placed from an archive cannot keep its + * digest — `docker save` of a digest reference produces an archive with no repo tag, because a + * repo digest only exists for an image a registry served (novox/hq 04-ISSUES/009). So an image + * pinned by digest, which is the only kind the host accepts + * ([ADR 0006](../../02-DECISIONS/0046-the-installer-fetches-what-it-pins.md)), could not be + * placed at all. + * + * The answer is a registry, and it is not a workaround for the lab: ADR 0006 names an OCI + * registry as substrate, and ADR 0006 says a first node fetches "upstream, wherever the image + * ordinarily lives". **This is that upstream** — scenery, like the transit router is the + * internet ([ADR 0016](../../02-DECISIONS/0033-a-router-is-scenery-not-a-node.md)). + * + * The digests it serves are its own, not Docker Hub's, and that is correct rather than a + * compromise. What ADR 0006 requires is a reference that is exact and cannot move. A digest + * assigned by this registry is both. + */ + +import { spawn } from "node:child_process"; + +import { incus, incusOk, succeeds } from "../incus/client.ts"; +import { macFor, networkName } from "./names.ts"; +import { addressLink } from "./address.ts"; +import { around, log, shorten } from "../log.ts"; +import { BASE_IMAGE_ALIAS, placeImage } from "./place.ts"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +/** The image the registry itself runs from. Placed by tag, which archives keep. */ +export const REGISTRY_IMAGE = "registry:2"; + +/** Where the registry serves, inside its machine. */ +export const REGISTRY_PORT = 5000; + +export class RegistryError extends Error { + constructor(message: string) { + super(message); + this.name = "RegistryError"; + } +} + +export interface StockedImage { + /** What the scenario asked for, as written. */ + requested: string; + /** The repository path the registry serves it under. */ + repository: string; + /** The digest THIS registry assigned. What a declaration pins. */ + digest: string; +} + +export interface Stock { + /** A directory holding the registry's data, ready to be placed in a machine. */ + dataDir: string; + images: StockedImage[]; +} + +/** + * Build a registry's data directory on this workstation, with the given images in it. + * + * Runs a throwaway registry here — where there IS a network — pushes into it, and keeps what + * it wrote. Research 012's reframing again: fetch at build time on a machine that has a + * network, apply on a target that needs nothing. + * + * The caller owns the returned directory and must remove it. + */ +export async function stockRegistry( + references: string[], + log: (message: string) => void = () => {}, +): Promise { + if (references.length === 0) return { dataDir: "", images: [] }; + + const dataDir = await mkdtemp(join(tmpdir(), "mesh-lab-registry-")); + const container = `mesh-lab-stock-${process.pid}`; + const port = 5000 + (process.pid % 1000); + + await docker(["rm", "-f", container], 60_000); + const started = await docker( + ["run", "-d", "--name", container, "-p", `${port}:5000`, "-v", `${dataDir}:/var/lib/registry`, + REGISTRY_IMAGE], + 300_000, + ); + if (!started.ok) { + await rm(dataDir, { recursive: true, force: true }); + throw new RegistryError( + `cannot run ${REGISTRY_IMAGE} on this workstation to stock a registry: ${started.stderr.trim()}`, + ); + } + + try { + await waitForRegistry(port); + const images: StockedImage[] = []; + + for (const reference of references) { + // The repository path a machine will pull from. A tag is dropped: what a declaration + // pins is the digest, and carrying the tag as well would invite pinning the wrong one. + const repository = repositoryFor(reference); + const target = `localhost:${port}/${repository}`; + + const tagged = await docker(["tag", reference, target], 60_000); + if (!tagged.ok) { + throw new RegistryError( + `${reference} is not on this workstation, and the lab does not fetch on a scenario's ` + + `behalf. Pull it here first.\n ${tagged.stderr.trim()}`, + ); + } + const pushed = await docker(["push", target], 900_000); + if (!pushed.ok) throw new RegistryError(`cannot push ${reference}: ${pushed.stderr.trim()}`); + + const digest = digestFrom(pushed.stdout + pushed.stderr); + if (!digest) { + throw new RegistryError( + `${reference} was pushed and the registry did not report a digest. Without one there ` + + `is nothing for a declaration to pin.`, + ); + } + images.push({ requested: reference, repository, digest }); + log(` stocked ${repository}@${digest}`); + } + + return { dataDir, images }; + } catch (err) { + await discardStock({ dataDir, images: [] }); + throw err; + } finally { + await docker(["rm", "-f", container], 60_000); + } +} + +/** + * Remove a stocked registry's data. + * + * Through a container, because a container wrote it. The registry runs as root inside, so the + * blobs it writes into a bind mount are owned by root and an ordinary process cannot remove + * them — `rmdir` fails with EACCES on a directory that looks like ours. + * + * Whoever made the files removes them. + */ +export async function discardStock(stock: Stock): Promise { + if (!stock.dataDir) return; + await docker(["run", "--rm", "-v", `${stock.dataDir}:/stock`, REGISTRY_IMAGE, + "sh", "-c", "rm -rf /stock/* /stock/.[!.]* 2>/dev/null || true"], 120_000); + await rm(stock.dataDir, { recursive: true, force: true }).catch(() => {}); +} + +/** `alpine:3.20` and `alpine` both serve from `alpine`; `foo/bar:1` from `foo/bar`. */ +export function repositoryFor(reference: string): string { + const withoutDigest = reference.split("@")[0] ?? reference; + const lastColon = withoutDigest.lastIndexOf(":"); + const lastSlash = withoutDigest.lastIndexOf("/"); + return lastColon > lastSlash ? withoutDigest.slice(0, lastColon) : withoutDigest; +} + +/** `docker push` prints `: digest: sha256:… size: …` on its last useful line. */ +export function digestFrom(output: string): string | null { + const match = output.match(/digest:\s*(sha256:[a-f0-9]{64})/); + return match?.[1] ?? null; +} + +async function waitForRegistry(port: number): Promise { + for (let i = 0; i < 30; i++) { + const probe = await docker(["run", "--rm", "--network", "host", REGISTRY_IMAGE, + "sh", "-c", `wget -q -O- http://localhost:${port}/v2/ >/dev/null 2>&1`], 30_000); + if (probe.ok) return; + await new Promise((r) => setTimeout(r, 1_000)); + } + throw new RegistryError("a registry was started on this workstation and never answered"); +} + +function docker( + args: string[], + timeoutMs: number, +): Promise<{ ok: boolean; stdout: string; stderr: string }> { + // The second of the three places the lab runs an external program (novox/hq 04-ISSUES/024). + // `docker push` of a large image is minutes of legitimate silence, which is exactly when a + // heartbeat earns its keep. + return around(`docker ${shorten(args)}`, () => runDocker(args, timeoutMs), { heartbeatMs: 15_000 }); +} + +function runDocker( + args: string[], + timeoutMs: number, +): Promise<{ ok: boolean; stdout: string; stderr: string }> { + return new Promise((resolve) => { + const child = spawn("docker", args, { stdio: ["ignore", "pipe", "pipe"] }); + let stdout = ""; + let stderr = ""; + const timer = setTimeout(() => child.kill("SIGKILL"), timeoutMs); + child.stdout.on("data", (d) => (stdout += d)); + child.stderr.on("data", (d) => (stderr += d)); + child.on("error", (err) => { + clearTimeout(timer); + resolve({ ok: false, stdout, stderr: err.message }); + }); + child.on("close", (code) => { + clearTimeout(timer); + // A docker failure is an answer here rather than an exception, so it would otherwise pass + // through the log looking exactly like a success. + if (code !== 0) { + log.debug(` exit ${code}: ${shorten([stderr.trim() || "(nothing on stderr)"], 400)}`); + } + resolve({ ok: code === 0, stdout, stderr }); + }); + }); +} + +// --- the registry inside a scenario ------------------------------------------------------------ + +/** + * Where the registry sits on its segment. + * + * A convention rather than a declaration, like the router's. `.250` is chosen to sit well away + * from the low addresses scenarios give their machines, so a scenario can be written without + * thinking about it and a collision is obvious when it happens. + */ +export const REGISTRY_HOST_OCTET = 250; + +/** The address the registry answers on, given the segment it is attached to. */ +export function registryAddress(cidr: string): string { + const [network] = cidr.split("/"); + const parts = (network ?? "").split("."); + if (parts.length !== 4) { + throw new RegistryError( + `cannot place a registry on '${cidr}': it is not an IPv4 network, and the registry needs ` + + `an address a machine can be pointed at.`, + ); + } + return `${parts[0]}.${parts[1]}.${parts[2]}.${REGISTRY_HOST_OCTET}`; +} + +/** What a declaration should pin, once a scenario is raised. */ +export function pinnedReference(address: string, image: StockedImage): string { + return `${address}:${REGISTRY_PORT}/${image.repository}@${image.digest}`; +} + +// --- raising it inside a scenario --------------------------------------------------------------- + +/** What a raised registry is, and what a declaration needs from it. */ +export interface RaisedRegistry { + machine: string; + segment: string; + address: string; + /** Each image, as a reference a declaration can pin. */ + pinned: string[]; +} + +/** + * Pick the segment the registry sits on. + * + * A public segment, because that is what stands in for the outside world — a first node fetches + * from upstream, and this is upstream. An IPv4 range, because a machine has to be pointed at it + * by address. + */ +export function registrySegment( + segments: Record, +): { name: string; cidr: string } | null { + for (const [name, segment] of Object.entries(segments)) { + if (segment.kind !== "public") continue; + const v4 = segment.cidr.find((c) => !c.includes(":")); + if (v4) return { name, cidr: v4 }; + } + return null; +} + +/** + * Raise a registry inside the scenario and load the stocked images into it. + * + * Scenery, in the same sense the transit router is: nothing under test runs on it, it holds no + * identity, and no assertion is made about its internals. It exists so that a machine can fetch + * an image the way a real one does — over the network, from a registry, by digest. + */ +export async function raiseRegistry( + scenario: { segments: Record }, + instanceId: string, + stock: Stock, + log: (message: string) => void = () => {}, +): Promise { + if (stock.images.length === 0) return null; + + const segment = registrySegment(scenario.segments); + if (!segment) { + throw new RegistryError( + `this scenario declares images and has no public IPv4 segment to serve them from.\n` + + ` The registry stands in for the outside world, so it sits on a public segment.`, + ); + } + + const address = registryAddress(segment.cidr); + const name = `mlab-${instanceId}-registry`; + const prefix = segment.cidr.slice(segment.cidr.lastIndexOf("/")); + + if (!(await succeeds(["config", "show", name], 15_000))) { + await incus([ + "init", BASE_IMAGE_ALIAS, name, "--vm", + "-c", "security.secureboot=false", + "-c", "limits.memory=1GiB", + "-c", `user.mesh-lab.instance=${instanceId}`, + // Tagged as a machine as well, so `destroy` finds it with one query — a router that + // carried only its own tag was left behind and held its networks open. + "-c", "user.mesh-lab.machine=registry", + "-c", "user.mesh-lab.registry=true", + ], 300_000); + await succeeds(["config", "device", "remove", name, "eth0"], 15_000); + await incus([ + "config", "device", "add", name, "eth0", "nic", + "nictype=bridged", + `parent=${networkName(instanceId, segment.name)}`, + `hwaddr=${macFor(instanceId, "registry", 0)}`, + ]); + } + await succeeds(["start", name], 60_000); + await waitForAgent(name); + + // Addressed the way every other machine is: a systemd-networkd unit matching the MAC. + // + // **This used to be `ip addr add`, and it stalled the lab.** An address set by hand leaves + // networkd waiting to configure a link it was never told about, so the link sits at + // `configuring`, `systemd-networkd-wait-online` never returns — its timeout is `infinity` — + // and `network-online.target` is never reached. Docker is ordered after that target, so + // `docker load` two lines below blocked on a socket whose daemon was queued behind a target + // that would never come. + // + // Matching on MAC and not on interface name is still the rule: a machine with a container + // runtime has a `docker0` that sorts before `enp5s0`, and naive selection configures that. + await addressLink(name, { + device: "eth0", + mac: macFor(instanceId, "registry", 0), + addresses: [`${address}${prefix}`], + // The registry takes the segment's default. It carried no MTU before this and still does + // not: what a scenario sets an MTU for is the path under test, and this is scenery. + mtu: undefined, + }); + + log(` registry on ${segment.name} at ${address}`); + + // The registry's own image, placed by tag — an archive keeps a tag and cannot keep a digest, + // which is the whole reason this machine exists. + // Logged, not silenced. This is the step a stall sat in for thirty-five minutes while the + // caller had passed it a callback that threw everything away (novox/hq 04-ISSUES/024). + await placeImage(name, "registry", REGISTRY_IMAGE, log); + + // The destination must EXIST before a recursive push, or incus copies the source's contents + // rather than the source — the data lands one directory too shallow, the registry finds + // nothing where it looks, and every pull fails with `not found`. + await incus(["exec", name, "--", "mkdir", "-p", "/srv/registry"], 60_000); + await incus(["file", "push", "-r", `${stock.dataDir}/docker`, `${name}/srv/registry/`], 900_000); + + await incus(["exec", name, "--", "docker", "run", "-d", + "--name", "registry", "--restart", "unless-stopped", + "-p", `${REGISTRY_PORT}:5000`, + "-v", "/srv/registry:/var/lib/registry", + REGISTRY_IMAGE], 300_000); + + // Read back that each image is SERVED, by asking for its manifest by digest — which is + // exactly what a machine will do. + // + // Not that the catalog endpoint answers: `{"repositories":[]}` contains the word + // `repositories`, so checking for that passed on a registry holding nothing at all, and the + // failure surfaced much later as a container that could not be pulled. + let answered = false; + for (let i = 0; i < 20 && !answered; i++) { + const ping = await incusOk(["exec", name, "--", "curl", "-s", "-o", "/dev/null", + "-w", "%{http_code}", "--max-time", "3", + `http://localhost:${REGISTRY_PORT}/v2/`], 30_000); + answered = ping?.trim() === "200"; + if (!answered) await new Promise((r) => setTimeout(r, 2_000)); + } + if (!answered) { + throw new RegistryError( + `the registry on ${name} started and never answered. Machines in this scenario cannot ` + + `fetch an image, so nothing that declares a container will work.`, + ); + } + + const pinned: string[] = []; + for (const image of stock.images) { + const code = await incusOk(["exec", name, "--", "curl", "-s", "-o", "/dev/null", + "-w", "%{http_code}", "--max-time", "5", + "-H", "Accept: application/vnd.docker.distribution.manifest.v2+json", + `http://localhost:${REGISTRY_PORT}/v2/${image.repository}/manifests/${image.digest}`, + ], 60_000); + if (code?.trim() !== "200") { + throw new RegistryError( + `the registry on ${name} is running and does not serve ${image.repository}@${image.digest} ` + + `(it answered ${code?.trim() || "nothing"}).\n` + + ` The images were stocked on this workstation and did not arrive intact, so a ` + + `machine declaring that image would fail to pull it.`, + ); + } + const reference = pinnedReference(address, image); + pinned.push(reference); + log(` serving ${reference}`); + } + return { machine: name, segment: segment.name, address, pinned }; +} + +async function waitForAgent(name: string): Promise { + for (let i = 0; i < 90; i++) { + if (await succeeds(["exec", name, "--", "true"], 10_000)) return; + await new Promise((r) => setTimeout(r, 2_000)); + } + throw new RegistryError(`${name} started and its agent never answered.`); +} diff --git a/src/lifecycle/router.ts b/src/lifecycle/router.ts index 5d28dc5..7fc62a4 100644 --- a/src/lifecycle/router.ts +++ b/src/lifecycle/router.ts @@ -6,7 +6,7 @@ * nothing to say about it. * * A router is **scenery, not a node**, so it is a container rather than a virtual machine - * (novox/hq ADR 0033). Nothing under test runs on it and no assertion is made about its + * (novox/hq ADR 0016). Nothing under test runs on it and no assertion is made about its * internals; it exists so packets behave the way they behave in the world. What it has to * reproduce is kernel behaviour, and a container has the same kernel. */ @@ -62,7 +62,7 @@ export async function ensureRouterImage(log: (message: string) => void = () => { // Default profile on purpose: this is the one container that needs to reach a repository. await incus(["launch", ROUTER_BASE, builder], 300_000); - await waitUntilUsable(builder, 120, () => {}); + await waitUntilUsable(builder, 120, log); // `exec` works before the container has an address. Usable means a command runs; it does // not mean the network is up, and the first attempt failed on DNS because those were @@ -279,7 +279,7 @@ export async function raiseTransit( } } await succeeds(["start", name], 60_000); - await waitUntilUsable(name, 120, () => {}); + await waitUntilUsable(name, 120, log); for (const [index, segment] of publicSegments.entries()) { const device = `eth${index}`; @@ -343,7 +343,7 @@ export async function raiseRouters( } for (const name of created) { - await waitUntilUsable(name, 120, () => {}); + await waitUntilUsable(name, 120, log); } for (const plan of plans) { diff --git a/src/lifecycle/supported.ts b/src/lifecycle/supported.ts index c78d132..55b07d8 100644 --- a/src/lifecycle/supported.ts +++ b/src/lifecycle/supported.ts @@ -14,7 +14,7 @@ */ import type { Scenario } from "../declaration/types.ts"; -import { PLACEABLE, planPlacements } from "./place.ts"; +import { IMAGE_PREFIX, isPlaceable, PLACEABLE, planPlacements } from "./place.ts"; export class UnsupportedError extends Error { readonly missing: string[]; @@ -34,19 +34,19 @@ export class UnsupportedError extends Error { export function assertSupported(scenario: Scenario): void { const missing: string[] = []; - // `place: [host]` works. Everything else in the vocabulary is named individually rather - // than refused as a whole, so a scenario that places a host and a substrate is told exactly - // which half the lab cannot do. + // `host`, `runtime` and `image:` work. Everything else in the vocabulary is named + // individually rather than refused as a whole, so a scenario that places a host and a + // substrate is told exactly which half the lab cannot do. const unplaceable = new Set(); for (const { artifacts } of planPlacements(scenario)) { for (const artifact of artifacts) { - if (!(PLACEABLE as readonly string[]).includes(artifact)) unplaceable.add(artifact); + if (!isPlaceable(artifact)) unplaceable.add(artifact); } } for (const artifact of [...unplaceable].sort()) { missing.push( - `place: ${artifact} — only ${PLACEABLE.join(", ")} can be placed; the tiers above ` + - `tier 0 do not exist yet`, + `place: ${artifact} — only ${PLACEABLE.join(", ")} and ${IMAGE_PREFIX} can ` + + `be placed; the tiers above tier 0 do not exist yet`, ); } diff --git a/src/log.ts b/src/log.ts new file mode 100644 index 0000000..179ae71 --- /dev/null +++ b/src/log.ts @@ -0,0 +1,163 @@ +/** + * What the lab was doing, written down while it does it. + * + * **The lab's own recurring fault, and the one it exists to catch elsewhere.** On 2026-09-01 a run + * stalled for thirty-five minutes and said nothing at all. The cause was a link that systemd was + * still configuring, three layers down inside a `docker load` that was blocked on a socket — and + * every one of those layers knew what it was waiting for. None of them said so + * (novox/hq 04-ISSUES/024). + * + * Three decisions follow from that, and each is doing work: + * + * **Every external command is logged, at the three places that run one.** Ninety-seven call sites + * reach a hypervisor or a container runtime through three wrappers, so instrumenting the wrappers + * covers all of them and nothing has to remember to log. + * + * **A command that is still running says so while it runs.** A line before and a line after tells + * you nothing until the after arrives, which is exactly the case that matters. So a command + * outstanding past a few seconds reports itself periodically, with how long it has been going. + * *"I cannot see progress" is not evidence of no progress* — but it should not be the only thing + * available either. + * + * **It goes to a file, written synchronously.** Node block-buffers stdout when it is redirected, + * and a test runner buffers it again, so a console log can sit minutes behind the run. That was + * read as a stall twice in one day — the second time straight after the real fix, where a + * buffering artifact argues the fix did not work. `appendFileSync` cannot lag. + */ + +import { appendFileSync, mkdirSync } from "node:fs"; +import { dirname } from "node:path"; + +/** How much to say. `off` writes nothing at all, for a caller that wants none of this. */ +export type Level = "off" | "info" | "debug" | "trace"; + +const ORDER: Record = { off: 0, info: 1, debug: 2, trace: 3 }; + +/** + * Where the log goes, and how much of it. + * + * Off unless asked for, because a suite that writes a debug log nobody reads is a suite that + * writes a debug log nobody reads. `MESH_LAB_LOG` turns it on; `MESH_LAB_LOG_FILE` says where. + */ +function configured(): { level: Level; file: string | null } { + const asked = (process.env["MESH_LAB_LOG"] ?? "").trim().toLowerCase(); + const level: Level = asked === "info" || asked === "debug" || asked === "trace" ? asked : "off"; + const file = process.env["MESH_LAB_LOG_FILE"]?.trim() || null; + return { level, file }; +} + +let level: Level = configured().level; +let file: string | null = configured().file; +const started = Date.now(); +let opened = false; + +/** Re-read the environment. For a test, and for a caller that sets it after import. */ +export function reconfigure(): void { + const now = configured(); + level = now.level; + file = now.file; + opened = false; +} + +/** Point the log somewhere explicitly, whatever the environment says. */ +export function logTo(path: string | null, at: Level = "debug"): void { + file = path; + level = path === null ? "off" : at; + opened = false; +} + +function elapsed(): string { + return ((Date.now() - started) / 1000).toFixed(1).padStart(7) + "s"; +} + +/** + * Write one line, synchronously. + * + * A failure to write is swallowed. A log that cannot be written is a nuisance; a run that dies + * because its log could not be written is a fault the log invented. + */ +function write(at: Level, message: string): void { + if (level === "off" || ORDER[at] > ORDER[level]) return; + const line = `${new Date().toISOString()} ${elapsed()} ${at.padEnd(5)} ${message}\n`; + if (!file) { + process.stderr.write(line); + return; + } + try { + if (!opened) { + mkdirSync(dirname(file), { recursive: true }); + opened = true; + } + appendFileSync(file, line); + } catch { + // Nothing sensible to do, and saying so on every line would be worse than silence. + } +} + +export const log = { + info: (message: string) => write("info", message), + debug: (message: string) => write("debug", message), + trace: (message: string) => write("trace", message), + /** Whether anything is being recorded, for a caller deciding whether to compose an expensive message. */ + on: () => level !== "off", +}; + +/** + * Run something, saying what it is and how long it took — and saying so *while* it runs. + * + * The heartbeat is the point. A stall is a command that started and has not finished, and the only + * thing that distinguishes it from ordinary work is how long it has been going — which nothing can + * tell you unless something is still counting. + */ +export async function around( + what: string, + run: () => Promise, + options: { heartbeatMs?: number; at?: Level; expectedToFail?: boolean } = {}, +): Promise { + const at = options.at ?? "debug"; + // Only when nothing at all is being recorded. **Skipping the wrapper whenever the step's own + // level is too low was wrong twice over**: it dropped the failure line, which is logged louder + // than the step precisely so it survives a turned-down log, and it dropped the heartbeat, which + // is the whole reason this function exists. Both went missing at exactly the level somebody + // would actually run — `info`. The gate belongs in `write`, which already has one. + if (level === "off") return run(); + + const beat = options.heartbeatMs ?? 15_000; + const from = Date.now(); + write(at, `→ ${what}`); + + const timer = setInterval(() => { + // **Reported as still running, not as stuck.** Which it is is not knowable from here, and a + // log that calls a slow step a hang teaches people to ignore it. + write("info", `… ${what} — still running after ${((Date.now() - from) / 1000).toFixed(0)}s`); + }, beat); + // Never hold the process open for a heartbeat. + timer.unref?.(); + + try { + const result = await run(); + write(at, `← ${what} (${((Date.now() - from) / 1000).toFixed(1)}s)`); + return result; + } catch (err) { + // Failures are logged at info even when the step was debug: whoever turned logging down still + // wants the thing that went wrong. + // + // **Unless failing is one of the answers.** Plenty of commands here are questions — does this + // network exist, is the agent up yet — and a failure is a no. Logging those as faults fills a + // normal run with ✗ and teaches whoever reads it that ✗ means nothing, which is the state to + // be in when one of them is real. + write(options.expectedToFail ? "debug" : "info", + `${options.expectedToFail ? "·" : "✗"} ${what} ` + + `(${((Date.now() - from) / 1000).toFixed(1)}s): ${(err as Error).message}`); + throw err; + } finally { + clearInterval(timer); + } +} + +/** A command line, shortened for a log without losing which command it was. */ +export function shorten(argv: string[], limit = 200): string { + const whole = argv.join(" "); + if (whole.length <= limit) return whole; + return whole.slice(0, limit - 1) + "…"; +} diff --git a/src/pinning.ts b/src/pinning.ts new file mode 100644 index 0000000..110f2e1 --- /dev/null +++ b/src/pinning.ts @@ -0,0 +1,56 @@ +/** + * Rewriting an image reference to the one a scenario's own registry serves. + * + * **A digest is not knowable until something is built** (novox/hq 04-ISSUES/025). A manifest in a + * repository can pin a third-party image, because somebody can ask a registry what a tag points + * at. It cannot pin an image the mesh builds itself: that image does not exist yet, and when it + * does its digest belongs to whichever registry served it. + * + * The bundle has always had this problem and solves it by rewriting references once the scenario's + * registry is up and its digests are known. Modules have exactly the same problem and were solving + * it by shipping sixty-four zeros, which parses, resolves, composes — and stops on the machine. + * + * So the rewriting is shared rather than copied, and matches on the **repository**, because that + * is the part a person writes and the only part that survives being served somewhere else. + */ + +/** `192.0.2.250:5000/ghcr.io/mailu/admin@sha256:…` → `ghcr.io/mailu/admin` */ +export function repositoryOf(pinned: string): string { + const at = pinned.indexOf("@"); + const body = at === -1 ? pinned : pinned.slice(0, at); + const slash = body.indexOf("/"); + // Everything after the registry. A reference with no slash at all is its own repository. + return slash === -1 ? body : body.slice(slash + 1); +} + +/** + * Replace every reference to a stocked repository with the reference this scenario serves. + * + * Matching is on the repository and ignores whatever registry and digest were written down — + * a file may name `postgres@sha256:7456…` or `mesh-provision-postgres@sha256:0000…` and both mean + * *the postgres this scenario has*. That is the whole point: the text says which image, the + * scenario says which copy. + * + * A repository the scenario did not stock is left alone rather than blanked. It may be reachable + * some other way, and silently emptying a reference would produce the exact failure this exists to + * prevent. + */ +export function pinnedInto(text: string, served: string[]): string { + let out = text; + for (const pinned of served) { + const repository = repositoryOf(pinned); + const escaped = repository.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + // Optionally a registry, then the repository, then any digest. Anchored on a quote or + // whitespace so a longer repository ending in a shorter one is not half-replaced. + out = out.replaceAll( + new RegExp(`(?<=^|["\\s])(?:[A-Za-z0-9_.:-]+\\/)*${escaped}@sha256:[0-9a-f]{64}`, "g"), + pinned, + ); + } + return out; +} + +/** Whether anything is still pinned to a placeholder, which would fail on the machine. */ +export function stillUnpinned(text: string): string[] { + return [...text.matchAll(/([A-Za-z0-9_.:/-]+)@sha256:0{64}/g)].map((m) => m[1]!); +} diff --git a/src/rebuild.ts b/src/rebuild.ts new file mode 100644 index 0000000..c945348 --- /dev/null +++ b/src/rebuild.ts @@ -0,0 +1,100 @@ +/** + * Rebuild what the lab runs, from source, before it runs. + * + * **A stale artifact reporting success against old rules is the fault this project keeps writing + * down** (novox/hq 04-ISSUES/005). The lab consumes three artifacts from two repositories, and they + * were rebuilt by hand, one at a time, from memory. A rename in the control plane's catalogue needs + * both the control-plane image *and* the builder binary, because both parse manifests; rebuilding + * one left a binary eleven hours old refusing a field the mesh had just renamed, and cost a full + * run to find out. + * + * In the repository rather than in a shell script beside it, for the reason 005 is about: a step + * that lives in somebody's terminal history runs when they remember, and remembering is not a + * mechanism. + */ + +import { spawnSync } from "node:child_process"; +import { repositories } from "./repos.ts"; + +export interface Build { + /** What it produces, for the log. */ + what: string; + /** The repository root to run in. */ + in: string; + argv: string[]; + env?: NodeJS.ProcessEnv; +} + +/** + * planned is what must be built, given where this run has been pointed. + * + * Derived from the same environment the suite is configured by, so there is one place that says + * where a repository is. A repository this run was not pointed at is not built — and, per + * {@link whatWasTested}, is not claimed in the receipt either. + */ +export function planned(env: NodeJS.ProcessEnv = process.env): Build[] { + const builds: Build[] = []; + const where = repositories(env); + const host = env["MESH_LAB_HOST_BINARY"]; + if (host && where["mesh-host"]) { + builds.push({ + what: "host", + in: where["mesh-host"], + argv: ["go", "build", "-ldflags=-s -w -X main.builtFor=arch", "-o", host, "./cmd/mesh-host"], + env: { CGO_ENABLED: "0" }, + }); + } + const control = where["mesh-control"]; + if (control) { + // **Every image the lab runs, not only the control plane's.** + // + // On 2026-09-01 a suite ran with a control-plane image built that minute and a provisioner + // image built the day before. The rotation test failed against a real database, and the + // failure looked exactly like the change under test being wrong — the provisioner was + // creating logins by a naming rule that had been replaced. + // + // This is the same fault the builder line below was added for, one target along. A rebuild + // that covers most of what a run uses is worse than one that covers none, because the run + // that follows it is believed. + builds.push({ + what: "images", + in: control, + argv: ["make", "image", "builder-image", "provisioner-image", "objectstore-image", + "redis-provisioner-image", "proxy-image"], + }); + const builder = env["MESH_LAB_BUILDER"]; + if (builder) { + // Both of these parse manifests. Building one and not the other is the eleven-hour-old + // binary above, so they are one step and not two. + builds.push({ + what: "builder", + in: control, + argv: ["go", "build", "-o", builder, "./cmd/mesh-builder"], + }); + } + } + return builds; +} + +/** rebuild runs the plan, and throws on the first failure rather than testing a stale artifact. */ +export function rebuild(env: NodeJS.ProcessEnv = process.env): string[] { + const built: string[] = []; + for (const build of planned(env)) { + const [command, ...args] = build.argv; + const ran = spawnSync(command!, args, { + cwd: build.in, + env: { ...env, ...build.env }, + encoding: "utf8", + }); + if (ran.status !== 0) { + // Loudly, and stopping. A suite that runs anyway is a suite reporting on code that is not + // the code in front of you, which is the whole of 005. + throw new Error( + `could not build the ${build.what}: ${build.argv.join(" ")} in ${build.in}\n\n` + + `${(ran.stderr || ran.stdout || String(ran.error)).trim()}`, + ); + } + built.push(build.what); + } + return built; +} diff --git a/src/repos.ts b/src/repos.ts new file mode 100644 index 0000000..dadabc9 --- /dev/null +++ b/src/repos.ts @@ -0,0 +1,28 @@ +/** + * Where the repositories this run was pointed at are. + * + * **One place**, because two things need it and they must agree: the rebuild builds these, and the + * receipt claims these. A receipt naming a repository the run did not build is exactly the + * false coverage novox/hq 04-ISSUES/005 is about, and it would arrive by nobody's decision — just + * two derivations drifting. + * + * Derived from the environment the suite is configured by, so a repository this run was not + * pointed at is neither built nor claimed. + */ + +import { dirname } from "node:path"; + +export interface Repositories { + /** Absolute path to the repository root, by name. */ + [name: string]: string; +} + +export function repositories(env: NodeJS.ProcessEnv = process.env): Repositories { + const found: Repositories = {}; + found["mesh-lab"] = process.cwd(); + const host = env["MESH_LAB_HOST_BINARY"]; + if (host) found["mesh-host"] = dirname(host); + const modules = env["MESH_LAB_MODULES"]; + if (modules) found["mesh-control"] = dirname(dirname(modules)); + return found; +} diff --git a/src/suite.ts b/src/suite.ts new file mode 100644 index 0000000..8496026 --- /dev/null +++ b/src/suite.ts @@ -0,0 +1,107 @@ +/** + * Run the end-to-end suite, and leave a receipt saying it ran. + * + * **Here rather than inside the tests, because the tests cannot know their own totals.** Node's + * runner reports them to whatever invoked it, and a test file inventing its own count would be a + * receipt that says whatever the last edit made it say. + * + * Here rather than in a shell script for the same reason the rebuild is: a step that lives in + * somebody's terminal history is a step that runs when they remember (novox/hq 04-ISSUES/005). + */ + +import { spawn } from "node:child_process"; +import { endToEnd, record, whatWasTested } from "./lastrun.ts"; +import { rebuild } from "./rebuild.ts"; + +/** counted is what the runner said, or nulls when it said nothing recognisable. */ +export function counted(output: string): { passed: number | null; failed: number | null } { + // The runner's own summary lines, each on a line of its own. Anchored, so a test *named* + // "pass 3" cannot be mistaken for the total — which is not a hypothetical worry in a suite whose + // tests are named in sentences. + // Stripped first: the runner colours its summary even when its stdout is a pipe, so the line is + // "\x1b[34m\u2139 pass 8\x1b[39m" and an anchored pattern never sees the start of it. Found by + // running this against the real runner — the fixture it was first written against was output I + // had imagined, which is a test that agrees with the mistake it was written beside. + const plain = output.replace(/\u001b\[[0-9;]*m/g, ""); + const total = (what: RegExp) => { + const found = plain.match(what); + return found ? Number(found[1]) : null; + }; + return { + passed: total(/^\s*(?:\u2139|#)\s*pass\s+(\d+)\s*$/m), + failed: total(/^\s*(?:\u2139|#)\s*fail\s+(\d+)\s*$/m), + }; +} + +export async function runSuite(args: string[]): Promise { + const ran = args.filter((a) => a !== "--no-build"); + const files = ran.length > 0 ? ran : [endToEnd]; + + if (!args.includes("--no-build")) { + // Before the run, always. The artifacts are built from two other repositories, and a suite + // that tests yesterday's binary reports on code nobody is looking at (novox/hq 04-ISSUES/005). + const built = rebuild(); + if (built.length > 0) console.log(`built: ${built.join(", ")}\n`); + } + // Read now, while it is true. The receipt names these, and reading them when the run ends + // names whatever was committed during the twenty minutes in between instead. + const against = whatWasTested(process.env); + + // **No canary.** There was one: a second scenario, one machine, raised first so a broken mesh + // failed in two minutes rather than in forty. It walked exactly the path the first three tests + // of the long run walk — a mesh comes up, a module lands, a consumer gets a credential — and + // the long run reaches the end of that path in about 160 seconds. + // + // So it cost a whole scenario, every passing run, to save about 45 seconds on a failing one. + // A scenario is three machines including a registry that boots a kernel to serve files, which + // is where the two minutes went. `test/integration/canary.test.ts` is still there and still + // runs when it is named; it is no longer raised on the way to everything else. + + const { code, seen } = await runFiles(files); + console.log("\n" + reportOn(counted(seen), (p, f) => record(p, f, files, process.env, against))); + return code; +} + +/** Run some test files, passing their output through as it arrives. */ +async function runFiles(files: string[]): Promise<{ code: number; seen: string }> { + const running = spawn( + process.execPath, + ["--test", "--test-concurrency=1", "--experimental-strip-types", + ...files], + { stdio: ["inherit", "pipe", "inherit"] }, + ); + + let seen = ""; + running.stdout.on("data", (chunk: Buffer) => { + // Passed through as it arrives: a suite that takes a quarter of an hour must not look hung. + process.stdout.write(chunk); + seen += chunk.toString(); + }); + + const code: number = await new Promise((resolve) => { + running.on("close", (c) => resolve(c ?? 1)); + }); + return { code, seen }; +} + +/** + * reportOn decides whether this run says anything worth recording, and records it if so. + * + * Separated from the spawning so the decision can be tested: **no receipt rather than a guessed + * one** is the rule that keeps the record meaning something, and it was written where nothing + * could check it — which is 04-ISSUES/005 in miniature, inside the fix for it. + */ +export function reportOn( + counts: { passed: number | null; failed: number | null }, + write: (passed: number, failed: number) => { against: Record; ran: string[] }, +): string { + if (counts.passed === null || counts.failed === null) { + // A run whose result could not be read is a run nobody can say anything about. Writing + // "0 failed" because nothing said otherwise is how a green record comes to mean nothing. + return "could not read what the runner reported; no receipt written"; + } + const receipt = write(counts.passed, counts.failed); + const against = Object.entries(receipt.against).map(([n, c]) => `${n} ${c}`).join(", "); + return `recorded: ${receipt.ran.join(", ")} — ${counts.passed} passed, ` + + `${counts.failed} failed, against ${against || "nothing in git"}`; +} diff --git a/src/warm.ts b/src/warm.ts new file mode 100644 index 0000000..f8762d0 --- /dev/null +++ b/src/warm.ts @@ -0,0 +1,210 @@ +/** + * A scenario kept between runs, already brought to a state worth starting from. + * + * **Bootstrapping a mesh takes minutes and proves the same thing every time.** The tests worth + * iterating on are the ones after it — assigning a module, adopting a workload, watching something + * fail. A warm instance is raised once, brought to that state, snapshotted, and restored on every + * later run in seconds. + * + * **The danger is precisely the one 04-ISSUES/005 is about**, one level down: a mesh snapshotted + * against yesterday's binaries will pass today's tests and report green, and nothing about the + * result would say what it was actually run against. So a warm instance records the commits it was + * built from, and is refused — not silently rebuilt, refused — when they have moved. + * + * **Fresh stays the default.** This is for iterating. A run that is meant to mean something raises + * from nothing, because "it passes" must not quietly come to mean "it passes against a mesh + * somebody bootstrapped last week". + */ + +import { readFileSync, writeFileSync, mkdirSync, rmSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { homedir } from "node:os"; + +import { list, restore, snapshot, snapshots, destroy } from "./lifecycle/operate.ts"; +import type { Against } from "./lastrun.ts"; +import { whatWasTested } from "./lastrun.ts"; + +/** The state a warm instance is kept at. One label, because a second is a state nobody named. */ +export const label = "warm"; + +export interface Warm { + scenario: string; + instanceId: string; + /** + * The image references the scenario's registry serves, pinned by digest. + * + * Kept because they are worked out while raising and a restored instance never raises. Without + * them a warm run knows nothing about what it can pull, and every test naming an image fails + * for a reason that has nothing to do with what it was testing. + */ + images: string[]; + /** The commit each repository was at when this was brought to its state. */ + against: Against; + at: string; +} + +/** Where the record lives: XDG state, beside the run receipt, for the same reason. */ +export function recordPath(): string { + const state = process.env["XDG_STATE_HOME"] ?? join(homedir(), ".local", "state"); + return join(state, "mesh-lab", "warm.json"); +} + +export function remember(warm: Warm): void { + const path = recordPath(); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, JSON.stringify(warm, null, 2) + "\n"); +} + +export function remembered(): Warm | null { + try { + return JSON.parse(readFileSync(recordPath(), "utf8")) as Warm; + } catch { + return null; + } +} + +export function forget(): void { + rmSync(recordPath(), { force: true }); +} + +export type Verdict = + | { use: "restore"; instanceId: string } + | { use: "raise"; why: string }; + +/** + * judge decides whether a remembered instance may be restored. + * + * **Every reason to refuse is a reason a test would otherwise pass while meaning nothing**, so + * each is named rather than collapsed into "not usable". + */ +export function judge( + warm: Warm | null, + scenario: string, + standing: string[], + hasSnapshot: boolean, + against: Against, +): Verdict { + if (!warm) return { use: "raise", why: "nothing is being kept warm" }; + if (warm.scenario !== scenario) { + return { use: "raise", why: `what is kept warm is ${warm.scenario}, and this is ${scenario}` }; + } + if (!standing.includes(warm.instanceId)) { + return { use: "raise", why: `${warm.instanceId} is no longer standing` }; + } + if (!hasSnapshot) { + return { use: "raise", why: `${warm.instanceId} has no ${label} snapshot to return to` }; + } + // The check that keeps this honest. A mesh built from code that has since moved would pass + // today's tests against yesterday's binaries, and say nothing about it. + // + // **Both directions, because comparing only what is in front of you clears what is not.** The + // first version walked the current repositories alone, so running without the environment that + // names where they are compared nothing and reported the mesh usable — a warm instance built + // from code that had since moved, cleared by a check that had looked at neither. That is + // 04-ISSUES/005's rule again: a record that says nothing about something is not a record that + // clears it. + for (const name of new Set([...Object.keys(warm.against), ...Object.keys(against)])) { + const then = warm.against[name]; + const now = against[name]; + if (then === now) continue; + if (!now) { + return { + use: "raise", + why: `${name} was at ${then} when this was warmed, and nothing says where it is now — ` + + `so nothing can say whether it moved`, + }; + } + return { + use: "raise", + why: `${name} was at ${then ?? "nothing recorded"} when this was warmed, ` + + `and is now at ${now}`, + }; + } + return { use: "restore", instanceId: warm.instanceId }; +} + +/** What is standing right now, by instance. */ +export async function standingNow(): Promise { + return (await list()).map((i) => i.instanceId); +} + +/** + * ready returns an instance already at its warm state, or says why one must be raised. + * + * It never raises: raising needs a scenario, images and a bootstrap, and all of that belongs to + * whoever is using this rather than here. + */ +export async function ready( + scenario: string, + env: NodeJS.ProcessEnv = process.env, +): Promise { + const warm = remembered(); + const standing = await standingNow(); + const has = warm ? (await snapshots(warm.instanceId)).includes(label) : false; + return judge(warm, scenario, standing, has, whatWasTested(env)); +} + +/** returnTo puts a warm instance back to its state, and says how long it took. */ +export async function returnTo( + instanceId: string, + log: (message: string) => void = () => {}, +): Promise { + const { usableSeconds } = await restore(instanceId, label, 180, log); + return usableSeconds; +} + +/** + * keep snapshots an instance as the state to come back to, and records what it was built from. + * + * Called once the caller has brought the scenario to whatever "ready to work" means for it. + */ +export async function keep( + scenario: string, + instanceId: string, + env: NodeJS.ProcessEnv = process.env, +): Promise { + await snapshot(instanceId, label); + const warm: Warm = { + scenario, + instanceId, + images: stockOf(instanceId), + against: whatWasTested(env), + at: new Date().toISOString(), + }; + remember(warm); + return warm; +} + +/** cool destroys what is being kept and forgets it. */ +export async function cool(): Promise { + const warm = remembered(); + forget(); + if (!warm) return null; + if ((await standingNow()).includes(warm.instanceId)) { + await destroy(warm.instanceId); + } + return warm.instanceId; +} + +/** + * What a raised scenario stocked, held until it is kept. + * + * Raising works the images out and snapshotting happens later, so this carries them between the + * two without the caller having to hold them. + */ +const stock = new Map(); + +export function rememberStock(instanceId: string, images: string[]): void { + stock.set(instanceId, images); +} + +function stockOf(instanceId: string): string[] { + return stock.get(instanceId) ?? []; +} + +/** What a restored instance's registry serves, from when it was warmed. */ +export function warmStock(instanceId: string): { images: string[] } { + const warm = remembered(); + if (!warm || warm.instanceId !== instanceId) return { images: [] }; + return { images: warm.images }; +} diff --git a/test/connect.test.ts b/test/connect.test.ts new file mode 100644 index 0000000..ab1d587 --- /dev/null +++ b/test/connect.test.ts @@ -0,0 +1,14 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; + +import { RULE } from "../src/lifecycle/connect.ts"; + +// The rule goes in its own file, beside the one this workstation already has. +// +// **Not into it.** The file next to this belongs to the mesh that really runs here, and it is +// generated — writing into it would be edited-away at best and would break real name resolution +// at worst. novox/hq: never edit a file something else owns. +test("the rule is its own file, not the one already there", () => { + assert.match(RULE, /^\/etc\/dnsmasq\.d\/mesh-lab\.conf$/); + assert.doesNotMatch(RULE, /hal/, "it would be writing into something else's file"); +}); diff --git a/test/enumerate.test.ts b/test/enumerate.test.ts new file mode 100644 index 0000000..93a8b4d --- /dev/null +++ b/test/enumerate.test.ts @@ -0,0 +1,41 @@ +// Set before importing: the client reads MESH_LAB_INCUS once, at module load. +// `false` is a real program that exits non-zero and prints nothing — which is also the worst +// case, because an empty stderr is how a failure arrives with no explanation. +process.env["MESH_LAB_INCUS"] = "false"; + +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { instanceExists, taggedInstances, taggedNetworks } from "../src/incus/client.ts"; + +/** + * "I cannot see" must never be answered as "there is nothing there." + * + * This is the fault the comment on `incusOk` warns about, committed by three of its own callers + * writing `?? "[]"`. It cost a session: `mesh-lab list` printed *no scenario instances standing* + * while two were standing, because the shell had no permission to reach the daemon. Nothing was + * wrong with the lab's knowledge of the instances — it had never managed to ask. + * + * The same shape as the fault the node host exists to prevent, in the tool that tests the host: + * a service that does not exist reported as `stopped`. + */ + +test("listing instances fails rather than reporting none", async () => { + await assert.rejects( + () => taggedInstances(), + "a failed `incus list` came back as an empty list; every caller would report nothing running", + ); +}); + +test("listing networks fails rather than reporting none", async () => { + await assert.rejects( + () => taggedNetworks(), + "a failed `incus network list` came back as an empty list", + ); +}); + +test("but a question whose failure genuinely means no still answers no", async () => { + // The distinction worth keeping. `instanceExists` asks about one named thing, and a daemon + // that will not answer is not evidence the instance exists — so false is honest here, and + // making this throw too would be over-correcting until nothing can be asked at all. + assert.equal(await instanceExists("anything"), false); +}); diff --git a/test/integration/assigned-audit.test.ts b/test/integration/assigned-audit.test.ts new file mode 100644 index 0000000..e94365c --- /dev/null +++ b/test/integration/assigned-audit.test.ts @@ -0,0 +1,221 @@ +/** + * The mesh assigns the audit logger, and it consumes over an account the mesh delivered. + * + * events.test.ts proves the events path with the audit logger started by hand. This proves the + * whole of novox/hq ADR 0048: the module is assigned through the control plane, the mesh issues it + * a broker account scoped to what it consumes, seals it to the machine, and the host runs it as a + * container that connects over amqps with that account — never the broker's own. The trail filling + * is the proof the delivered, scoped credential authenticated and the subscription bound. + * + * It needs the host binary, the substrate bundle, and the runtime image stocked by the scenario: + * + * MESH_LAB_HOST_BINARY=.../mesh-host + * MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-runtime-image.sh builds mesh-runtime-audit:development into the local daemon, + * which scenarios/audit-node.yml stocks — so no MESH_LAB_RUNTIME here; the host pulls it. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "audit-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +/** What the scenario's registry serves, by digest. */ +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +/** The control plane, a container on the node. */ +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +/** The pinned reference for one of the scenario's images, by repository. */ +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +/** The substrate bundle, its image references pointed at this scenario's own registry. */ +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + // Raise the substrate — store, broker, control — from the bundle. + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + // The node joins its own mesh, so it is a node the mesh can assign to, and start the host so it + // applies what it is pushed. + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("the mesh assigns the audit logger, and it consumes over the account the mesh delivered", { + skip, timeout: 900_000, +}, async () => { + // The assigned-module manifest (mesh-catalog), its runtime image the digest this registry serves. + const manifest = JSON.stringify({ + module: "audit-logger", + version: "1", + consumes: ["#"], + "own-secrets": { broker: "/var/lib/audit-logger/broker" }, + resources: [ + { id: "state", type: "directory", path: "/var/lib/audit-logger", mode: "0700" }, + { id: "trail", type: "directory", path: "/var/lib/audit-logger/trail", mode: "0700" }, + { + id: "run", type: "container", name: "mesh-audit-logger", image: pinned("mesh-runtime-audit"), + network: "host", + volumes: [ + "/var/lib/audit-logger/broker:/run/secrets/broker:ro", + "/var/lib/audit-logger/trail:/trail", + ], + env: { MESH_BROKER_FILE: "/run/secrets/broker", AUDIT_LOG: "/trail/audit.log" }, + }, + ], + }); + await must(`printf %s ${quote(manifest)} > /tmp/audit.json && docker cp /tmp/audit.json mesh-control:/audit.json`); + await mesh("module add /audit.json"); + + // The mesh issues its scoped account and seals it to this machine, then assigns and pushes it. + const issued = await mesh(`module issue audit-logger --node ${MACHINE}`); + assert.match(issued, /scoped to what it emits and consumes/, issued); + await mesh(`assign ${MACHINE} audit-logger`); + await mesh(`push ${MACHINE}`); + await settled(); + + // The container the mesh started is running. + const running = await must(`docker ps --format '{{.Names}}'`); + assert.match(running, /mesh-audit-logger/, + `the audit logger was assigned and is not running:\n${(await on(`tail -30 /var/log/mesh-host.log`)).out}`); + + // The credential on disk is the scoped account over amqps, sealed — not the broker's own. + const credential = await must(`cat /var/lib/audit-logger/broker`); + assert.match(credential, /"url":"amqps:\/\/anchor-audit-logger:/, `not the scoped account:\n${credential}`); + assert.doesNotMatch(credential, /guest:guest/, "the audit logger holds the broker's own account"); + assert.match(credential, /"fingerprint":"(sha256:)?[0-9a-f]{64}"/, "no fingerprint to pin the broker"); + + // An event, emitted by a probe over the bootstrap account (the audit logger's own may not emit). + await must( + `docker run --rm --network host -e MESH_BROKER_URL=amqp://guest:guest@127.0.0.1:5672/ ` + + `-e MESH_MODULE=probe -e MESH_NODE=${MACHINE} ${pinned("mesh-runtime-audit")} ` + + `emit module.probe.site.created '{"domain":"my-app"}'`, + ); + + // It reaches the trail the assigned container writes — proof its delivered account authenticated. + let line: Record | undefined; + const until = Date.now() + 60_000; + while (Date.now() < until) { + const raw = await must(`cat /var/lib/audit-logger/trail/audit.log 2>/dev/null || true`); + line = raw.split("\n").map((l) => l.trim()).filter(Boolean).map((l) => JSON.parse(l) as Record) + .find((e) => e.type === "module.probe.site.created"); + if (line) break; + await new Promise((r) => setTimeout(r, 3000)); + } + assert.ok(line, `the event never reached the trail:\n${(await on(`docker logs mesh-audit-logger 2>&1 | tail -20`)).out}`); + assert.equal(line.source, "probe"); + assert.deepEqual(line.body, { domain: "my-app" }); + + // And the account the mesh made for it is a real one on the broker — the trail above already + // proved it authenticated and read its queue. That it reaches no further than its own queue is + // the scope CreateModuleAccount applies, checked as patterns in mesh-control's own tests. + const users = await must(`docker exec mesh-broker lavinmqctl list_users 2>&1`); + assert.match(users, /anchor-audit-logger/, `the scoped account is not on the broker:\n${users}`); +}); diff --git a/test/integration/assigned-grafana.test.ts b/test/integration/assigned-grafana.test.ts new file mode 100644 index 0000000..0b0b37f --- /dev/null +++ b/test/integration/assigned-grafana.test.ts @@ -0,0 +1,216 @@ +/** + * The mesh assigns grafana's tool runtime, configured entirely by the assignment's settings — the + * ADR 0051 + 0052 case: config is the assignment's, delivered as a settings-merged file the runtime + * reads, not a credential baked into the manifest. + * + * plex/sonarr prove a runtime that self-detects its key from the app's own config. This proves the + * other half: the operator states grafana's URL and an API token as settings for this node, the + * control plane merges them into the module's mergeable config file, and the runtime reads that file + * at start, registers grafana's tools, and serves them under its scoped account. There is no live + * Grafana — that the serve queue is bound is the proof the settings reached the runtime and its + * tools loaded from them. + * + * MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-module-runtime.sh grafana builds mesh-runtime-grafana:development into the local + * daemon, which scenarios/grafana-node.yml stocks — so no MESH_LAB_RUNTIME here; the host pulls it. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "grafana-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("the mesh assigns grafana's runtime, configured by settings, and it serves its tools", { + skip, timeout: 900_000, +}, async () => { + // A grafana manifest with no credential in it: its runtime, and a mergeable config file the + // settings will fill. This is the whole point of ADR 0051 — the manifest carries defaults and + // structure, the assignment carries the URL and token. + const manifest = JSON.stringify({ + module: "grafana", + version: "1", + emits: ["module.grafana.alert.firing"], + "own-secrets": { broker: "/var/lib/mesh/grafana/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/grafana", mode: "0700" }, + { id: "config", type: "file", path: "/var/lib/mesh/grafana/config.json", mode: "0600", content: "{}\n", merge: "json" }, + { + id: "runtime", type: "container", name: "mesh-grafana", image: pinned("mesh-runtime-grafana"), + network: "host", + volumes: [ + "/var/lib/mesh/grafana/broker:/run/secrets/broker:ro", + "/var/lib/mesh/grafana/config.json:/run/config/config.json:ro", + ], + env: { + MESH_BROKER_FILE: "/run/secrets/broker", + MESH_GRAFANA_CONFIG_FILE: "/run/config/config.json", + }, + }, + ], + }); + await must(`printf %s ${quote(manifest)} > /tmp/grafana.json && docker cp /tmp/grafana.json mesh-control:/grafana.json`); + await mesh("module add /grafana.json"); + + // The operator states grafana's URL and API token as settings for this node — the config the + // runtime will read. Nothing about them is in the manifest. + const settings = JSON.stringify({ url: "http://127.0.0.1:3000", token: "lab-grafana-token" }); + await must(`printf %s ${quote(settings)} > /tmp/grafana-settings.json && docker cp /tmp/grafana-settings.json mesh-control:/grafana-settings.json`); + await mesh(`settings set grafana /grafana-settings.json --node ${MACHINE}`); + + const issued = await mesh(`module issue grafana --node ${MACHINE}`); + assert.match(issued, /scoped to what it emits and consumes/, issued); + await mesh(`assign ${MACHINE} grafana`); + await mesh(`push ${MACHINE}`); + await settled(); + + const running = await must(`docker ps --format '{{.Names}}'`); + assert.match(running, /mesh-grafana/, + `grafana's runtime was assigned and is not running:\n${(await on(`tail -30 /var/log/mesh-host.log`)).out}`); + + // The settings reached the node: the rendered config file carries what was set, not the manifest's + // empty default. + const config = await must(`cat /var/lib/mesh/grafana/config.json`); + assert.match(config, /lab-grafana-token/, `the settings did not merge into the config file:\n${config}`); + + const credential = await must(`cat /var/lib/mesh/grafana/broker`); + assert.match(credential, /"url":"amqps:\/\/anchor-grafana:/, `not the scoped account:\n${credential}`); + assert.doesNotMatch(credential, /guest:guest/, "grafana's runtime holds the broker's own account"); + + // The runtime read that config, built its client from the settings-provided token, registered its + // tools, and bound their serve queues — the queue on the broker is the proof the settings-config + // path reached serving, with no credential in the manifest and no live Grafana. + let served = ""; + const untilServing = Date.now() + 60_000; + while (Date.now() < untilServing) { + served = await must(`docker exec mesh-broker lavinmqctl list_queues name 2>&1 || true`); + if (/serve\.grafana\.grafana_status/.test(served)) break; + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(served, /serve\.grafana\.grafana_status/, + `grafana's runtime never bound its serve queue (settings not read?):\n` + + `${(await on(`docker logs mesh-grafana 2>&1 | tail -20`)).out}\n---\n${served}`); + + const users = await must(`docker exec mesh-broker lavinmqctl list_users 2>&1`); + assert.match(users, /anchor-grafana/, `the scoped account is not on the broker:\n${users}`); +}); diff --git a/test/integration/assigned-plex.test.ts b/test/integration/assigned-plex.test.ts new file mode 100644 index 0000000..de5b638 --- /dev/null +++ b/test/integration/assigned-plex.test.ts @@ -0,0 +1,237 @@ +/** + * The mesh assigns plex's tool runtime, and it serves plex's tools over an account the mesh + * delivered — the whole of novox/hq ADR 0052. + * + * assigned-audit proves an assigned *consumer* (ADR 0048). This proves an assigned module that runs + * its OWN code as its OWN process under its OWN scoped account and *serves tools*: the module is + * assigned through the control plane, the mesh issues it an account scoped to serve.plex.* (and its + * events), seals it to the machine, and the host runs it as a container that binds amqps with that + * account. A caller then invokes plex.plex_reachable over the mesh and gets the tool's own answer — + * proof the invocation routed to the assigned runtime, ran plex's real code, and replied, all under + * the scoped account and never the broker's own. + * + * It needs the host binary, the substrate bundle, and the runtime image stocked by the scenario: + * + * MESH_LAB_HOST_BINARY=.../mesh-host + * MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-module-runtime.sh plex builds mesh-runtime-plex:development into the local daemon, + * which scenarios/plex-node.yml stocks — so no MESH_LAB_RUNTIME here; the host pulls it. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "plex-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +/** What the scenario's registry serves, by digest. */ +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +/** The control plane, a container on the node. */ +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +/** The pinned reference for one of the scenario's images, by repository. */ +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +/** The substrate bundle, its image references pointed at this scenario's own registry. */ +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + // Raise the substrate — store, broker, control — from the bundle. + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + // The node joins its own mesh, so it is a node the mesh can assign to, and start the host so it + // applies what it is pushed. + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("the mesh assigns plex's runtime, and it serves plex's tools over the account the mesh delivered", { + skip, timeout: 900_000, +}, async () => { + // A minimal plex manifest: its tools/events runtime (no Plex server or media mounts in the lab), + // its emits and consumes so the account is scoped to those too, and a token in the environment so + // the tools register without a running Plex to detect one from. The runtime image is the digest + // this scenario's registry serves. + const manifest = JSON.stringify({ + module: "plex", + version: "1", + emits: [ + "module.plex.playback.started", + "module.plex.playback.stopped", + "module.plex.item.added", + ], + consumes: ["module.*.download.completed"], + "own-secrets": { broker: "/var/lib/mesh/plex/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/plex", mode: "0700" }, + { + id: "runtime", type: "container", name: "mesh-plex", image: pinned("mesh-runtime-plex"), + network: "host", + volumes: ["/var/lib/mesh/plex/broker:/run/secrets/broker:ro"], + env: { + MESH_BROKER_FILE: "/run/secrets/broker", + MESH_PLEX_URL: "http://127.0.0.1:32400", + MESH_PLEX_TOKEN: "lab-token", + }, + }, + ], + }); + await must(`printf %s ${quote(manifest)} > /tmp/plex.json && docker cp /tmp/plex.json mesh-control:/plex.json`); + await mesh("module add /plex.json"); + + // The mesh issues plex's scoped account and seals it to this machine, then assigns and pushes it. + const issued = await mesh(`module issue plex --node ${MACHINE}`); + assert.match(issued, /scoped to what it emits and consumes/, issued); + await mesh(`assign ${MACHINE} plex`); + await mesh(`push ${MACHINE}`); + await settled(); + + // The runtime container the mesh started is running. + const running = await must(`docker ps --format '{{.Names}}'`); + assert.match(running, /mesh-plex/, + `plex's runtime was assigned and is not running:\n${(await on(`tail -30 /var/log/mesh-host.log`)).out}`); + + // The credential on disk is the scoped account over amqps, sealed — not the broker's own. + const credential = await must(`cat /var/lib/mesh/plex/broker`); + assert.match(credential, /"url":"amqps:\/\/anchor-plex:/, `not the scoped account:\n${credential}`); + assert.doesNotMatch(credential, /guest:guest/, "plex's runtime holds the broker's own account"); + assert.match(credential, /"fingerprint":"(sha256:)?[0-9a-f]{64}"/, "no fingerprint to pin the broker"); + + // The runtime registered and is serving its tools — the queue it declared is on the broker, + // named for the scope its account is granted (serve.plex.*). + let served = ""; + const untilServing = Date.now() + 60_000; + while (Date.now() < untilServing) { + served = await must(`docker exec mesh-broker lavinmqctl list_queues name 2>&1 || true`); + if (/serve\.plex\.plex_reachable/.test(served)) break; + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(served, /serve\.plex\.plex_reachable/, + `plex's runtime never bound its serve queue:\n${(await on(`docker logs mesh-plex 2>&1 | tail -20`)).out}\n---\n${served}`); + + // A caller invokes plex.plex_reachable over the mesh, from the bootstrap account (a caller, like + // mesh-control's command API — plex's own account serves, it does not call). The reply is the + // tool's own answer: it ran in the assigned runtime and reported the Plex server is unreachable + // (there is none in the lab). A reply at all — not a timeout — is the proof the invocation routed + // to the assigned runtime and ran plex's real code under its scoped account. + const invoked = await must( + `docker run --rm --network host -e MESH_BROKER_URL=amqp://guest:guest@127.0.0.1:5672/ ` + + `${pinned("mesh-runtime-plex")} invoke plex plex_reachable`, + 120_000, + ); + const line = invoked.split("\n").map((l) => l.trim()).filter(Boolean).pop() ?? ""; + const result = JSON.parse(line) as { reachable: boolean; url: string; error?: string }; + assert.equal(result.reachable, false, `expected the lab's Plex to be unreachable:\n${invoked}`); + assert.match(result.url, /127\.0\.0\.1:32400/, `the tool ran but not against the configured server:\n${invoked}`); + + // And the account the mesh made for it is a real one on the broker, scoped — proven above by the + // serve queue authenticating and the invocation round-tripping under it. + const users = await must(`docker exec mesh-broker lavinmqctl list_users 2>&1`); + assert.match(users, /anchor-plex/, `the scoped account is not on the broker:\n${users}`); +}); diff --git a/test/integration/assigned-redis.test.ts b/test/integration/assigned-redis.test.ts new file mode 100644 index 0000000..574f031 --- /dev/null +++ b/test/integration/assigned-redis.test.ts @@ -0,0 +1,279 @@ +/** + * The mesh assigns redis — a *provider* — and its runtime runs the provisioner AND the tools as one + * process under one account the mesh delivered (novox/hq ADR 0052). + * + * assigned-plex proves an assigned module that serves tools. This proves the provider half: redis's + * runtime binds its scoped account, serves redis's tools against the real server (redis_ping → + * PONG), and — the thing 0052 fixes — runs its provisioner in that same broker-bound process, so a + * grant is provisioned and its lifecycle event is emitted onto the mesh. Before 0052 the provisioner + * ran in a container with no broker and its emit could not fire at all. + * + * A caveat this test makes explicit: runProvisioner needs a seal key ($MESH_SEAL_KEY) and the mesh + * has no way yet to deliver one to a provider's runtime (04-ISSUES). The manifest here sets a + * lab-local key so the mechanism can be proven; the delivery is a separate, open design question. + * + * It needs the host binary, the substrate bundle, and the runtime image stocked by the scenario: + * + * MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-module-runtime.sh redis builds mesh-runtime-redis:development into the local + * daemon, which scenarios/redis-node.yml stocks — so no MESH_LAB_RUNTIME here; the host pulls it. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "redis-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("the mesh assigns redis, and its runtime serves tools and provisions grants over the account the mesh delivered", { + skip, timeout: 900_000, +}, async () => { + // A redis manifest with both halves it needs on this node: the redis server, and one broker-bound + // runtime that serves redis's tools AND runs its provisioner. Both reach the server over the host + // (127.0.0.1:6379) with the same admin password the mesh generated. MESH_SEAL_KEY is lab-local — + // the mesh cannot yet deliver one to a provider's runtime (see the file header / 04-ISSUES). + const manifest = JSON.stringify({ + module: "redis", + version: "1", + emits: ["module.redis.cache.provisioned", "module.redis.cache.deprovisioned"], + // redis's events entrypoint subscribes to its own lifecycle events (an audit-trail log), so it + // consumes them too — declared, or the substrate never makes the queue the runtime binds and it + // crashes on start with a 404 (novox/hq ADR 0046: a consume is declared). + consumes: ["module.redis.cache.provisioned", "module.redis.cache.deprovisioned"], + "own-secrets": { default: "/var/lib/redis-module/default.secret", broker: "/var/lib/mesh/redis/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/redis", mode: "0700" }, + { id: "state", type: "directory", path: "/var/lib/redis-module", mode: "0700" }, + { id: "grants-dir", type: "directory", path: "/var/lib/redis-module/grants", mode: "0700" }, + { id: "data", type: "directory", path: "/services/redis/data", mode: "0700", owner: "999:999" }, + { + id: "server-conf", type: "file", path: "/var/lib/redis-module/redis.conf", mode: "0644", + content: "requirepass ${secret:default}\nappendonly no\ndir /data\n", + }, + { + id: "server", type: "container", name: "redis", image: pinned("redis"), network: "host", + volumes: [ + "/services/redis/data:/data", + "/var/lib/redis-module/redis.conf:/etc/redis/redis.conf:ro", + ], + args: ["/etc/redis/redis.conf"], + }, + { + id: "runtime", type: "container", name: "mesh-redis", image: pinned("mesh-runtime-redis"), + network: "host", + volumes: [ + "/var/lib/mesh/redis/broker:/run/secrets/broker:ro", + "/var/lib/redis-module/grants:/var/lib/redis-module/grants", + "/var/lib/redis-module/default.secret:/run/secrets/default:ro", + ], + env: { + MESH_BROKER_FILE: "/run/secrets/broker", + GRANTS: "/var/lib/redis-module/grants", + MESH_PROVISION_REDIS: "127.0.0.1:6379", + MESH_PROVISION_PASSWORD_FILE: "/run/secrets/default", + MESH_SEAL_KEY: "lab-only-seal-key", + }, + }, + ], + }); + await must(`printf %s ${quote(manifest)} > /tmp/redis.json && docker cp /tmp/redis.json mesh-control:/redis.json`); + await mesh("module add /redis.json"); + + const issued = await mesh(`module issue redis --node ${MACHINE}`); + assert.match(issued, /scoped to what it emits and consumes/, issued); + await mesh(`assign ${MACHINE} redis`); + await mesh(`push ${MACHINE}`); + await settled(); + + // The server and the runtime the mesh started are both running. + const running = await must(`docker ps --format '{{.Names}}'`); + assert.match(running, /\bredis\b/, `redis's server is not running:\n${(await on(`tail -30 /var/log/mesh-host.log`)).out}`); + assert.match(running, /mesh-redis/, `redis's runtime is not running:\n${(await on(`docker logs mesh-redis 2>&1 | tail -20`)).out}`); + + // The credential on disk is the scoped account over amqps, sealed — not the broker's own. + const credential = await must(`cat /var/lib/mesh/redis/broker`); + assert.match(credential, /"url":"amqps:\/\/anchor-redis:/, `not the scoped account:\n${credential}`); + assert.doesNotMatch(credential, /guest:guest/, "redis's runtime holds the broker's own account"); + assert.match(credential, /"fingerprint":"(sha256:)?[0-9a-f]{64}"/, "no fingerprint to pin the broker"); + + // The runtime registered and is serving its tools — the serve queue is on the broker. + let served = ""; + const untilServing = Date.now() + 60_000; + while (Date.now() < untilServing) { + served = await must(`docker exec mesh-broker lavinmqctl list_queues name 2>&1 || true`); + if (/serve\.redis\.redis_ping/.test(served)) break; + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(served, /serve\.redis\.redis_ping/, + `redis's runtime never bound its serve queue:\n${(await on(`docker logs mesh-redis 2>&1 | tail -30`)).out}\n---\n${served}`); + + // A caller invokes redis.redis_ping over the mesh: the tool runs in the assigned runtime, reaches + // the real redis, and answers PONG. A positive round-trip against a real backend. + const pinged = await must( + `docker run --rm --network host -e MESH_BROKER_URL=amqp://guest:guest@127.0.0.1:5672/ ` + + `${pinned("mesh-runtime-redis")} invoke redis redis_ping`, + 120_000, + ); + const pingLine = pinged.split("\n").map((l) => l.trim()).filter(Boolean).pop() ?? ""; + const ping = JSON.parse(pingLine) as { ok: unknown }; + assert.ok(String(ping.ok).toUpperCase().includes("PONG") || ping.ok === true, + `redis_ping did not answer PONG through the mesh:\n${pinged}`); + + // The provider path: a grant appears (as the control plane would write it), and the provisioner — + // running inside the same broker-bound runtime — creates the ACL user and emits the lifecycle + // event. The sealed credential the harness writes only after adapter.create() returns is the + // proof create() ran to completion; and because emit() awaits the broker's publish confirm + // (ADR 0047), a completed create() means the provisioned event was accepted onto the mesh. + const grant = JSON.stringify({ resource: "redis-cache", consumer: "app-one", node: MACHINE, values: {} }); + await must(`printf %s ${quote(grant)} > /var/lib/redis-module/grants/app-one.grant.json`); + + let credentialWritten = false; + const untilProvisioned = Date.now() + 60_000; + while (Date.now() < untilProvisioned) { + const ls = await on(`ls /var/lib/redis-module/grants/`); + if (ls.ok && /app-one\.redis-cache\.credential/.test(ls.out)) { credentialWritten = true; break; } + await new Promise((r) => setTimeout(r, 3000)); + } + assert.ok(credentialWritten, + `the provisioner never provisioned the grant (no emit under a bound broker?):\n` + + `${(await on(`docker logs mesh-redis 2>&1 | tail -30`)).out}`); + + // No emit failed: the provisioner's announce() logs "emit ... failed" only when the broker refused + // the publish. Its absence, with the credential written, is the provisioner emitting on the mesh. + const runtimeLog = (await on(`docker logs mesh-redis 2>&1`)).out; + assert.doesNotMatch(runtimeLog, /emit .*failed/, + `the provisioner's emit was refused — the account cannot publish its lifecycle event:\n${runtimeLog}`); + + // And the ACL user the provisioner created is really on the redis server — the provisioning did + // its own half, not only the mesh bookkeeping. Asked through the same served tool surface. + const acl = await must( + `docker run --rm --network host -e MESH_BROKER_URL=amqp://guest:guest@127.0.0.1:5672/ ` + + `${pinned("mesh-runtime-redis")} invoke redis redis_command '{"command":"ACL LIST"}'`, + 120_000, + ); + assert.match(acl, /app-one/, `the provisioner did not create the consumer's ACL user on redis:\n${acl}`); + + // The scoped account the mesh made for it is a real one on the broker. + const users = await must(`docker exec mesh-broker lavinmqctl list_users 2>&1`); + assert.match(users, /anchor-redis/, `the scoped account is not on the broker:\n${users}`); +}); diff --git a/test/integration/assigned-sonarr.test.ts b/test/integration/assigned-sonarr.test.ts new file mode 100644 index 0000000..1232d37 --- /dev/null +++ b/test/integration/assigned-sonarr.test.ts @@ -0,0 +1,220 @@ +/** + * The mesh assigns sonarr's tool runtime, and it serves sonarr's tools over an account the mesh + * delivered — the Servarr case of novox/hq ADR 0052. + * + * assigned-plex proved a tools+events module that self-detects its token from a mounted config dir. + * This proves that self-configuring pattern generalises to the Servarr family: sonarr's runtime + * detects its API key from the server's own config.xml (a file resource stands in for the running + * Sonarr here), registers its tools, and serves them under a scoped account. There is no live Sonarr + * to reach — that the serve queue is bound is the proof the key was detected and the tools loaded. + * + * MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-module-runtime.sh sonarr builds mesh-runtime-sonarr:development into the local + * daemon, which scenarios/sonarr-node.yml stocks — so no MESH_LAB_RUNTIME here; the host pulls it. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "sonarr-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("the mesh assigns sonarr's runtime, and it detects its key and serves its tools", { + skip, timeout: 900_000, +}, async () => { + // A minimal sonarr manifest: its tool runtime, and a config.xml the runtime detects its API key + // from — the file resource stands in for the running Sonarr that would write it. No Sonarr server + // or media mounts; the tools simply have nothing live to reach. + const manifest = JSON.stringify({ + module: "sonarr", + version: "1", + emits: ["module.sonarr.episode.grabbed", "module.sonarr.download.completed"], + consumes: [], + "own-secrets": { broker: "/var/lib/mesh/sonarr/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/sonarr", mode: "0700" }, + { id: "config", type: "directory", path: "/services/sonarr/config", mode: "0700" }, + { + id: "config-xml", type: "file", path: "/services/sonarr/config/config.xml", mode: "0644", + content: "\n 8989\n labdetectedapikey0000000000000000\n\n", + }, + { + id: "runtime", type: "container", name: "mesh-sonarr", image: pinned("mesh-runtime-sonarr"), + network: "host", + volumes: [ + "/var/lib/mesh/sonarr/broker:/run/secrets/broker:ro", + "/services/sonarr/config:/var/lib/sonarr/config:ro", + ], + env: { + MESH_BROKER_FILE: "/run/secrets/broker", + MESH_SONARR_URL: "http://127.0.0.1:8989", + MESH_SONARR_CONFIG_DIR: "/var/lib/sonarr/config", + }, + }, + ], + }); + await must(`printf %s ${quote(manifest)} > /tmp/sonarr.json && docker cp /tmp/sonarr.json mesh-control:/sonarr.json`); + await mesh("module add /sonarr.json"); + + const issued = await mesh(`module issue sonarr --node ${MACHINE}`); + assert.match(issued, /scoped to what it emits and consumes/, issued); + await mesh(`assign ${MACHINE} sonarr`); + await mesh(`push ${MACHINE}`); + await settled(); + + const running = await must(`docker ps --format '{{.Names}}'`); + assert.match(running, /mesh-sonarr/, + `sonarr's runtime was assigned and is not running:\n${(await on(`tail -30 /var/log/mesh-host.log`)).out}`); + + const credential = await must(`cat /var/lib/mesh/sonarr/broker`); + assert.match(credential, /"url":"amqps:\/\/anchor-sonarr:/, `not the scoped account:\n${credential}`); + assert.doesNotMatch(credential, /guest:guest/, "sonarr's runtime holds the broker's own account"); + assert.match(credential, /"fingerprint":"(sha256:)?[0-9a-f]{64}"/, "no fingerprint to pin the broker"); + + // The runtime detected its API key from config.xml, registered its tools, and bound their serve + // queues — the queue on the broker is the proof the whole chain worked with no live Sonarr. + let served = ""; + const untilServing = Date.now() + 60_000; + while (Date.now() < untilServing) { + served = await must(`docker exec mesh-broker lavinmqctl list_queues name 2>&1 || true`); + if (/serve\.sonarr\.sonarr_status/.test(served)) break; + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(served, /serve\.sonarr\.sonarr_status/, + `sonarr's runtime never bound its serve queue (key not detected?):\n` + + `${(await on(`docker logs mesh-sonarr 2>&1 | tail -20`)).out}\n---\n${served}`); + + // A caller invokes sonarr_status over the mesh: it routes to the assigned runtime, which runs + // sonarr's real code and reports Sonarr unreachable (there is none). A reply — not a timeout — is + // the proof the invocation reached the runtime under its scoped account. + const invoked = await on( + `docker run --rm --network host -e MESH_BROKER_URL=amqp://guest:guest@127.0.0.1:5672/ ` + + `${pinned("mesh-runtime-sonarr")} invoke sonarr sonarr_status`, + 120_000, + ); + assert.doesNotMatch(invoked.out, /timed out/, + `sonarr_status timed out — nothing served the invocation:\n${invoked.out}`); + + const users = await must(`docker exec mesh-broker lavinmqctl list_users 2>&1`); + assert.match(users, /anchor-sonarr/, `the scoped account is not on the broker:\n${users}`); +}); diff --git a/test/integration/builds.test.ts b/test/integration/builds.test.ts new file mode 100644 index 0000000..674645a --- /dev/null +++ b/test/integration/builds.test.ts @@ -0,0 +1,168 @@ +/** + * A machine in the mesh builds a module, and the mesh records what came out. + * + * The chain this closes: a repository exists, the mesh asks for it to be built, a build machine + * takes the work, publishes what it made, and the catalogue then says what the module is, which + * commit it came from, and — after the source moves — that it is behind. + * + * Against a real broker and a real registry, because what is under test is that four processes + * agree over a wire. Everything either side of the wire is already asserted in its own suite. + * + * MESH_LAB_HOST_BINARY a built mesh-host + * MESH_LAB_BUNDLE the substrate bundle + * MESH_LAB_BUILDER a built mesh-builder + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; +import { incus } from "../../src/incus/client.ts"; +import { machineName } from "../../src/lifecycle/names.ts"; + +const capability = await labIsUsable(); +const host = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; +const builder = process.env["MESH_LAB_BUILDER"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !host || !existsSync(host) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle" + : !builder || !existsSync(builder) + ? "MESH_LAB_BUILDER is not set to a built mesh-builder" + : false; + +const SCENARIO = "first-node"; +const MACHINE = "anchor"; +let instanceId = ""; +let registry = ""; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `${command} 2>&1; echo "__exit=$?"`, + ]); + const marker = stdout.lastIndexOf("__exit="); + return { out: stdout.slice(0, marker), ok: Number(stdout.slice(marker + 7).trim()) === 0 }; +} + +async function must(command: string): Promise { + const { out, ok } = await on(command); + if (!ok) throw new Error(`${command}\n${out}`); + return out; +} + +async function mesh(command: string): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`); +} + +/** The bundle, pointed at this scenario's own registry. */ +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const pinned of images) { + const repository = pinned.slice(pinned.indexOf("/") + 1, pinned.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), pinned); + } + return text; +} + +before(async () => { + if (skip) return; + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), {}); + instanceId = raised.instanceId; + const first = raised.images[0]; + assert.ok(first, "the scenario stocked no images, so there is no registry to publish to"); + registry = first.slice(0, first.indexOf("/")); + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`); + + // A module repository on the machine. Local rather than fetched, because what is under test is + // the mesh's chain and not whether the lab can reach a forge. + await must(`mkdir -p /root/shell/files`); + await must(`printf %s ${quote(JSON.stringify({ + module: "shell", + version: "1", + provides: ["login-shell"], + build: { artifacts: [{ name: "config", kind: "archive", from: "files" }] }, + resources: [ + { id: "package", type: "package", package: "zsh" }, + { id: "operator", type: "user", name: "operator", shell: "/bin/sh" }, + { + id: "dotfiles", type: "archive", artifact: "config", + path: "/home/operator/.config/shell", owner: "operator", + }, + ], + }))} > /root/shell/module.json`); + await must(`printf %s "alias ll='ls -l'\n" > /root/shell/files/aliases.zsh`); + await must(`cd /root/shell && git init -q . && git add -A && ` + + `git -c user.email=lab -c user.name=lab commit -qm first`); + + await incus([ + "file", "push", builder, `${machineName(instanceId, MACHINE)}/usr/local/bin/mesh-builder`, + "--mode", "0755", + ], 180_000); + // The build machine, holding its own broker credential and nothing else. + await must( + `MESH_BROKER_AMQP='amqp://guest:guest@127.0.0.1:5672/' MESH_REGISTRY=${registry} ` + + `MESH_WORKSPACE=/var/lib/mesh-builder ` + + `nohup /usr/local/bin/mesh-builder > /var/log/mesh-builder.log 2>&1 & sleep 3`, + ); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("the mesh asks, a machine builds, and the catalogue records it", { skip, timeout: 900_000 }, async () => { + const said = await mesh("build /root/shell --wait 300s"); + assert.match(said, /built on/, said); + assert.match(said, /config\s+archive/, `nothing was published:\n${said}`); + + const listed = await mesh("module list"); + assert.match(listed, /^shell\s+1\s+built [0-9a-f]{8}/m, listed); + + // And the artifact is really there, at the digest the manifest names. + const digest = /blobs\/(sha256:[0-9a-f]{64})/.exec(said); + assert.ok(digest, `the build named no digest:\n${said}`); + const head = await on(`curl -sfI ${registry}/v2/shell/config/blobs/${digest[1]} >/dev/null`); + assert.ok(head.ok, "the registry does not have the blob the manifest points at"); +}); + +test("a build that cannot succeed says why, and records nothing", { skip, timeout: 600_000 }, async () => { + // A failure is a result. A build that fails silently is indistinguishable from a builder that + // is not running, and those want completely different responses. + const { out, ok } = await on( + `docker exec mesh-control /mesh-control build /root/does-not-exist --wait 120s`, + ); + assert.equal(ok, false, "a build of nothing reported success"); + assert.match(out, /could not build/, out); + const listed = await mesh("module list"); + assert.doesNotMatch(listed, /does-not-exist/, "a failed build was recorded"); +}); + +test("when the source moves, the catalogue says the module is behind", { skip, timeout: 600_000 }, async () => { + await must(`cd /root/shell && printf %s "alias la='ls -la'\n" >> files/aliases.zsh && ` + + `git add -A && git -c user.email=lab -c user.name=lab commit -qm second`); + const moved = (await must(`cd /root/shell && git rev-parse HEAD`)).trim(); + + await mesh(`module moved shell ${moved}`); + assert.match(await mesh("module list"), /^shell\s+1\s+behind [0-9a-f]{8} < [0-9a-f]{8}/m); + + // And building again catches it up, with a different digest because the content differs. + const rebuilt = await mesh("build /root/shell --wait 300s"); + assert.match(rebuilt, new RegExp(`built on .* from ${moved.slice(0, 8)}`), rebuilt); + assert.match(await mesh("module list"), /^shell\s+1\s+built [0-9a-f]{8}/m); +}); diff --git a/test/integration/canary.test.ts b/test/integration/canary.test.ts new file mode 100644 index 0000000..7584f22 --- /dev/null +++ b/test/integration/canary.test.ts @@ -0,0 +1,177 @@ +/** + * The shortest run that would have caught today's faults. + * + * **A suite that takes forty minutes is a suite you find out from once a day.** Every fault found + * on 2026-09-01 — a module pinned to an image that does not exist, a consumer given a password and + * no name to present with it, a credential file nothing could read, a search for a password that + * read the password as an option — would have shown up in the first three minutes of it. The other + * thirty-seven proved things that were already working. + * + * So this runs first, on one machine, with the three images the mesh needs for itself and nothing + * else. If it fails there is no point spending the rest. + * + * It is deliberately *not* a smaller copy of the full suite. It walks one path end to end — a mesh + * comes up, a module reaches a machine, and a consumer gets a credential it can actually use — + * because that path is where everything went wrong, and a canary that checks many things shallowly + * is a canary nobody can read the failure of. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; + +import { raise } from "../../src/lifecycle/raise.ts"; +import { exec, destroy } from "../../src/lifecycle/operate.ts"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { labIsUsable } from "./harness.ts"; +import { pinnedInto } from "../../src/pinning.ts"; + +const capability = await labIsUsable(); +const host = process.env["MESH_LAB_HOST_BINARY"] ?? ""; +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !host || !existsSync(host) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle" + : false; + +const SCENARIO = "first-node"; +const MACHINE = "anchor"; +const HOST_PATH = "/usr/local/bin/mesh-host"; +let instanceId = ""; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + return { out: stdout.slice(0, marker), ok: Number(stdout.slice(marker + 7).trim()) === 0 }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +const mesh = (command: string, timeoutMs?: number) => + must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); + +before(async () => { + if (skip) return; + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${ + pinnedInto(readFileSync(bundle, "utf8"), raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 300_000); + + // **And the machine joins.** Applying the bundle raises a control plane; it does not tell that + // control plane a machine exists. Leaving this out is what the first run of this canary found, + // in under three minutes: the mesh answered, said "0 machine(s)", and every assignment after it + // failed with `no node of that name`. + await mesh(`node add ${MACHINE}`); + const said = await mesh(`token issue --node ${MACHINE}`); + const token = said.split("\n").map((l) => l.trim()) + .find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(token, `no token in:\n${said}`); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + + // The host has to be running for a push to reach it. + await must(`pgrep -x mesh-host >/dev/null || ` + + `(setsid nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 < /dev/null & sleep 3)`); +}); + +after(async () => { + // The canary owns its scenario and takes it down. Left standing it would hold a machine for + // the forty minutes of the run it exists to protect. + if (instanceId) await destroy(instanceId).catch(() => {}); +}); + +test("a mesh comes up and answers", { skip, timeout: 600_000 }, async () => { + const said = await mesh("status"); + assert.match(said, /anchor/, `the mesh does not know the machine it is running on:\n${said}`); +}); + +// One module, no images, nothing to stock. What is under test is the chain — added, assigned, +// resolved, planned, pushed, applied, reported — not what is at the end of it. +test("a module reaches the machine", { skip, timeout: 600_000 }, async () => { + await must(`printf %s ${quote(JSON.stringify({ + module: "canary", version: "1", + resources: [ + { id: "state", type: "directory", path: "/var/lib/canary", mode: "0700" }, + { id: "note", type: "file", path: "/var/lib/canary/it-arrived", content: "yes\n", mode: "0644" }, + ], + }))} > /tmp/canary.json`); + await must(`docker cp /tmp/canary.json mesh-control:/canary.json`); + await mesh("module add /canary.json"); + await mesh(`assign ${MACHINE} canary`); + await mesh(`push ${MACHINE}`, 300_000); + + assert.equal((await must(`cat /var/lib/canary/it-arrived`)).trim(), "yes"); + assert.match(await must(`stat -c %a /var/lib/canary`), /^700/); +}); + +// **The half that broke all day.** A provider and a consumer on one machine: the mesh makes a +// credential, tells the provider who asked, and gives the consumer a file it can read — with the +// name to present, which it could not have known (novox/hq 04-ISSUES/021, 022, 023). +test("a consumer gets a credential it can use", { skip, timeout: 600_000 }, async () => { + await must(`printf %s ${quote(JSON.stringify({ + module: "canary-store", version: "1", + provides: [{ name: "canary-database", scope: "mesh" }], + serves: { "canary-database": { port: 5432 } }, + receives: { "canary-database": "/var/lib/canary-store/asked.json" }, + grants: { "canary-database": "/var/lib/canary-store/grants" }, + resources: [ + { id: "state", type: "directory", path: "/var/lib/canary-store", mode: "0700" }, + { id: "grants", type: "directory", path: "/var/lib/canary-store/grants", mode: "0700" }, + ], + }))} > /tmp/canary-store.json`); + await must(`printf %s ${quote(JSON.stringify({ + module: "canary-app", version: "1", + requires: ["canary-database"], + contributes: { "canary-database": { name: "canaryapp" } }, + binds: { "canary-database": "/var/lib/canary-app/where.json" }, + secrets: { "canary-database": "/var/lib/canary-app/password" }, + resources: [ + { id: "state", type: "directory", path: "/var/lib/canary-app", mode: "0700" }, + { + id: "env", type: "file", path: "/var/lib/canary-app/database.env", mode: "0600", + content: "PGHOST=${bound:canary-database:at}\nPGPORT=${bound:canary-database:port}\n" + + "PGUSER=${bound:canary-database:as}\nPGPASSWORD=${secret:canary-database}\n", + }, + ], + }))} > /tmp/canary-app.json`); + for (const name of ["canary-store", "canary-app"]) { + await must(`docker cp /tmp/${name}.json mesh-control:/${name}.json`); + await mesh(`module add /${name}.json`); + await mesh(`assign ${MACHINE} ${name}`); + } + await mesh(`push ${MACHINE}`, 300_000); + + const password = (await must(`cat /var/lib/canary-app/password`)).trim(); + assert.ok(password.length >= 40, `the consumer's credential is ${password.length} characters`); + + // Every hole filled, and filled with the right thing. + const env = await must(`cat /var/lib/canary-app/database.env`); + assert.match(env, /^PGPORT=5432$/m, `the port did not arrive as a port:\n${env}`); + assert.match(env, /^PGUSER=mesh_[a-z0-9_]+_canary_app$/m, + `the consumer was not told what name to present:\n${env}`); + assert.doesNotMatch(env, /\$\{/, `a placeholder reached the machine as a value:\n${env}`); + assert.ok(env.includes(`PGPASSWORD=${password}`), + `the file holds a different password from the credential file:\n${env}`); + + // And the provider was told who asked, which is what makes the credential real. + const asked = await must(`cat /var/lib/canary-store/asked.json`); + assert.match(asked, /canary-app/, `the provider was not told who asked:\n${asked}`); + assert.match(asked, /"as": "mesh_[a-z0-9_]+_canary_app"/, + `the provider was not told what to call the login:\n${asked}`); +}); diff --git a/test/integration/certificates.test.ts b/test/integration/certificates.test.ts new file mode 100644 index 0000000..81f4d46 --- /dev/null +++ b/test/integration/certificates.test.ts @@ -0,0 +1,191 @@ +/** + * A public name, served with a certificate from an authority the mesh did not run. + * + * The mesh's own authority certifies `.internal` names and is proven elsewhere. This is the other + * half of the split: a name reachable from outside needs a certificate somebody else's browser + * already trusts, which means ordering one over ACME and answering a challenge **at the name being + * certified**. + * + * Against a real ACME server rather than a stub, for the reason the lab exists: what is under test + * is whether an order, a challenge and a handshake agree with each other, and a stub would be told + * to agree. + */ + +import { test, after, before } from "node:test"; +import assert from "node:assert/strict"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; +import { incus } from "../../src/incus/client.ts"; +import { machineName } from "../../src/lifecycle/names.ts"; + +const capability = await labIsUsable(); +const proxy = process.env["MESH_LAB_ROUTE_PROXY"] ?? ""; +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !proxy + ? "set MESH_LAB_ROUTE_PROXY to a built proxy (mesh-control: go build ./examples/route-proxy)" + : false; + +const SCENARIO = "a-public-name"; +const MACHINE = "anchor"; +const NAME = "photos.example"; +const ACME = "/var/lib/acme"; + +let instanceId = ""; + +function shellQuote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `${command} 2>&1; echo "__exit=$?"`, + ]); + const marker = stdout.lastIndexOf("__exit="); + return { out: stdout.slice(0, marker), ok: Number(stdout.slice(marker + 7).trim()) === 0 }; +} + +async function must(command: string): Promise { + const { out, ok } = await on(command); + if (!ok) throw new Error(`${command}\n${out}`); + return out; +} + +before(async () => { + if (skip) return; + const scenario = loadScenario(`scenarios/${SCENARIO}.yml`); + const instance = await raise(scenario, {}); + instanceId = instance.instanceId; + + const pebble = instance.images.find((r) => r.includes("pebble")); + assert.ok(pebble, `the scenario stocked no ACME server: ${instance.images.join(", ")}`); + + await must(`mkdir -p ${ACME}/cache`); + + // The authority's own API certificate is signed by a root nothing trusts yet. Taken out of the + // image rather than disabling verification, which is the same reason the proxy names a bundle: + // "skip" would still apply on the day this points at a public authority. + await must(`docker create --name pebble-certs ${pebble}`); + await must(`docker cp pebble-certs:/test/certs/pebble.minica.pem ${ACME}/authority-api.pem`); + await must(`docker rm pebble-certs`); + + // **The challenge must arrive on port 80**, which is where a proxy serving a public name + // listens. The authority's own default is 5002 — convenient for its test suite and wrong here, + // because the thing being proven is that the real path works. + // + // **Its own configuration, with one field changed.** The first version of this wrote a config + // from scratch and silently dropped two fields the default carries; the order then came back + // valid with no certificate to fetch, and the failure looked like a client bug. Take what works + // and change the one thing that must differ. + await must(`docker create --name pebble-config ${pebble}`); + await must(`docker cp pebble-config:/test/config/pebble-config.json ${ACME}/pebble.json`); + await must(`docker rm pebble-config`); + await must( + `python3 -c "import json,sys;` + + `c=json.load(open('${ACME}/pebble.json'));` + + `c['pebble']['httpPort']=80;` + + `json.dump(c,open('${ACME}/pebble.json','w'),indent=2)"`, + ); + + // The name resolves to this machine, so the authority's challenge reaches the proxy rather than + // whatever else on the internet answers to it. + await must(`grep -q ${shellQuote(NAME)} /etc/hosts || echo "127.0.0.1 ${NAME}" >> /etc/hosts`); + + await must( + `docker run -d --name acme --network host ` + + `-v ${ACME}/pebble.json:/test/config/pebble-config.json:ro ` + + `${pebble} -config /test/config/pebble-config.json -dnsserver 127.0.0.53:53`, + ); + + let up = false; + for (let i = 0; i < 60 && !up; i++) { + ({ ok: up } = await on( + `curl -sf --cacert ${ACME}/authority-api.pem https://127.0.0.1:14000/dir -o /dev/null`, + )); + if (!up) await new Promise((r) => setTimeout(r, 1000)); + } + assert.ok(up, `the ACME server never answered:\n${(await on(`docker logs acme`)).out}`); + + await incus([ + "file", "push", proxy, + `${machineName(instanceId, MACHINE)}/usr/local/bin/mesh-route-proxy`, + "--mode", "0755", + ], 180_000); + + // Something for the route to point at, so the proxy is serving a real name and not a hole. + await must( + `printf %s ${shellQuote(JSON.stringify({ + given: [{ from: "photos", node: "", at: "", values: { name: NAME, port: 8080 } }], + }))} > ${ACME}/routes.json`, + ); + await must( + `nohup sh -c 'while true; do printf "HTTP/1.1 200 OK\\r\\nContent-Length: 5\\r\\n\\r\\nhello" | nc -l -p 8080 -q 1; done' >/dev/null 2>&1 &`, + ); +}, { timeout: 1_200_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("a public name is served with a certificate the mesh did not issue", { + skip, timeout: 600_000, +}, async () => { + await must( + `ROUTES=${ACME}/routes.json LISTEN=:80 TLS_LISTEN=:443 ` + + `ACME_CACHE=${ACME}/cache ` + + `ACME_DIRECTORY=https://127.0.0.1:14000/dir ` + + `ACME_CA_BUNDLE=${ACME}/authority-api.pem ` + + `nohup /usr/local/bin/mesh-route-proxy >${ACME}/proxy.log 2>&1 & sleep 3`, + ); + + // The authority's issuing root, so the handshake can be checked rather than merely completed. + await must( + `curl -sf --cacert ${ACME}/authority-api.pem https://127.0.0.1:15000/roots/0 > ${ACME}/issuer.pem`, + ); + + // The first request is what triggers the order: autocert obtains on demand for a name its + // policy allows. Retried because ordering, the challenge and issuance take a moment. + let served = { out: "", ok: false }; + for (let i = 0; i < 40 && !served.ok; i++) { + served = await on(`curl -sf --cacert ${ACME}/issuer.pem https://${NAME}/ `); + if (!served.ok) await new Promise((r) => setTimeout(r, 2000)); + } + if (!served.ok) { + // Both sides, gathered before asserting. The proxy's log says what it tried; the authority's + // says whether it ever heard from it — and "the client never spoke to it" and "it refused + // what the client said" are different faults with nothing in common. + const proxyLog = (await on(`cat ${ACME}/proxy.log`)).out; + const authority = (await on(`docker logs acme 2>&1 | tail -40`)).out; + const directory = (await on( + `curl -s --cacert ${ACME}/authority-api.pem https://127.0.0.1:14000/dir`)).out; + assert.fail( + `the name was never served over TLS: ${served.out}\n\n` + + `── the proxy tried:\n${proxyLog}\n` + + `── the authority heard:\n${authority}\n` + + `── the directory it was pointed at:\n${directory}\n`); + } + assert.match(served.out, /hello/); + + // And it is the authority's certificate, not something self-signed that happens to work. + const issuer = await must( + `echo | openssl s_client -connect ${NAME}:443 -servername ${NAME} 2>/dev/null ` + + `| openssl x509 -noout -issuer -subject`, + ); + assert.match(issuer, /Pebble/i, `the certificate was not issued by the ACME server:\n${issuer}`); + assert.match(issuer, new RegExp(NAME), `the certificate is not for the name asked for:\n${issuer}`); +}); + +test("no certificate is ordered for a name the mesh does not route", { + skip, timeout: 300_000, +}, async () => { + // The policy that stops a quota being spent by a scan. Refused before any order is placed, so + // the authority never sees it. + const { out } = await on( + `echo | openssl s_client -connect 127.0.0.1:443 -servername nobody-asked-for-this.example 2>&1 | head -20`, + ); + assert.doesNotMatch(out, /Pebble/i, + `a certificate was obtained for a name nothing routes here:\n${out}`); +}); diff --git a/test/integration/events.test.ts b/test/integration/events.test.ts new file mode 100644 index 0000000..1264d47 --- /dev/null +++ b/test/integration/events.test.ts @@ -0,0 +1,221 @@ +/** + * An event a module emits reaches an audit trail, over the broker the mesh raised. + * + * Everything else here proves the broker carries *commands* — a declaration crosses it, a + * credential is delivered over it. This proves the other half of the bus (novox/hq ADR 0046): the + * events exchange, where a module emits and any number listen, and the audit logger consumes `#` + * and writes down what happened. It runs against the `mesh-broker` this scenario's own host raised + * from the substrate bundle — not a broker a test stood up — because "the mesh hosts the broker" + * (tier-1 substrate) is the thing being relied on. + * + * The wire shape it asserts is ADR 0047: metadata rides as headers so the body is only the + * payload, a consumer gets a durable per-consumer queue `..events`, and a + * dead-letter exchange `mesh.events.dead` stands behind it. The queue and that exchange existing + * on the raised broker is the check that the runtime provisioned the contract, not just that a + * message happened to arrive. + * + * It needs the host binary and the substrate bundle, like the mesh walk, plus a runtime image: + * + * MESH_LAB_HOST_BINARY=.../mesh-host + * MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * MESH_LAB_RUNTIME=.../mesh-runtime-audit.tar (docker save of the runtime+audit-logger image; + * built by scripts/build-runtime-image.sh) + * + * The runtime is run as a container against the broker here, rather than assigned through the + * control plane. Assigning it — so the mesh delivers its broker credential the way it does the + * builder's — is the next step; this proves the events path itself first. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; +import { incus } from "../../src/incus/client.ts"; +import { machineName } from "../../src/lifecycle/names.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; +const runtime = process.env["MESH_LAB_RUNTIME"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : !runtime || !existsSync(runtime) + ? "MESH_LAB_RUNTIME is not set to a runtime image tar (scripts/build-runtime-image.sh)" + : false; + +const SCENARIO = "first-node"; +const MACHINE = "anchor"; +/** Where the audit-logger container writes its trail, on the machine — mounted from a host dir. */ +const TRAIL_DIR = "/var/lib/mesh-audit"; +const TRAIL = `${TRAIL_DIR}/audit.log`; +/** The broker the substrate raised, reachable on the node's loopback (novox/hq ADR 0001). */ +const BROKER = "amqp://guest:guest@127.0.0.1:5672/"; + +let instanceId = ""; +/** The tag `docker load` reported for the runtime image, so the container names what was loaded. */ +let runtimeImage = ""; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +/** + * A command on the machine, its exit read from a marker on its own line. + * + * `exec 2>&1` on its own first line and the marker on its own last line, so a command that carries + * a heredoc — the substrate bundle is written with one — terminates where it says it does rather + * than swallowing the marker (the fault mesh.test.ts documents). + */ +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +/** + * The substrate bundle, its image references pointed at this scenario's own registry. + * + * A digest belongs to whatever registry serves it, so the committed bundle names a registry that + * is not this one; matching by repository and rewriting to the digest this registry assigned is + * what makes it applicable (the same rewrite mesh.test.ts does). + */ +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const pinned of images) { + const repository = pinned.slice(pinned.indexOf("/") + 1, pinned.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll( + new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), + pinned, + ); + } + return text; +} + +/** Read the trail back as parsed JSON lines. */ +async function trail(): Promise[]> { + const raw = await must(`cat ${TRAIL} 2>/dev/null || true`); + return raw + .split("\n") + .map((l) => l.trim()) + .filter(Boolean) + .map((l) => JSON.parse(l) as Record); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + + // The node raises its substrate — store, broker and the rest — from the bundle, applied from a + // file because the digests are this registry's and are not known until it is up. + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + + const running = await must(`docker ps --format '{{.Names}}'`); + assert.match(running, /mesh-broker/, `the substrate did not raise a broker:\n${running}`); + + // Bring the runtime+audit-logger image onto the machine. Loaded, not pulled: the machine has no + // route out (novox/hq the lab is a closed address space), so the image arrives as a tar the way + // the builder binary does, and `docker load` names what it loaded. + await incus([ + "file", "push", runtime, `${machineName(instanceId, MACHINE)}/tmp/runtime.tar`, "--mode", "0644", + ], 300_000); + const loaded = await must(`docker load < /tmp/runtime.tar`); + const named = loaded.match(/Loaded image:\s*(\S+)/)?.[1]; + assert.ok(named, `docker load did not name the image:\n${loaded}`); + runtimeImage = named; + + // Start the audit-logger: the runtime bound to the mesh's broker, consuming `#`, its trail on a + // mounted directory so the assertions read what it actually wrote. + await must(`mkdir -p ${TRAIL_DIR}`); + await must( + `docker run -d --name mesh-audit --network host ` + + `-e MESH_BROKER_URL=${quote(BROKER)} -e MESH_MODULE=audit-logger -e MESH_NODE=${MACHINE} ` + + `-e AUDIT_LOG=/trail/audit.log -v ${TRAIL_DIR}:/trail ` + + `${runtimeImage}`, + ); + // Give the subscription a moment to bind before anything is emitted at it. + await new Promise((r) => setTimeout(r, 4000)); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("an emitted event reaches the audit trail with its metadata in headers", { + skip, timeout: 300_000, +}, async () => { + // Emitted as a different module (source=probe), so the trail's source is the emitter's, read + // back from the header — proof it did not come from the body. + await must( + `docker exec -e MESH_MODULE=probe -e MESH_NODE=${MACHINE} mesh-audit ` + + `node dist/main.js emit module.probe.site.created '{"domain":"my-app"}'`, + ); + // A node-origin event too (novox/hq ADR 0047 reserves module.* / mesh.* / node.*), to show the + // audit sink takes all of them, not only a module's. + await must( + `docker exec -e MESH_MODULE=probe -e MESH_NODE=${MACHINE} mesh-audit ` + + `node dist/main.js emit node.${MACHINE}.tick '{"n":1}'`, + ); + + // Poll: the trail is written by a separate container reacting to the broker, so it is not there + // the instant emit returns. + let lines: Record[] = []; + const until = Date.now() + 30_000; + while (Date.now() < until) { + lines = await trail(); + if (lines.length >= 2) break; + await new Promise((r) => setTimeout(r, 2000)); + } + + const types = lines.map((l) => l.type); + assert.ok( + types.includes("module.probe.site.created") && types.includes(`node.${MACHINE}.tick`), + `both events did not reach the trail; it holds ${JSON.stringify(types)}\n` + + `${(await on(`docker logs mesh-audit 2>&1 | tail -20`)).out}`, + ); + + const site = lines.find((l) => l.type === "module.probe.site.created")!; + assert.equal(site.source, "probe", "x-source did not survive as the trail's source"); + assert.equal(site.node, MACHINE, "x-node did not survive"); + assert.ok(typeof site.id === "string" && site.id.length > 0, "no x-event-id was recorded"); + assert.deepEqual(site.body, { domain: "my-app" }, "the body was not exactly the payload"); +}); + +test("the runtime provisioned the ADR 0047 queue and dead-letter on the raised broker", { + skip, timeout: 120_000, +}, async () => { + // Asked of the broker itself, so this is the shape that actually exists on the bus, not the + // shape the code intends. A durable per-consumer queue and a dead-letter home are what make the + // trail survive a restart and set a poison event aside — assert they are there, not assumed. + const queues = await must(`docker exec mesh-broker lavinmqctl list_queues name durable`); + assert.match(queues, new RegExp(`${MACHINE}\\.audit-logger\\.events`), + `the durable per-consumer queue is missing:\n${queues}`); + + const exchanges = await must(`docker exec mesh-broker lavinmqctl list_exchanges name`); + assert.match(exchanges, /mesh\.events(\s|$)/m, `the events exchange is missing:\n${exchanges}`); + assert.match(exchanges, /mesh\.events\.dead/, `the dead-letter exchange is missing:\n${exchanges}`); +}); diff --git a/test/integration/harness.ts b/test/integration/harness.ts index 6f9f8c0..1e06547 100644 --- a/test/integration/harness.ts +++ b/test/integration/harness.ts @@ -1,7 +1,7 @@ /** * Integration tests run against a real hypervisor. Mocking it is forbidden — a test that * fakes the system under integration asserts that the fake behaves as expected, which is - * the shape of test this project exists to stop shipping (novox/hq ADR 0034). + * the shape of test this project exists to stop shipping (novox/hq ADR 0017). * * Consequence, accepted: these are slow, and they need a machine that can raise scenarios. * They skip rather than fail where it cannot, so that a machine without a hypervisor gets diff --git a/test/integration/mesh-grant-end-to-end.test.ts b/test/integration/mesh-grant-end-to-end.test.ts new file mode 100644 index 0000000..1543ef5 --- /dev/null +++ b/test/integration/mesh-grant-end-to-end.test.ts @@ -0,0 +1,261 @@ +/** + * The whole grant, mesh-driven end to end — novox/hq ADR 0053 with nothing hand-written. + * + * The earlier provider tests put the contributions file and the password on disk by hand, standing + * in for the control plane. This one does not: a provider (redis) and a consumer (a module that + * `requires` redis-cache) are both assigned, and the *mesh* mints the password, seals a copy to each + * end, writes redis its contributions file and the consumer its bound file, and the host unseals + * each side's secret. redis's provisioner — reading only what the mesh wrote — creates the ACL user + * under the login the mesh derived, with the password the mesh minted. The proof is the consumer's + * end: the credential the mesh delivered *it* authenticates against the login redis created for it. + * Mint on one side and create on the other agreeing, with no shared key and nothing placed by the + * test, is the entire provider/consumer contract working as one thing. + * + * MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-module-runtime.sh redis builds mesh-runtime-redis:development, which + * scenarios/redis-node.yml stocks. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "redis-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("the mesh grants a consumer redis's cache, and the credential it delivers authenticates", { + skip, timeout: 900_000, +}, async () => { + // The PROVIDER: redis in its committed shape — server and a broker-bound runtime on the private + // redis network, the runtime running the provisioner. + const redisManifest = JSON.stringify({ + module: "redis", + version: "1", + provides: [{ name: "redis-cache", scope: "mesh" }], + serves: { "redis-cache": {} }, + emits: ["module.redis.cache.provisioned", "module.redis.cache.deprovisioned"], + consumes: ["module.redis.cache.provisioned", "module.redis.cache.deprovisioned"], + receives: { "redis-cache": "/var/lib/redis-module/grants/mesh.json" }, + grants: { "redis-cache": "/var/lib/redis-module/grants" }, + "own-secrets": { default: "/var/lib/redis-module/default.secret", broker: "/var/lib/mesh/redis/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/redis", mode: "0700" }, + { id: "state", type: "directory", path: "/var/lib/redis-module", mode: "0700" }, + { id: "grants-dir", type: "directory", path: "/var/lib/redis-module/grants", mode: "0700" }, + { id: "data", type: "directory", path: "/services/redis/data", mode: "0700", owner: "999:999" }, + { + id: "server-conf", type: "file", path: "/var/lib/redis-module/redis.conf", mode: "0644", + content: "requirepass ${secret:default}\nappendonly no\ndir /data\n", + }, + { id: "net", type: "network", name: "redis" }, + { + id: "server", type: "container", name: "redis", image: pinned("redis"), network: "redis", + ports: ["6379"], + volumes: ["/services/redis/data:/data", "/var/lib/redis-module/redis.conf:/etc/redis/redis.conf:ro"], + args: ["/etc/redis/redis.conf"], + }, + { + id: "runtime", type: "container", name: "mesh-redis", image: pinned("mesh-runtime-redis"), + network: "redis", + volumes: [ + "/var/lib/mesh/redis/broker:/run/secrets/broker:ro", + "/var/lib/redis-module/grants:/var/lib/redis-module/grants:ro", + "/var/lib/redis-module/default.secret:/run/secrets/default:ro", + ], + env: { + MESH_BROKER_FILE: "/run/secrets/broker", + MESH_RECEIVES: "/var/lib/redis-module/grants/mesh.json", + MESH_PROVISION_REDIS: "redis:6379", + MESH_PROVISION_PASSWORD_FILE: "/run/secrets/default", + }, + }, + ], + }); + + // The CONSUMER: a module that requires redis-cache and no more. It runs no code here — the mesh + // delivers it a bound file (where redis is, and the login to present) and its sealed password, + // which the host unseals onto the machine. That delivery is exactly what a real consumer reads. + const consumerManifest = JSON.stringify({ + module: "cacheuser", + version: "1", + requires: ["redis-cache"], + // `contributes` (not just `requires`) is what makes a consumer *ask* — the grant forms from a + // contribution. It must be non-empty; redis's provisioner ignores the value (it uses the login + // the mesh derives), so the name is only what marks this module as wanting a cache. + contributes: { "redis-cache": { name: "cacheuser" } }, + binds: { "redis-cache": "/var/lib/cacheuser/redis.json" }, + secrets: { "redis-cache": "/var/lib/cacheuser/redis.secret" }, + resources: [{ id: "state", type: "directory", path: "/var/lib/cacheuser", mode: "0700" }], + }); + + await must(`printf %s ${quote(redisManifest)} > /tmp/redis.json && docker cp /tmp/redis.json mesh-control:/redis.json`); + await mesh("module add /redis.json"); + await mesh(`module issue redis --node ${MACHINE}`); + await mesh(`assign ${MACHINE} redis`); + + await must(`printf %s ${quote(consumerManifest)} > /tmp/cacheuser.json && docker cp /tmp/cacheuser.json mesh-control:/cacheuser.json`); + await mesh("module add /cacheuser.json"); + await mesh(`assign ${MACHINE} cacheuser`); + + await mesh(`push ${MACHINE}`); + await settled(); + + // The mesh matched the two and wrote redis its contributions file — the test wrote nothing here. + const contributions = await must(`cat /var/lib/redis-module/grants/mesh.json`); + assert.match(contributions, /"as"/, `the mesh did not write redis a contributions file:\n${contributions}`); + + // The mesh delivered the consumer its bound file and its unsealed password. + let boundRaw = ""; + const untilBound = Date.now() + 60_000; + while (Date.now() < untilBound) { + const got = await on(`cat /var/lib/cacheuser/redis.json 2>/dev/null`); + if (got.ok && /"as"/.test(got.out)) { boundRaw = got.out; break; } + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(boundRaw, /"as"/, `the consumer was never told about its cache:\n${boundRaw}`); + const bound = JSON.parse(boundRaw) as { as: string; from: string; provision: string }; + assert.equal(bound.provision, "redis-cache"); + assert.ok(bound.from, `the consumer was not told which node serves its cache:\n${boundRaw}`); + const as = bound.as; + const password = (await must(`cat /var/lib/cacheuser/redis.secret`)).trim(); + assert.ok(as && password, `the consumer's login or password was empty (as=${as})`); + + // redis's provisioner, reading only the mesh's contributions, created the ACL user. Wait for it. + const adminPw = await must(`cat /var/lib/redis-module/default.secret`); + let acl = ""; + const untilAcl = Date.now() + 60_000; + while (Date.now() < untilAcl) { + acl = (await on(`docker exec redis redis-cli -a ${quote(adminPw)} --no-auth-warning ACL LIST 2>/dev/null`)).out; + if (acl.includes(as)) break; + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(acl, new RegExp(as.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")), + `redis never created the login the mesh granted (${as}):\n${(await on(`docker logs mesh-redis 2>&1 | tail -30`)).out}\n---\n${acl}`); + + // THE PROOF, from the consumer's side: the credential the mesh delivered *it* — the login from its + // bound file, the password from its secret — authenticates against the login redis created. Mint + // and create agreeing across the two ends, with no shared key and nothing the test placed. + const authed = await on(`docker exec redis redis-cli --user ${quote(as)} --pass ${quote(password)} --no-auth-warning PING 2>&1`); + assert.doesNotMatch(authed.out, /WRONGPASS|NOPERM|no password/i, + `the consumer's mesh-delivered credential did not authenticate — the two ends do not agree:\n${authed.out}`); + assert.match(authed.out, /PONG/, `expected PONG authenticating as the granted consumer:\n${authed.out}`); +}); diff --git a/test/integration/mesh.test.ts b/test/integration/mesh.test.ts new file mode 100644 index 0000000..5f2923f --- /dev/null +++ b/test/integration/mesh.test.ts @@ -0,0 +1,2095 @@ +/** + * A mesh, raised from nothing, joined by two machines, delivering a credential neither the mesh + * nor the broker can read. + * + * Everything before this proves a part. This proves the parts meet — which is the thing the + * project keeps saying cannot be checked any other way (novox/hq ADR 0001: every fault of + * 2026-08-22 was found in production because nothing could be stood up locally). + * + * It needs a host binary and the substrate bundle: + * + * MESH_LAB_HOST_BINARY=.../mesh-host + * MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * + * The bundle's image references are rewritten to the ones this scenario's own registry serves. + * A digest belongs to whatever registry serves it, so a committed bundle names a registry that is + * not this one — rewriting is what makes it applicable rather than a placeholder to tidy away. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { pinnedInto, stillUnpinned } from "../../src/pinning.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; +import { incus } from "../../src/incus/client.ts"; +import { machineName } from "../../src/lifecycle/names.ts"; +import { ready, returnTo, keep, rememberStock, warmStock } from "../../src/warm.ts"; + +/** Whether this run keeps its mesh for the next one. Off unless asked for. */ +const warming = process.env["MESH_LAB_WARM"] === "1"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; +const builder = process.env["MESH_LAB_BUILDER"] ?? ""; +/** mesh-control's `examples/modules`, so the manifests proven here are the ones that ship. */ +const moduleExamples = process.env["MESH_LAB_MODULES"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "two-nodes"; +let instanceId = ""; +/** The scenario's own registry, which serves the images a module may mirror. */ +let registry = ""; +/** What that registry actually serves, by repository. */ +let stocked: string[] = []; + +/** + * The pinned reference for one of the scenario's images. + * + * By digest, because the lab's registry drops tags when it stocks: `registry:2` is not there and + * asking for it fails with "not found", which reads like a missing image rather than a naming + * convention. A digest is also what a declaration pins, so this is the reference a module would + * really carry. + */ +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(machine: string, command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + // Each part on its own line, and stderr redirected once for the whole script. + // + // It was `${command} 2>&1; echo ...` on a single line, which quietly broke every command + // containing a heredoc: the terminator line became `MARKER 2>&1; echo ...`, matched nothing, and + // the heredoc swallowed the rest of the script — including the echo. `exec 2>&1` needs no + // trailing text on the command's last line, so a heredoc terminates where it says it does. + const { stdout } = await exec(instanceId, machine, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) { + // **Never success.** `Number("")` is 0, so a missing marker used to read as exit 0 — a + // command whose output was swallowed reported that it worked, which is the one answer a test + // harness must never give. + return { out: stdout, ok: false }; + } + const said = stdout.slice(marker + 7).trim(); + return { out: stdout.slice(0, marker), ok: said === "0" }; +} + +async function must(machine: string, command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(machine, command, timeoutMs); + if (!ok) throw new Error(`${machine}: ${command}\n${out}`); + return out; +} + +/** The control plane, which runs in a container on the first node. */ +async function mesh(command: string, timeoutMs?: number): Promise { + return must("anchor", `docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +/** + * Wait until a machine has actually applied what it was last sent. + * + * **`push` sends; it does not wait.** It prints "sent N resource(s)" and returns, and the machine + * applies afterwards. So asserting on what a machine is running immediately after a push is a race + * — and the one this fixes lost it silently: `status` still described the *previous* apply, so it + * reported nothing wrong while the containers from this declaration did not exist yet. + * + * Asked of the mesh rather than of the machine, and in its own terms. A node is settled when it is + * neither waiting for what it was sent nor wrong about what it applied — the same two questions + * `status` answers, read as JSON so a test is not parsing a report meant for a person. + * + * A machine that reports a failure ends this at once rather than on the timeout: it is not going + * to become right by being waited for, and a refusal read after two minutes of polling is the same + * refusal, later. + */ +async function settled(node: string, withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + // **Could not ask** and **asked, and the answer was bad** are different facts, and only the + // second is this machine's fault. The control plane is a container on the node being polled: + // while it applies a declaration, an exec into it can lose its fifo to containerd, and a poll + // loop that treats that as a verdict reports the mesh broken because the question missed. + // And a poll that *threw* — an exec timeout, a lost fifo — is also "could not ask", not a + // verdict. The distinction failed once as an IncusError surfacing at minute four of a wait + // whose machine was merely slow. + let state: { + wrong: { node: string; outcome: string; refused?: string; + failed?: { id: string; error: string }[] }[]; + waiting: { node: string; never: boolean }[]; + reported: { node: string; outcome: string; current: boolean }[]; + } | undefined; + let said = ""; + try { + const asked = await on("anchor", + `docker exec mesh-control /mesh-control status --json`); + said = asked.out; + // Parsed inside the try on purpose: a truncated answer from a struggling machine is the + // same fact as no answer, and the likeliest moment for one is exactly the machine this + // poll is watching. + if (asked.ok) state = JSON.parse(said); + } catch (err) { + said = (err as Error).message; + } + if (!state) { + last = said; + await new Promise((r) => setTimeout(r, 5000)); + continue; + } + + const bad = state.wrong.find((w) => w.node === node); + if (bad) { + const why = [bad.refused, ...(bad.failed ?? []).map((f) => `${f.id}: ${f.error}`)] + .filter(Boolean).join("\n "); + throw new Error(`${node} did not apply what it was sent (${bad.outcome}):\n ${why}`); + } + // Caught up is an equality, not an ordering. This compared timestamps once — report newer + // than send — and lost the race it invited: the previous test's closing apply reported + // after this test's push, newer and still about the old declaration. The report now names + // the declaration it applied, and the mesh says whether that is the one it last sent. + const word = state.reported.find((r) => r.node === node); + const acted = word?.outcome === "applied" && word.current; + if (!state.waiting.some((w) => w.node === node) && acted) return; + last = said; + // Unhurried on purpose: each poll is an exec into a container on the machine that is busy + // applying, and asking four times a minute rather than thirty is the difference between + // observing the apply and competing with it. + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${node} never caught up with what it was sent within ` + + `${Math.round(withinMs / 1000)}s. Last answer, or the reason there was none:\n${last}`); +} + +/** + * The bundle, with every image reference pointed at this scenario's registry. + * + * Matched by repository rather than by the whole reference, because the address and the digest + * both differ from whatever the committed bundle names — and a bundle that names the wrong + * registry is not wrong, it is built for a different target. + */ +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const pinned of images) { + const repository = pinned.slice(pinned.indexOf("/") + 1, pinned.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll( + new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), + pinned, + ); + } + return text; +} + +/** Take a token out of what `token issue` printed. It is the one base64url blob on its own line. */ +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +before(async () => { + if (skip) return; + + // A mesh kept between runs, when one is being kept and still counts. + // + // **Bootstrapping proves the same thing every time**, and the tests worth iterating on are the + // ones after it. Off by default: a run that is meant to mean something raises from nothing, + // because "it passes" must not come to mean "it passes against a mesh somebody bootstrapped + // last week". + if (warming) { + const said = await ready(SCENARIO); + if (said.use === "restore") { + instanceId = said.instanceId; + const seconds = await returnTo(instanceId); + stocked = warmStock(instanceId).images; + + // **A snapshot captures disk, not memory.** Restoring reboots the machine, so everything + // this suite started by hand is gone — the host most of all. Without it the mesh looks + // perfectly healthy from the control plane's side: a module is assigned, a declaration is + // sent and recorded, and nothing on the machine is listening to apply it. That is exactly + // how this was first met, and it cost an hour to see. + // + // The real answer is a host started by init, which is what the design says it is anyway + // (novox/hq 05-the-node-host: a root service, installed as a package). Until the lab places + // it that way, the warm path restarts what it knows it started. + for (const machine of ["anchor", "laptop"]) { + await must(machine, `pgrep -x mesh-host >/dev/null || ` + + `(nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3)`); + } + const running = await on("anchor", `pgrep -x mesh-host >/dev/null && echo yes || echo no`); + assert.equal(running.out.trim(), "yes", + "the host did not come back after a restore, so nothing would apply anything"); + + console.log(`warm: returned ${instanceId} to its state in ${seconds.toFixed(1)}s, ` + + `and started the host again`); + return; + } + console.log(`warm: raising fresh — ${said.why}`); + } + + // **Progress is printed, and that is not decoration** (novox/hq 04-ISSUES/024). A raise takes + // minutes and said nothing until it finished, so a stall and ordinary work were the same + // thing to look at — and the one time it mattered, thirty-five minutes of nothing was read as + // a slow test until somebody went and looked inside the machine. + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + + // The first node raises everything from a file rather than from a bundle built into the binary, + // because the digests are this registry's and are not known until it is up. + await must("anchor", `cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must("anchor", `${HOST_PATH} apply /tmp/substrate.lock`); + stocked = raised.images; + const first = raised.images[0]; + assert.ok(first, "the scenario stocked no images, so nothing can be mirrored"); + registry = first.slice(0, first.indexOf("/")); + + // A build machine, so anything here can ask the mesh to build something. Placed rather than + // assumed: nothing else in this scenario would start one. + if (builder) { + await incus([ + "file", "push", builder, `${machineName(instanceId, "anchor")}/usr/local/bin/mesh-builder`, + "--mode", "0755", + ], 180_000); + await must("anchor", `mkdir -p /var/lib/mesh-builder`); + await must("anchor", + `MESH_BROKER_AMQP='amqp://guest:guest@127.0.0.1:5672/' MESH_REGISTRY=${registry} ` + + `MESH_WORKSPACE=/var/lib/mesh-builder ` + + `nohup /usr/local/bin/mesh-builder > /var/log/mesh-builder.log 2>&1 & sleep 3`); + } + if (warming) { + // Snapshotted only now, with everything up: a state worth returning to is the one after the + // part nobody wants to repeat. + await rememberStock(instanceId, stocked); + const warm = await keep(SCENARIO, instanceId); + console.log(`warm: ${warm.instanceId} kept, against ` + + Object.entries(warm.against).map(([n, c]) => `${n} ${c}`).join(", ")); + } +}, { timeout: 1_800_000 }); + +after(async () => { + // A kept instance survives on purpose, and `mesh-lab warm cool` is how it goes away. Everything + // else is destroyed, because an instance nobody meant to keep is one nobody will remember. + if (warming) return; + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("a bare machine becomes a mesh", { skip, timeout: 600_000 }, async () => { + const running = await must("anchor", `docker ps --format '{{.Names}}'`); + for (const container of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(running, new RegExp(container), `${container} is not running`); + } + // Answering, not merely up. A container that is running is not a control plane that replies — + // a distinction this project has already paid for once. + assert.ok((await mesh("status")).length > 0); +}); + +test("both machines join it, and the token is all they need", { skip, timeout: 900_000 }, async () => { + for (const [machine, node] of [["anchor", "anchor"], ["laptop", "laptop"]] as const) { + await mesh(`node add ${node}`); + const token = tokenFrom(await mesh(`token issue --node ${node}`)); + // No --name. The token says what the mesh calls the machine, which is the fault this walk + // found the first time it was run. + const said = await must(machine, `${HOST_PATH} enrol --token ${quote(token)}`); + assert.match(said, new RegExp(`enrolled as ${node}`), said); + assert.match(said, /sealing key/, "no sealing key was generated"); + } + + const recorded = await must("anchor", + `docker exec mesh-store psql -U postgres -d inventory -qAt ` + + `-c "select name from node where sealing_key is not null order by name"`); + assert.equal(recorded.trim().split("\n").map((l) => l.trim()).sort().join(","), "anchor,laptop", + "the mesh did not record a sealing key for both machines"); +}); + +test("a credential reaches both ends and the mesh holds neither", { skip, timeout: 900_000 }, async () => { + // The whole argument, on real machines: the two ends must hold the SAME password, and it must + // appear nowhere the mesh or the broker could read it. + await must("anchor", `printf %s '{"module":"postgres","version":"1",` + + `"provides":[{"name":"postgres-database","scope":"mesh"}],"serves":{"postgres-database":{"port":5432}},` + + `"grants":{"postgres-database":"/var/lib/mesh-host/grants"},` + + `"receives":{"postgres-database":"/var/lib/mesh-host/grants/mesh.json"},"resources":[]}' > /tmp/pg.json`); + // The consumer also writes a configuration file with a hole in it, which is how nearly every + // real program takes a credential: a sealed file is a password alone, and almost nothing reads + // one. The mesh cannot compose the document — it discarded the value — so the module supplies it + // with `${secret:...}` in it and the host, the only thing that sees both halves, fills it in. + await must("anchor", `printf %s '{"module":"meshboard","version":"1",` + + `"requires":["postgres-database"],"contributes":{"postgres-database":{"name":"meshboard"}},` + + `"binds":{"postgres-database":"/etc/meshboard/database.json"},` + + `"secrets":{"postgres-database":"/etc/meshboard/database.password"},` + + `"resources":[{"id":"env","type":"file","path":"/etc/meshboard/database.env","mode":"0600",` + + `"content":"PGHOST=$\{bound:postgres-database:at\}\\nPGPORT=$\{bound:postgres-database:port\}\\n` + + `PGUSER=$\{bound:postgres-database:as\}\\nPGPASSWORD=$\{secret:postgres-database\}\\n"}]}' ` + + `> /tmp/app.json`); + await must("anchor", `docker cp /tmp/pg.json mesh-control:/pg.json`); + await must("anchor", `docker cp /tmp/app.json mesh-control:/app.json`); + await mesh("module add /pg.json"); + await mesh("module add /app.json"); + + await mesh("overlay place anchor --hub --endpoint 192.0.2.10:51820 --site lab"); + await mesh("overlay place laptop --site lab"); + for (const node of ["anchor", "laptop"]) await mesh(`assign ${node} networking`); + await mesh("assign anchor postgres"); + await mesh("assign laptop meshboard"); + + for (const machine of ["anchor", "laptop"]) { + await must(machine, `nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); + } + await mesh("push"); + await new Promise((r) => setTimeout(r, 8000)); + + const onConsumer = (await must("laptop", `cat /etc/meshboard/database.password`)).trim(); + // Named after the machine *and* the module, because a consumer is both (novox/hq + // 04-ISSUES/022) — a node routinely runs several modules wanting one database. + const onProvider = (await must("anchor", + `cat /var/lib/mesh-host/grants/laptop.meshboard.secret`)).trim(); + assert.ok(onConsumer.length >= 40, `the consumer's credential is ${onConsumer.length} characters`); + assert.equal(onConsumer, onProvider, + "the two ends hold different passwords, so nothing could ever authenticate"); + + // Only the machine it is for may read it. + assert.match(await must("laptop", `stat -c %a /etc/meshboard/database.password`), /^600/); + + // And the configuration with holes in it arrived filled. **This is the only place the two + // substitutions are proven against a real host**: the mesh fills what it knows in the clear + // before sending, the host opens the sealed value and fills the rest on the machine, and the + // two expressions that find the holes live in different repositories. + const filled = await must("laptop", `cat /etc/meshboard/database.env`); + assert.match(filled, /^PGPASSWORD=.+$/m, `the password was never put in:\n${filled}`); + assert.ok(filled.includes(`PGPASSWORD=${onConsumer}`), + `the file holds a different password from the credential file:\n${filled}`); + assert.match(filled, /^PGUSER=mesh_laptop_meshboard$/m, + `the consumer was not told what name to present:\n${filled}`); + assert.match(filled, /^PGPORT=5432$/m, `the port did not arrive as a port:\n${filled}`); + assert.doesNotMatch(filled, /\$\{/, + `a placeholder survived to the machine and would be read as a value:\n${filled}`); + assert.match(await must("laptop", `stat -c %a /etc/meshboard/database.env`), /^600/); + + // The password is in that file and nowhere the mesh could read it — which is the whole point of + // filling the hole on the machine rather than composing the document in the control plane. + + // And it is nowhere it could have been read on the way. The declaration crossed the broker; the + // database is the control plane's; the state is what the node reported back. + for (const [machine, where] of [ + ["laptop", "/var/lib/mesh-host/declared.json"], + ["laptop", "/var/lib/mesh-host/state.json"], + ["anchor", "/var/lib/mesh-host/declared.json"], + ] as const) { + // **`--` first, or the password is read as options.** A generated credential is random, so + // one of them eventually begins with a dash — this one started `-S` and grep refused the + // whole invocation. The test then compared an error message against "0" and reported the + // password as leaked, which is the worst way for a search to fail: it says it found + // something. + const { out } = await on(machine, `grep -c -e ${quote(onConsumer)} -- ${where}`); + assert.equal(out.trim(), "0", `the password is in ${where} on ${machine}`); + } + const inTheMesh = await must("anchor", + `docker exec mesh-store psql -U postgres -d inventory -qAt ` + + `-c "select count(*) from secret where for_consumer like '%${onConsumer}%' ` + + `or for_provider like '%${onConsumer}%'"`); + assert.equal(inTheMesh.trim(), "0", "the control plane's database holds the password in the clear"); +}); + +test("the consumer is also told where its database is", { skip, timeout: 300_000 }, async () => { + // A password with no address is not a connection. This is the readable half, which stays + // readable on purpose — it is the secret half that could not be composed, not this one. + const told = JSON.parse(await must("laptop", `cat /etc/meshboard/database.json`)); + assert.equal(told.from, "anchor"); + assert.equal(told.at, "anchor.internal"); + assert.equal(told.serves.port, 5432); + // And the name it resolves to was written by the mesh as well, on this machine. + assert.match(await must("laptop", `grep anchor.internal /etc/hosts`), /10\.42\.0\.\d+/); +}); + +test("when a machine cannot do what it was told, the mesh says which and why", { skip, timeout: 900_000 }, async () => { + // Demonstrated with inserted rows first, which proves the query and not the path. This sends a + // real machine something it will genuinely fail at, and asks the mesh afterwards. + // + // A package that does not exist, because that is a failure of the ordinary kind: the host tries, + // the package manager says no, and some of the declaration is applied and some is not — which + // is the situation `status` exists to distinguish from a machine that refused everything. + await must("anchor", `printf %s '{"module":"impossible","version":"1","resources":[` + + `{"id":"nothing","type":"package","package":"a-package-that-does-not-exist"}]}' > /tmp/imp.json`); + await must("anchor", `docker cp /tmp/imp.json mesh-control:/imp.json`); + await mesh("module add /imp.json"); + await mesh("assign laptop impossible"); + await mesh("push laptop"); + await new Promise((r) => setTimeout(r, 10_000)); + + const said = await mesh("status"); + assert.match(said, /not doing what they were told/, said); + assert.match(said, /laptop/, said); + // The machine's own words about the resource that failed, not a summary written at this end. + assert.match(said, /impossible\.nothing/, `the failing resource is not named:\n${said}`); + + // And the distinction survives: this machine FAILED, it did not refuse. Refused means it is + // exactly as it was; failed means it is in a state nobody declared, and they are fixed in + // different places. + assert.match(said, /laptop\s+failed/, said); + + // The other machine is not implicated. + assert.doesNotMatch(said.split("not heard from")[0] ?? said, /anchor\s+(failed|refused)/, + "a machine that did as it was told is listed as wrong"); +}); + +test("a declaration waits for a machine that is switched off", { skip, timeout: 900_000 }, async () => { + // A machine is disconnected as an ordinary situation, not an exception (novox/hq ADR 0004), so + // a push to one that is not listening must wait rather than vanish. The queue is durable and the + // message persistent, which ought to be enough — but a lost declaration is silent, and "ought to + // be" is not a property. + // + // The machine is not merely idle here: its host is stopped, so nothing is consuming its queue. + + // Nothing this test asserts should depend on what another left behind. The machine still has a + // deliberately-impossible module from the test above, and while that is assigned the mesh never + // updates its account of what the machine holds — a partial report is not an account, on + // purpose (novox/hq 04-ISSUES/010). + await mesh("unassign laptop impossible"); + + // Stop listening, and prove it stopped — a test that pushed to a machine that was still running + // would pass having checked nothing. + // + // By process name, never by matching the command line: `pkill -f` matches the shell running it + // too, which kills the connection carrying the command and hangs the caller waiting for a reply + // that will never come. Cost an hour once, in this file. + await must("laptop", `pkill -x mesh-host || true; sleep 1`); + const listening = await on("laptop", `pgrep -x mesh-host`); + assert.equal(listening.ok, false, "the host is still running, so this proves nothing"); + + await must("anchor", `printf %s '{"module":"while-away","version":"1","resources":[` + + `{"id":"note","type":"file","path":"/etc/mesh-while-away","content":"waited"}]}' > /tmp/away.json`); + await must("anchor", `docker cp /tmp/away.json mesh-control:/away.json`); + await mesh("module add /away.json"); + await mesh("assign laptop while-away"); + await mesh("push laptop"); + + // Nothing has happened on the machine, because nothing is there to do it. + const before = await on("laptop", `test -f /etc/mesh-while-away`); + assert.equal(before.ok, false, "a machine with no host applied a declaration"); + + // And now it listens again. No second push, and nobody says anything. + await must("laptop", `nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 8`); + let arrived = false; + for (let i = 0; i < 20 && !arrived; i++) { + arrived = (await on("laptop", `test -f /etc/mesh-while-away`)).ok; + if (!arrived) await new Promise((r) => setTimeout(r, 2000)); + } + if (!arrived) { + // Everything needed to tell "the message was never queued" from "the host never read it". + const log = await on("laptop", `tail -20 /var/log/mesh-host.log`); + const queues = await on("anchor", + `docker exec mesh-broker lavinmqctl list_queues name messages 2>&1 | head -10`); + const owned = await on("anchor", + `docker exec mesh-store psql -U postgres -d inventory -qAt -c "select name, outcome from node_report r join node n on n.id=r.node"`); + assert.fail(`a declaration sent to a switched-off machine was lost\n` + + `--- the host's log ---\n${log.out}\n--- the broker's queues ---\n${queues.out}\n` + + `--- what each machine last did ---\n${owned.out}`); + } + assert.equal((await must("laptop", `cat /etc/mesh-while-away`)).trim(), "waited"); + + // And the mesh's account of what that machine holds catches up too, or a later declaration + // would tell it to remove what it has just been given. + await new Promise((r) => setTimeout(r, 4000)); + const owned = await must("anchor", + `docker exec mesh-store psql -U postgres -d inventory -qAt ` + + `-c "select owned from node where name = 'laptop'"`); + assert.match(owned, /while-away\.note/, `the mesh does not know the machine holds it: ${owned}`); +}); + +test("unassigning takes away exactly what it should", { skip, timeout: 900_000 }, async () => { + // Removal is the half nobody tests. The mesh takes away what IT declared and no longer declares, + // and never what the machine raised for itself from its bundle — which is the fault that + // destroyed a substrate once (novox/hq 04-ISSUES/010). + // + // Two modules, so the test can tell "removed the right one" from "removed everything". + for (const [name, path] of [["kept", "/etc/mesh-kept"], ["going", "/etc/mesh-going"]] as const) { + await must("anchor", `printf %s '{"module":"${name}","version":"1","resources":[` + + `{"id":"note","type":"file","path":"${path}","content":"${name}"}]}' > /tmp/${name}.json`); + await must("anchor", `docker cp /tmp/${name}.json mesh-control:/${name}.json`); + await mesh(`module add /${name}.json`); + await mesh(`assign anchor ${name}`); + } + await mesh("push anchor"); + await new Promise((r) => setTimeout(r, 6000)); + assert.ok((await on("anchor", `test -f /etc/mesh-kept`)).ok, "the first module did not arrive"); + assert.ok((await on("anchor", `test -f /etc/mesh-going`)).ok, "the second module did not arrive"); + + await mesh("unassign anchor going"); + await mesh("push anchor"); + await new Promise((r) => setTimeout(r, 6000)); + + assert.equal((await on("anchor", `test -f /etc/mesh-going`)).ok, false, + "an unassigned module's file is still there"); + assert.ok((await on("anchor", `test -f /etc/mesh-kept`)).ok, + "unassigning one module took another one's file with it"); + + // And the substrate this machine raised from its own bundle is untouched. It was not declared by + // the mesh, so the mesh must never remove it — the machine would take its own control plane + // away, which is exactly what happened before origins existed. + const running = await must("anchor", `docker ps --format '{{.Names}}'`); + for (const container of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(running, new RegExp(container), + `${container} was removed by a declaration that never declared it`); + } +}); + +test("a machine keeps what it was given when the mesh says nothing about it", { skip, timeout: 600_000 }, async () => { + // The other direction of the same rule. A node that is sent a declaration mentioning none of its + // private network must not lose it: the network came from a module that is still assigned, and + // "not in this message" is not "no longer wanted". + assert.ok((await on("laptop", `test -f /etc/wireguard/mesh0.conf`)).ok, + "the private network's configuration is gone"); + assert.ok((await on("laptop", `grep -q anchor.internal /etc/hosts`)).ok, + "the mesh's names are gone"); +}); + +test("a machine that fell behind catches up without being named", { skip, timeout: 900_000 }, async () => { + // `status` says which machines are not doing what they were told; something has to act on it. + // `push --behind` is that something, and it is a command rather than a timer to begin with — + // a scheduler is then a scheduler over this, rather than a second path to the same act. + + // Break one machine, in a way that is fixable: a package that does not exist yet. + await must("anchor", `printf %s '{"module":"fixable","version":"1","resources":[` + + `{"id":"pkg","type":"package","package":"a-package-that-does-not-exist-yet"},` + + `{"id":"note","type":"file","path":"/etc/mesh-fixable","content":"here"}]}' > /tmp/fix.json`); + await must("anchor", `docker cp /tmp/fix.json mesh-control:/fix.json`); + await mesh("module add /fix.json"); + await mesh("assign laptop fixable"); + await mesh("push laptop"); + await new Promise((r) => setTimeout(r, 8000)); + + assert.match(await mesh("status"), /laptop\s+failed/, "the machine did not report a failure"); + // And the resources that COULD be applied were — one broken thing no longer blocks the rest + // (novox/hq 04-ISSUES/011). + assert.ok((await on("laptop", `test -f /etc/mesh-fixable`)).ok, + "a resource after the failing one was never attempted"); + + // Nothing is behind on the other machine, so nothing is pushed to it. + const named = await mesh("push --behind"); + assert.match(named, /laptop/, named); + assert.doesNotMatch(named, /sent anchor/, `a machine that was fine was pushed to:\n${named}`); + + // Fix the cause, the way somebody would: the module stops asking for the impossible thing. + await must("anchor", `printf %s '{"module":"fixable","version":"1","resources":[` + + `{"id":"note","type":"file","path":"/etc/mesh-fixable","content":"here"}]}' > /tmp/fix.json`); + await must("anchor", `docker cp /tmp/fix.json mesh-control:/fix.json`); + await mesh("module add /fix.json"); + + // And nobody names the machine. + await mesh("push --behind"); + await new Promise((r) => setTimeout(r, 8000)); + + const after = await mesh("status"); + assert.doesNotMatch(after, /laptop\s+(failed|refused)/, + `the machine did not recover:\n${after}`); + + // With nothing behind, it says so rather than doing nothing quietly. + assert.match(await mesh("push --behind"), /every machine is doing what it was told/); +}); + +test("the mesh runs its own artifact store", { skip, timeout: 900_000 }, async () => { + // Artifacts go to a registry, and the only registries that existed were raised by the lab or by + // the bootstrap bundle. A mesh had no way to run its own. + // + // **Named, not mirrored** (novox/hq 04-ISSUES/029). Mirroring publishes to the artifact store, + // and the builder will not start without one — so a module that provides the store and builds + // its own image asks the mesh to put an artifact into the thing that artifact is needed to + // create. It worked here only because the scenario's registry was already standing to receive + // the push, which is exactly why a real first mesh would have found this and the lab did not. + // + // So the image is named by digest, the way the bundle names the three a first node starts from. + // A registry is the one module that cannot be delivered by the mesh's own delivery. + await must("anchor", `mkdir -p /root/registry && printf %s '{"module":"registry","version":"1",` + + `"provides":[{"name":"artifact-store","scope":"mesh"}],` + + `"capabilities":["container-runtime"],` + + `"claims":[{"name":"the-artifact-store","scope":"node"}],` + + `"serves":{"artifact-store":{"port":5000}},` + + `"resources":[` + + `{"id":"state","type":"directory","path":"/var/lib/mesh/registry","mode":"0700"},` + + `{"id":"store","type":"container","name":"mesh-registry","image":"${pinned("registry")}",` + + `"ports":["5000:5000"],"volumes":["mesh-registry-data:/var/lib/registry"]}]}' ` + + `> /root/registry/module.json`); + // **Added, not built** — and this is the half that proves the fix. Building needs a builder, + // and a builder will not start without an artifact store to publish to, so a mesh that has just + // bootstrapped cannot build the module that gives it one. Adding the manifest directly is the + // path a real first mesh has to take, so it is the path this walks. + await must("anchor", `docker cp /root/registry/module.json mesh-control:/registry.json`); + await mesh("module add /registry.json"); + await mesh("assign anchor registry"); + await mesh("push anchor"); + await new Promise((r) => setTimeout(r, 12_000)); + + // Running, and answering — a container that is up is not a registry that replies. + assert.match(await must("anchor", `docker ps --format '{{.Names}}'`), /mesh-registry/); + let answers = false; + for (let i = 0; i < 20 && !answers; i++) { + answers = (await on("anchor", `curl -sf http://127.0.0.1:5000/v2/ -o /dev/null`)).ok; + if (!answers) await new Promise((r) => setTimeout(r, 2000)); + } + assert.ok(answers, "the mesh's own registry is running and does not answer"); + + // And reachable from another machine over the private network, which is the whole point of an + // artifact store being a mesh-scoped provision. + assert.ok((await on("laptop", `curl -sf http://anchor.internal:5000/v2/ -o /dev/null`)).ok, + "the artifact store is not reachable from another machine, so nothing else can use it"); +}); + +// Defends novox/hq ADR 0007: the mesh is its own certificate authority for internal names. +test("a machine serves its internal name with a certificate the mesh issued", { + skip, timeout: 900_000, +}, async () => { + // The mesh's own authority certifies names only the mesh knows (novox/hq 08-connectivity). + // Asserted with a real handshake: a certificate that parses and does not chain fails at the + // moment something connects, which is the worst place to find out. + await must("anchor", `printf %s '{"module":"served","version":"1",` + + `"certificate":{"into":"/etc/mesh/serving.crt","authority":"/etc/mesh/authority.crt"},` + + `"resources":[{"id":"dir","type":"directory","path":"/etc/mesh","mode":"0755"}]}' ` + + `> /tmp/served.json`); + await must("anchor", `docker cp /tmp/served.json mesh-control:/served.json`); + await mesh("module add /served.json"); + await mesh("assign anchor served"); + await mesh("push anchor"); + await new Promise((r) => setTimeout(r, 8000)); + + assert.ok((await on("anchor", `test -s /etc/mesh/serving.crt`)).ok, "no certificate arrived"); + assert.ok((await on("anchor", `test -s /etc/mesh/authority.crt`)).ok, "no authority arrived"); + + // The name it was issued for is the one the mesh gave this machine. + const named = await must("anchor", + `openssl x509 -in /etc/mesh/serving.crt -noout -ext subjectAltName 2>/dev/null || ` + + `docker run --rm -v /etc/mesh:/m ${pinned("registry")} sh -c ` + + `"apk add --no-cache openssl >/dev/null 2>&1; openssl x509 -in /m/serving.crt -noout -text" | grep -A1 'Alternative'`); + assert.match(named, /anchor\.internal/, `the certificate is not for this machine's name:\n${named}`); + + // And a real handshake: the machine serves TLS with the key it generated, and another machine + // verifies it against the mesh's authority and nothing else. + await must("anchor", `openssl s_server -cert /etc/mesh/serving.crt ` + + `-key /var/lib/mesh-host/serving.key -accept 8443 -naccept 1 -quiet ` + + `> /var/log/tls.log 2>&1 & sleep 2`); + await must("laptop", `mkdir -p /etc/mesh`); + const authority = await must("anchor", `cat /etc/mesh/authority.crt`); + await must("laptop", `cat > /etc/mesh/authority.crt <<'MESHCA'\n${authority}\nMESHCA`); + + const shook = await on("laptop", + `echo | openssl s_client -connect anchor.internal:8443 ` + + `-CAfile /etc/mesh/authority.crt -verify_return_error -brief 2>&1`); + assert.ok(shook.ok, `the handshake failed:\n${shook.out}\n` + + `what the server said:\n${(await on("anchor", `cat /var/log/tls.log`)).out}`); + assert.match(shook.out, /Verification: OK/, shook.out); +}); + +// Defends novox/hq ADR 0007: what a machine exposes is what its modules declared, and nothing +// arrives at a port nobody asked for. +test("a machine filters exactly what its modules declared, and nothing else", { + skip, timeout: 900_000, +}, async () => { + // The rule set is derived from what is assigned, not kept in step by hand — and the proof that + // matters is not that a file arrived but that packets are treated differently because of it. + // A rule nothing enforces is the fault this mechanism exists to remove (novox/hq 04-ISSUES/003). + // + // Note what the module cannot contain: an action. The link may not carry one (novox/hq ADR 0005), + // so the mesh writes the rule set and declares that a service must reflect it. `restart-on` is + // the shape that rule leaves, and this is the first thing to use it for its real purpose. + // A listener is written to a file rather than squeezed through three levels of shell quoting. + // The first attempt did the latter, never started, and the test failed on its own setup — + // which reads exactly like the firewall working. + await must("laptop", `cat > /root/listen.py <<'LISTENER'\n` + + `import socket, sys, threading\n` + + `def serve(port):\n` + + ` s = socket.socket()\n` + + ` s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n` + + ` s.bind(("0.0.0.0", port))\n` + + ` s.listen(8)\n` + + ` while True:\n` + + ` c, _ = s.accept()\n` + + ` c.send(str(port).encode())\n` + + ` c.close()\n` + + `for p in (9101, 9102):\n` + + ` threading.Thread(target=serve, args=(p,), daemon=True).start()\n` + + `threading.Event().wait()\n` + + `LISTENER`); + await must("laptop", `nohup python3 /root/listen.py > /var/log/listen.log 2>&1 & sleep 2`); + + // Two paths to the same machine, which is what makes "from the mesh" testable at all: over the + // private network, and over the segment both machines happen to share. A rule that opens a port + // to the mesh must accept the first and refuse the second — and a test that only ever used one + // path could not tell "open to the mesh" from "open". + const reach = async (where: string, port: number) => { + const said = await on("anchor", + `timeout 5 python3 -c "import socket;s=socket.create_connection(('${where}',${port}),4);` + + `print(s.recv(32).decode());s.close()"`); + return said.ok; + }; + const overlay = (port: number) => reach("laptop.internal", port); + const segment = (port: number) => reach("192.0.2.20", port); + + // Reachable both ways before any rule set exists, so what changes afterwards is the rule set and + // not the listener. Without this the test would pass against a service that never started. + assert.ok(await overlay(9101), "the declared port never opened, so nothing below tests anything"); + assert.ok(await overlay(9102), "the undeclared port never opened"); + assert.ok(await segment(9101), "the declared port is not reachable off the private network yet, " + + "so closing it later would prove nothing"); + + await must("anchor", `printf %s '{"module":"talker","version":"1",` + + `"listens":[{"port":9101,"from":"mesh","why":"the thing this test is about"}],` + + `"resources":[]}' > /tmp/talker.json`); + // The rule set goes where this machine's nftables unit reads from, and the unit is declared to + // reflect it. No command anywhere. + // The module ships the unit that loads its rules, rather than using the one the distribution's + // nftables package provides. That unit is `Type=oneshot` with no `RemainAfterExit`, so it does + // its work and goes inactive — and a host asked for a service that is "running" reports, quite + // correctly, that it is stopped. There is no state in the vocabulary for "ran and exited having + // done its job", so a module that wants one brings a unit that stays. + // + // Which is also the right shape: how a machine enforces rules is a fact about the machine, and + // the mesh has no business depending on what a distribution happens to package. + await must("anchor", `printf %s '{"module":"firewall","version":"1",` + + `"capabilities":["firewall"],` + + `"filtering":{"into":"/etc/mesh/filter.nft"},` + + `"resources":[{"id":"nftables","type":"package","package":"nftables"},` + + `{"id":"dir","type":"directory","path":"/etc/mesh","mode":"0755"},` + + `{"id":"unit","type":"file","path":"/etc/systemd/system/mesh-filter.service",` + + `"mode":"0644","content":"[Unit]\\nDescription=What the mesh computed for this machine\\n` + + `[Service]\\nType=oneshot\\nRemainAfterExit=yes\\n` + + `ExecStart=/usr/bin/nft -f /etc/mesh/filter.nft\\n[Install]\\nWantedBy=multi-user.target\\n"},` + + `{"id":"filter","type":"service","unit":"mesh-filter.service","state":"running",` + + `"boot":"enabled","restart-on":["filtering"]}]}' > /tmp/firewall.json`); + for (const f of ["talker", "firewall"]) { + await must("anchor", `docker cp /tmp/${f}.json mesh-control:/${f}.json`); + await mesh(`module add /${f}.json`); + } + await mesh("assign laptop talker"); + await mesh("assign laptop firewall"); + await mesh("push laptop"); + await new Promise((r) => setTimeout(r, 20_000)); + + const written = await must("laptop", `cat /etc/mesh/filter.nft`); + // A rule names its source. Not decoration: it is the only thing that answers "why is this open". + assert.match(written, /# talker . the thing this test is about/, + `the rule does not name what caused it:\n${written}`); + // The mesh's addresses are the ones on the private network, which is what "from the mesh" + // means — not the segment the machines happen to share. + assert.match(written, /ip saddr \{ [0-9., ]+ \} tcp dport 9101 accept/, + `"from the mesh" resolved to nothing:\n${written}`); + assert.doesNotMatch(written, /dport 9102/, `a port no module declared was opened:\n${written}`); + + // Loaded, not merely written. The service was restarted because a file it reflects changed. + const table = await must("laptop", `nft list table inet mesh`); + assert.match(table, /dport 9101 accept/, `the rule set was never loaded:\n${table}`); + + // And it filters. Three assertions, and the third is the one that makes "from the mesh" mean + // something rather than being a synonym for "open". + assert.ok(await overlay(9101), + "the declared port is closed on the private network, so the machine is filtering more than " + + "it was told to"); + assert.ok(!(await overlay(9102)), + "a port no module declared is still reachable, so the rule set restricts nothing"); + assert.ok(!(await segment(9101)), + "the declared port answers off the private network, so `from: mesh` restricted nothing"); + + // The machine did not lock itself out of the mesh: it is still taking declarations. + assert.doesNotMatch(await mesh("status"), /laptop\s+(failed|refused)/, + "the machine stopped doing what it was told after applying its own rule set"); + + // Removing the module that wanted the port closes it, with nobody editing a rule. This is the + // whole claim of a derived firewall, and it is also the second load — which must replace the + // table rather than add to it. + await mesh("unassign laptop talker"); + await mesh("push laptop"); + await new Promise((r) => setTimeout(r, 20_000)); + assert.ok(!(await overlay(9101)), + "the port stayed open after the module that wanted it was removed"); +}); + +test("the builder is a module the mesh assigns, with a credential the mesh delivered", { + skip: skip || (!builder ? "set MESH_LAB_BUILDER to a built mesh-builder" : false), + timeout: 900_000, +}, async () => { + // Until this, the builder was a program somebody started on a machine with whatever credential + // they had to hand — in practice the broker's administrative one. A program documented as + // holding its own credential and given somebody else's is worse than one with no story at all. + // + // So: the mesh issues a scoped account, seals it to the machine, and delivers it with the + // declaration. Nobody types it and the mesh cannot read it back. + await must("anchor", `mkdir -p /root/builder && printf %s '{"module":"builder","version":"1",` + + `"requires":["artifact-store"],"capabilities":["container-runtime"],` + + `"claims":[{"name":"the-build-machine","scope":"node"}],` + + `"binds":{"artifact-store":"/var/lib/mesh/builder/artifact-store.json"},` + + `"own-secrets":{"broker":"/var/lib/mesh/builder/broker"},` + + `"build":{"artifacts":[{"name":"builder","kind":"upstream",` + + `"from":"${pinned("mesh-builder")}"}]},` + + `"resources":[` + + `{"id":"state","type":"directory","path":"/var/lib/mesh/builder","mode":"0700"},` + + `{"id":"workspace","type":"directory","path":"/var/lib/mesh/builder/workspace","mode":"0700"},` + + `{"id":"run","type":"container","name":"mesh-builder","artifact":"builder",` + + `"network":"host",` + + `"volumes":["/var/lib/mesh/builder:/var/lib/mesh/builder",` + + `"/var/run/docker.sock:/var/run/docker.sock"],` + + `"env":{"MESH_BROKER_FILE":"/var/lib/mesh/builder/broker",` + + `"MESH_BINDING":"/var/lib/mesh/builder/artifact-store.json",` + + `"MESH_WORKSPACE":"/var/lib/mesh/builder/workspace"}}]}' > /root/builder/module.json`); + await must("anchor", `cd /root/builder && git init -q . && git add -A && ` + + `git -c user.email=lab -c user.name=lab commit -qm builder`); + + // The builder's own image is built by the builder that is already running — the same + // chicken-and-egg as the registry, resolved the same way. The one started by hand does this + // last piece of work and is then replaced by the module it just built. + await mesh("build /root/builder --wait 300s", 420_000); + + // The mesh makes the account and seals the URL to this machine. Nothing is printed that would + // work if it were pasted somewhere else. + const issued = await mesh("builder issue lab-builder --node anchor"); + assert.match(issued, /sealed to anchor/, issued); + assert.doesNotMatch(issued, /amqps:\/\/lab-builder:/, + "the credential was printed, so the one copy that matters is on a terminal"); + + // Now the hand-started one goes, or two builders race for the same queue and whichever answers + // proves nothing. By process name: `pkill -f` matches the shell running it too, which kills the + // connection carrying the command and hangs the caller waiting for a reply that will never + // come. Cost an hour once, in this file. + await on("anchor", `pkill -x mesh-builder`); + await new Promise((r) => setTimeout(r, 2000)); + assert.ok(!(await on("anchor", `pgrep -x mesh-builder`)).ok, + "the hand-started builder is still running, so this would test that one"); + + await mesh("assign anchor builder"); + await mesh("push anchor"); + await new Promise((r) => setTimeout(r, 20_000)); + + const running = await must("anchor", `docker ps --format '{{.Names}}'`); + assert.match(running, /mesh-builder/, + `the builder was assigned and is not running:\n${running}\n` + + `${(await on("anchor", `tail -30 /var/log/mesh-host.log`)).out}`); + + // Running is not connected. A builder that cannot reach the broker sits there, and every + // outward sign — the container is up, the credential is on disk — says it is working. + await new Promise((r) => setTimeout(r, 5000)); + const said = await on("anchor", `docker logs mesh-builder 2>&1 | tail -20`); + assert.doesNotMatch(said.out, /cannot reach the broker/, + `the builder is running and cannot reach the broker:\n${said.out}`); + + // The credential arrived, is readable only by the machine, and is the scoped account rather + // than the broker's own. + assert.match(await must("anchor", `stat -c %a /var/lib/mesh/builder/broker`), /^600/); + const credential = await must("anchor", `cat /var/lib/mesh/builder/broker`); + assert.match(credential, /"url":"amqps:\/\/lab-builder:/, + "the builder is using an account that is not its own"); + assert.doesNotMatch(credential, /guest:guest/, "the builder holds the broker's own account"); + // And what to check the broker against. A mesh's broker presents a certificate of the mesh's + // own, so a URL alone reaches only a broker some public authority vouches for — which is no + // mesh broker at all, and fails at TLS with an error about an unknown authority. + assert.match(credential, /"fingerprint":"(sha256:)?[0-9a-f]{64}"/, + `the builder was given nothing to verify the broker with:\n${credential}`); + + // And it works: the mesh asks this builder to build something, and it does. Answering is the + // only proof that the delivered credential authenticates — a container that is up with a + // credential it cannot use looks identical from outside. + // Somewhere the builder can actually see. A builder that is a module runs in a container, so + // the machine's filesystem is not its own — a path like /root only works for a builder somebody + // started on the host, which is what the first build above used. In a real mesh a module is + // cloned from the forge over a URL; here it goes in the directory the module already mounts, + // which is the same fact wearing different clothes. + const repo = "/var/lib/mesh/builder/repositories/built"; + await must("anchor", `mkdir -p ${repo} && printf %s '{"module":"built","version":"1",` + + `"resources":[{"id":"marker","type":"file","path":"/etc/built","content":"yes","mode":"0644"}]}' ` + + `> ${repo}/module.json`); + await must("anchor", `cd ${repo} && git init -q . && git add -A && ` + + `git -c user.email=lab -c user.name=lab commit -qm built`); + try { + await mesh(`build ${repo} --wait 300s`, 420_000); + } catch (why) { + // The builder's own account of itself. Without it the failure is "nothing consumed the + // queue", which names no cause and is the same sentence whether the credential was refused, + // the queue was never declared, or the process died three seconds in. + const said = (await on("anchor", `docker logs mesh-builder 2>&1 | tail -40`)).out; + throw new Error(`${(why as Error).message}\n\nwhat the builder said:\n${said}`); + } + + // Naming the module, and not merely containing its name: `builds` says "nothing has been built + // yet" when there is nothing, and that sentence contains the word this was matching on. + const recorded = await mesh("builds built"); + assert.doesNotMatch(recorded, /nothing has been built/, + `the build was accepted and no build was recorded against the module:\n${recorded}`); + assert.match(recorded, /built/, recorded); +}); + +test("rotating a credential moves both ends, and the old one stops working", { + skip, timeout: 900_000, +}, async () => { + // The invariant novox/hq ADR 0001 records as unowned, and it was measurably false in HAL: on + // 2026-08-22 a provision documented as never rotating minted a new password on every adoption + // and updated only the provider's row. Consumers on three nodes held dead credentials for two + // days while the mesh reported success. + // + // So this is checked against a real database with a real login, three times: the delivered + // credential works, the rotated one works, and the one that was rotated away does not. Two ends + // holding a matching string proves they agree; only an authentication proves they are right. + const store = "/var/lib/mesh/postgres"; + await must("anchor", `printf %s '{"module":"realstore","version":"1",` + + `"provides":[{"name":"real-postgres-database","scope":"mesh"}],` + + `"capabilities":["container-runtime"],` + + `"serves":{"real-postgres-database":{"port":5433}},` + + `"own-secrets":{"superuser":"${store}/superuser"},` + + `"grants":{"real-postgres-database":"${store}/grants"},` + + // Both halves. `grants` is where each consumer's sealed password lands; `receives` is the + // manifest saying who asked and for what. Without the second the provisioner finds a + // directory of unexplained secrets and says nothing has been granted — which is true, and + // reads exactly like a credential that was never delivered. + `"receives":{"real-postgres-database":"${store}/grants/mesh.json"},` + + `"listens":[{"port":5433,"from":"mesh","why":"a database the mesh provisions"}],` + + `"resources":[` + + `{"id":"state","type":"directory","path":"${store}","mode":"0755"},` + + `{"id":"grants","type":"directory","path":"${store}/grants","mode":"0755"},` + + `{"id":"postgres-database","type":"container","name":"real-store",` + + `"image":"${pinned("postgres")}",` + + `"ports":["5433:5432"],` + + `"volumes":["${store}/superuser:/run/superuser:ro"],` + + `"env":{"POSTGRES_PASSWORD_FILE":"/run/superuser"}},` + + `{"id":"provisioner","type":"container","name":"real-provisioner",` + + `"image":"${pinned("mesh-provision-postgres")}","network":"host",` + + `"volumes":["${store}:${store}:ro"],` + + `"env":{"GRANTS":"${store}/grants",` + + `"MESH_PROVISION_PASSWORD_FILE":"${store}/superuser",` + + `"MESH_PROVISION_POSTGRES":"postgres://postgres@127.0.0.1:5433/postgres?sslmode=disable"}}]}' ` + + `> /tmp/realstore.json`); + await must("anchor", `printf %s '{"module":"realapp","version":"1",` + + `"requires":["real-postgres-database"],"contributes":{"real-postgres-database":{"name":"realapp"}},` + + `"binds":{"real-postgres-database":"/etc/realapp/where.json"},` + + `"secrets":{"real-postgres-database":"/etc/realapp/password"},` + + `"resources":[{"id":"dir","type":"directory","path":"/etc/realapp","mode":"0755"}]}' ` + + `> /tmp/realapp.json`); + for (const f of ["realstore", "realapp"]) { + await must("anchor", `docker cp /tmp/${f}.json mesh-control:/${f}.json`); + await mesh(`module add /${f}.json`); + } + await mesh("assign anchor realstore"); + await mesh("assign laptop realapp"); + await mesh("push"); + await new Promise((r) => setTimeout(r, 30_000)); + + // A real login from the consumer's machine, over the private network — not over loopback, where + // pg_hba trusts anything and every password looks correct. That was done here once and the test + // passed for an afternoon while verifying nothing: a deliberately wrong password returned a row. + // As the role the provisioner made, into the database it made. The provisioner names a role + // after the machine and a database after what the module asked for — which is the contract, and + // getting it wrong here made the test fail against a provisioner that had done its job. + // By address, resolved on the machine. + // + // **This container is not the mesh's.** The mesh gives its names to the containers it declares, + // and this one is started by the test with `docker run` — nothing declared it, so nothing + // configured it. That boundary is the right one: a container somebody runs by hand is not the + // mesh's to configure, and reaching into every container on a machine is what a resolver in + // resolv.conf would be for. + // + // So the workaround stays here, and the proof that names work inside containers is its own + // test, against a container the mesh declared. + const where = (await must("laptop", + `getent hosts anchor.internal | head -1 | cut -d' ' -f1`)).trim(); + assert.match(where, /^[0-9.]+$/, `the mesh's name for anchor does not resolve here: ${where}`); + + const login = async (password: string) => + await on("laptop", `docker run --rm -e PGPASSWORD=${quote(password)} ` + + // The role the provisioner made: mesh__, because a consumer is a module on + // a machine (novox/hq 04-ISSUES/022). + `${pinned("postgres")} psql -h ${where} -p 5433 -U mesh_laptop_realapp ` + + `-d realapp -qAt -c "select 1"`, 120_000); + + const diagnostics = async () => + `provisioner:\n${(await on("anchor", `docker logs real-provisioner 2>&1 | tail -20`)).out}\n` + + `grants:\n${(await on("anchor", `ls -l ${store}/grants`)).out}`; + + const first = (await must("laptop", `cat /etc/realapp/password`)).trim(); + assert.ok(first.length >= 40, `the consumer's credential is ${first.length} characters`); + let works = false; + for (let i = 0; i < 20 && !works; i++) { + works = (await login(first)).ok; + if (!works) await new Promise((r) => setTimeout(r, 5000)); + } + assert.ok(works, `the delivered credential does not authenticate:\n` + + `${(await login(first)).out}\n${await diagnostics()}`); + + // Now rotate. One command: the record changes AND both ends are sent, because leaving the + // sending to a later command is the fault above, exactly. + const said = await mesh("rotate real-postgres-database", 180_000); + assert.match(said, /anchor/, `rotation did not touch the provider:\n${said}`); + assert.match(said, /laptop/, `rotation did not touch the consumer:\n${said}`); + await new Promise((r) => setTimeout(r, 25_000)); + + const second = (await must("laptop", `cat /etc/realapp/password`)).trim(); + assert.notEqual(second, first, "the consumer was handed back the credential just rotated away"); + + // The new one authenticates — the only proof the provider was told the same thing the consumer + // was given. Two files agreeing proves they agree, not that either is right. + let now = false; + for (let i = 0; i < 20 && !now; i++) { + now = (await login(second)).ok; + if (!now) await new Promise((r) => setTimeout(r, 5000)); + } + assert.ok(now, `after rotation the new credential does not authenticate, so the two ends ` + + `disagree — which is the fault this exists to make impossible:\n${await diagnostics()}`); + + // And the old one does not. Without this the test passes against a provider that added a + // password without replacing one, which is a rotation that rotates nothing. + assert.ok(!(await login(first)).ok, + "the password that was rotated away still authenticates, so nothing was rotated"); +}); + +test("a route is a grant: a workload is reached by the name it asked for", { + skip, timeout: 900_000, +}, async () => { + // novox/hq 08-connectivity §3. The mirror of a database grant: there the consumer supplies a + // name and receives credentials; here it supplies a target and receives a name. Nothing new in + // the vocabulary — a route is a provision like any other. + // + // The workload is the registry image, because it is an HTTP server this scenario already has. + // What is being tested is the mesh's arrangement, not the workload. + await must("anchor", `printf %s '{"module":"frontdoor","version":"1",` + + `"provides":[{"name":"route","scope":"mesh"}],` + + `"capabilities":["container-runtime"],` + + `"receives":{"route":"/etc/frontdoor/routes.json"},` + + `"serves":{"route":{"domain":"mesh.test"}},` + + `"listens":[{"port":8081,"from":"mesh","why":"the front door"}],` + + `"resources":[{"id":"dir","type":"directory","path":"/etc/frontdoor","mode":"0755"},` + + `{"id":"proxy","type":"container","name":"front-door",` + + `"image":"${pinned("mesh-route-proxy")}","network":"host",` + + `"volumes":["/etc/frontdoor:/etc/frontdoor:ro"],` + + `"env":{"ROUTES":"/etc/frontdoor/routes.json","LISTEN":":8081"}}]}' ` + + `> /tmp/frontdoor.json`); + // The workload declares the port it listens on as well as the route it wants. Both, because + // they are different questions: one says who may reach it, the other says by what name — and + // the earlier test left this machine filtering, so a module that asked for a route and not for + // the port would be unreachable by the proxy it just asked for. + await must("anchor", `printf %s '{"module":"storefront","version":"1",` + + `"requires":["route"],"capabilities":["container-runtime"],` + + `"contributes":{"route":{"name":"shop.mesh.test","port":8088}},` + + `"binds":{"route":"/etc/storefront/route.json"},` + + `"listens":[{"port":8088,"from":"mesh","why":"the proxy reaches it here"}],` + + `"resources":[{"id":"dir","type":"directory","path":"/etc/storefront","mode":"0755"},` + + `{"id":"app","type":"container","name":"storefront",` + + `"image":"${pinned("registry")}","ports":["8088:5000"]}]}' > /tmp/storefront.json`); + for (const f of ["frontdoor", "storefront"]) { + await must("anchor", `docker cp /tmp/${f}.json mesh-control:/${f}.json`); + await mesh(`module add /${f}.json`); + } + await mesh("assign anchor frontdoor"); + await mesh("assign laptop storefront"); + await mesh("push"); + await new Promise((r) => setTimeout(r, 25_000)); + + // The provider was told who asked, and where that machine is — which it needs in order to + // reach back, and which it must not have to derive from a naming convention. + const routes = await must("anchor", `cat /etc/frontdoor/routes.json`); + assert.match(routes, /shop\.mesh\.test/, `the proxy was not told about the route:\n${routes}`); + assert.match(routes, /"at": *"laptop\.internal"/, + `the proxy was not told where the consumer is, so it cannot reach it:\n${routes}`); + + // And the consumer was told what the provider serves, which is how it knows its own name. + const bound = await must("laptop", `cat /etc/storefront/route.json`); + assert.match(bound, /mesh\.test/, `the consumer was not told the public name:\n${bound}`); + + // The whole point: a request for the name reaches the workload, across the private network. + let reached = false; + let said = ""; + for (let i = 0; i < 20 && !reached; i++) { + const answer = await on("anchor", + `curl -sf -H 'Host: shop.mesh.test' http://127.0.0.1:8081/v2/ -o /dev/null -w '%{http_code}'`); + said = answer.out; + reached = answer.ok && said.trim() === "200"; + if (!reached) await new Promise((r) => setTimeout(r, 4000)); + } + assert.ok(reached, `a request for the name did not reach the workload (${said}):\n` + + `${(await on("anchor", `docker logs front-door 2>&1 | tail -20`)).out}`); + + // Withdrawal, which 08-connectivity lists as open: a stale public name pointing at nothing + // fails more visibly than a stale grant, so it must not survive the module leaving. + await mesh("unassign laptop storefront"); + await mesh("push"); + + // **Waited for, not slept through.** The mesh withdrawing a route and the machine acting on it + // are different things, and a fixed sleep between them tests whichever the clock happened to + // land on — this assertion passed twice and failed once on nothing but timing. + // + // A machine that has not applied yet is also not a machine that failed, so `status` cannot + // stand in for this: "not yet" and "never" look identical there, and only one of them is worth + // failing over. + let after = ""; + let withdrawn = false; + for (let i = 0; i < 20 && !withdrawn; i++) { + after = await must("anchor", `cat /etc/frontdoor/routes.json`); + withdrawn = !after.includes("shop.mesh.test"); + if (!withdrawn) await new Promise((r) => setTimeout(r, 3000)); + } + assert.ok(withdrawn, + `the route outlived the module that asked for it:\n${after}\n\n` + + `the machine did apply — this is what the mesh would send now:\n` + + `${(await on("anchor", `docker exec mesh-control /mesh-control plan anchor --files`)).out}`); + + let gone = false; + for (let i = 0; i < 15 && !gone; i++) { + const answer = await on("anchor", + `curl -s -H 'Host: shop.mesh.test' http://127.0.0.1:8081/v2/ -o /dev/null -w '%{http_code}'`); + gone = answer.out.trim() === "404"; + if (!gone) await new Promise((r) => setTimeout(r, 3000)); + } + assert.ok(gone, "the proxy still serves a name whose module was unassigned"); +}); + +test("model access is answered by a record, and the key the mesh took is one it cannot read", { + skip, timeout: 900_000, +}, async () => { + // novox/hq ADR 0024. The first provision no machine answers: a hosted model is on nobody's + // node and is reached over the public internet, so the rule that refuses two ends sharing no + // private network must not apply to it. + await must("anchor", `printf %s '{"module":"assistant","version":"1",` + + `"requires":["model-access"],` + + `"binds":{"model-access":"/etc/assistant/model.json"},` + + `"secrets":{"model-access":"/etc/assistant/key"},` + + `"resources":[{"id":"dir","type":"directory","path":"/etc/assistant","mode":"0755"}]}' ` + + `> /tmp/assistant.json`); + await must("anchor", `docker cp /tmp/assistant.json mesh-control:/assistant.json`); + await mesh("module add /assistant.json"); + + // The licences first. With none recorded at all the honest answer is that nothing provides + // model-access — correct, and a different refusal from the one being tested. + await mesh(`licence add anthropic personal --serves '{"model":"a-model"}'`); + await mesh(`licence add anthropic the-organisation --serves '{"model":"a-model"}'`); + + // Refused at the earliest point somebody could meet it: assigning records the assignment and + // then says the machine's set cannot be applied. The refusal names both candidates and the + // command. ADR 0024 warns this will be felt — which is correct, and correct is not the same as + // usable. + const refused = await on("anchor", `docker exec mesh-control /mesh-control assign laptop assistant`); + assert.ok(!refused.ok, + `a consumer was given model access without anybody saying which:\n${refused.out}`); + for (const want of ["personal", "the-organisation", "licence use"]) { + assert.match(refused.out, new RegExp(want), + `the refusal does not name ${want}:\n${refused.out}`); + } + + await mesh("licence use personal laptop assistant"); + + // Chosen, and still no key: the mesh has one thing to deliver and has not been given it. + const noKey = await on("anchor", `docker exec mesh-control /mesh-control plan laptop`); + assert.ok(!noKey.ok, `a module was planned with a licence that has no key:\n${noKey.out}`); + assert.match(noKey.out, /licence key personal/, noKey.out); + + // The accept verb. Given on standard input rather than as an argument, because a key in a + // command line is a key in shell history and in every process listing taken while it ran. + const secret = "sk-test-" + "0123456789abcdef".repeat(2); + const accepted = await must("anchor", + `printf %s ${quote(secret)} | docker exec -i mesh-control /mesh-control licence key personal`); + assert.match(accepted, /sealed to 1 holder/, accepted); + assert.doesNotMatch(accepted, new RegExp(secret), + "the key was echoed back, so the one copy that matters is on a terminal"); + + await mesh("push laptop"); + await new Promise((r) => setTimeout(r, 15_000)); + + // What is public arrives, and says it is a record rather than leaving an empty address that a + // reader would take for something the mesh failed to fill in. + const bound = await must("laptop", `cat /etc/assistant/model.json`); + assert.match(bound, /personal/, `the consumer was not told which licence it is on:\n${bound}`); + assert.match(bound, /a-model/, `what the licence serves did not arrive:\n${bound}`); + assert.match(bound, /not a machine/, `the binding leaves an unexplained empty address:\n${bound}`); + + // And the key arrives, readable only by this machine. + assert.equal((await must("laptop", `cat /etc/assistant/key`)).trim(), secret, + "the key that arrived is not the key that was given"); + assert.match(await must("laptop", `stat -c %a /etc/assistant/key`), /^600/); + + // The mesh cannot read it back. This is the whole argument: what is stored is unusable by + // whoever holds it, the control plane included. + const stored = await must("anchor", + `docker exec mesh-store psql -U postgres -d licences -qAt ` + + `-c "select coalesce(sealed,'') from licence_holder"`); + assert.ok(stored.trim().length > 0, "nothing was stored, so nothing was sealed"); + assert.doesNotMatch(stored, new RegExp(secret), + "the key is in the control plane's own database in the open"); + + // Nor is it anywhere it could have been read on the way. + for (const where of ["/var/lib/mesh-host/declared.json", "/var/lib/mesh-host/state.json"]) { + // Whether grep found it, not how many times. `grep -c` prints 0 and exits non-zero when it + // finds nothing, so the obvious `|| echo 0` prints a second one and the count is never what + // it looks like. + const found = await on("laptop", `grep -q ${quote(secret)} ${where}`); + assert.ok(!found.ok, `the key is in the open in ${where}`); + } +}); + +test("a new commit reaches a machine that is already running the old one", { + skip: skip || (!builder ? "set MESH_LAB_BUILDER to a built mesh-builder" : false), + timeout: 900_000, +}, async () => { + // novox/hq ADR 0010 names the real risk of replacing a pipeline with a comparison: losing the + // question "did my change go out?". This is that question, end to end — a commit, a build, a + // catalogue, and a machine that ends up running what the source says. + const repo = "/var/lib/mesh/builder/repositories/delivered"; + const write = async (what: string) => + await must("anchor", `mkdir -p ${repo} && printf %s '{"module":"delivered","version":"1",` + + `"resources":[{"id":"marker","type":"file","path":"/etc/delivered",` + + `"content":"${what}","mode":"0644"}]}' > ${repo}/module.json`); + + await write("first"); + await must("anchor", `cd ${repo} && git init -q . && git add -A && ` + + `git -c user.email=lab -c user.name=lab commit -qm first`); + await mesh(`build ${repo} --wait 300s`, 420_000); + await mesh("assign laptop delivered"); + await mesh("push laptop"); + await new Promise((r) => setTimeout(r, 15_000)); + assert.equal((await must("laptop", `cat /etc/delivered`)).trim(), "first"); + + // Nothing has moved, so nothing is behind — and it says so rather than doing nothing quietly. + assert.match(await mesh("build --behind"), /every module the mesh holds is what its source last had/); + + // Now the source moves. + await write("second"); + await must("anchor", `cd ${repo} && git add -A && ` + + `git -c user.email=lab -c user.name=lab commit -qm second`); + const moved = (await must("anchor", `cd ${repo} && git rev-parse HEAD`)).trim(); + await mesh(`module moved delivered ${moved}`); + + // The mesh says which module is behind, and by how much, before anything is built. + const behind = await mesh("status"); + assert.match(behind, /delivered/, `status does not name the module that moved:\n${behind}`); + assert.match(behind, /build --behind/, `status does not say how to catch up:\n${behind}`); + + // The loop, in one command: everything behind its source is built and recorded. + const built = await mesh("build --behind --wait 300s", 420_000); + assert.match(built, /delivered/, built); + + // The machine is still running the old one until it is told — the mesh changing its mind is + // not the same as a machine acting on it, and collapsing the two is how a mesh reports success + // for something that has not happened. + assert.equal((await must("laptop", `cat /etc/delivered`)).trim(), "first", + "the machine changed before anything was sent to it"); + + await mesh("push laptop"); + await new Promise((r) => setTimeout(r, 15_000)); + assert.equal((await must("laptop", `cat /etc/delivered`)).trim(), "second", + "the machine is still running what the source no longer says"); + + // And it is no longer out of date, which is the half that makes the answer trustworthy: a + // status that says "behind" for ever is one nobody reads. + const after = await mesh("status"); + assert.doesNotMatch(after, /delivered.*<.*[0-9a-f]{8}/, + `the module is still reported as behind after catching up:\n${after}`); +}); + +test("the board names the machine that is not doing what it was told", { + skip, timeout: 900_000, +}, async () => { + // novox/hq 03-DESIGN/01-to-be/11-a-board.md. The board being replaced reads every context's + // database directly; this one asks the same questions through the same functions the commands + // use, and holds nothing. So the check is that what it says matches what the mesh says, and + // that it says the thing a person opened it for. + await must("anchor", `printf %s '{"module":"board","version":"1",` + + `"listens":[{"port":8090,"from":"mesh","why":"the board"}],` + + `"resources":[]}' > /tmp/board.json`); + await must("anchor", `docker cp /tmp/board.json mesh-control:/board.json`); + await mesh("module add /board.json"); + + // Served from the control plane's own container, reading the mesh on every request. + await must("anchor", `docker exec -d mesh-control /mesh-control board --listen 0.0.0.0:8090`); + await new Promise((r) => setTimeout(r, 3000)); + + const read = async (path: string) => + await on("anchor", `curl -sf http://127.0.0.1:8090${path}`, 30_000); + + let up = false; + let said = { out: "", ok: false }; + for (let i = 0; i < 15 && !up; i++) { + said = await read("/"); + up = said.ok; + if (!up) await new Promise((r) => setTimeout(r, 2000)); + } + assert.ok(up, `the board does not answer:\n${said.out}`); + + // Something a person opened it for: give a machine a declaration it cannot apply. + // + // A unit that does not exist, because the host says so immediately and in its own words. A file + // in a missing directory is not impossible — the host creates the parents, which is correct and + // made the first version of this test break nothing at all. + await must("anchor", `printf %s '{"module":"impossible","version":"1",` + + `"resources":[{"id":"nowhere","type":"service","unit":"nothing-like-this.service",` + + `"state":"running"}]}' > /tmp/impossible.json`); + await must("anchor", `docker cp /tmp/impossible.json mesh-control:/impossible.json`); + await mesh("module add /impossible.json"); + await mesh("assign laptop impossible"); + await mesh("push laptop"); + + let named = false; + let page = ""; + for (let i = 0; i < 20 && !named; i++) { + page = (await read("/")).out; + named = page.includes("laptop") && page.includes(">failed<"); + if (!named) await new Promise((r) => setTimeout(r, 3000)); + } + assert.ok(named, `the board does not name the machine that failed:\n${page}`); + + // The host's own words, which say exactly what it could not do. A board that said only + // "failed" would make a person go and ask the thing they opened the board to avoid asking. + assert.match(page, /nowhere/, `the board does not say what failed:\n${page}`); + + // It agrees with the command, because both read the same thing. Two answers to "which machine + // is broken" is worse than either alone. + const asJSON = JSON.parse((await read("/mesh.json")).out); + assert.equal(asJSON.wrong[0].node, "laptop", `the page and the JSON disagree: ${JSON.stringify(asJSON)}`); + assert.equal(asJSON.wrong[0].outcome, "failed"); + + const fromCommand = JSON.parse(await mesh("status --json")); + assert.deepEqual(asJSON.wrong, fromCommand.wrong, + "the board and the command disagree about which machine is broken"); + + // And it changes nothing: the mesh is exactly as it was after being read. + const before = await mesh("status"); + await read("/"); + assert.equal(await mesh("status"), before, "reading the board changed the mesh"); + + await mesh("unassign laptop impossible"); + await mesh("push laptop"); +}); + +// Defends novox/hq ADR 0007: filtering the hub must not cut the overlay it carries. +test("the hub can be filtered without severing the mesh", { + skip, timeout: 900_000, +}, async () => { + // The machine that most needs a firewall was the one that could not have one. A hub is dialled + // by every node at other sites; a machine that is not a hub dials out and needs nothing open. + // They are the same module, so a static `listens` cannot say it — and the machine it gets wrong + // is the one facing the public internet. + // + // The failure this guards against is not subtle and is very hard to recover from: a rule set + // that closes the hub's own port takes the private network down, and the mesh's way of fixing + // anything is to send a declaration over it. + // Its own directory. Another module on this machine already declares /etc/mesh, and the mesh + // refuses two modules declaring one path rather than letting the second quietly win — which it + // did here, correctly, the first time this ran. + const rules = "/etc/mesh-hub/filter.nft"; + await must("anchor", `printf %s '{"module":"hubfilter","version":"1",` + + `"capabilities":["firewall"],` + + `"filtering":{"into":"${rules}"},` + + `"resources":[{"id":"nftables","type":"package","package":"nftables"},` + + `{"id":"dir","type":"directory","path":"/etc/mesh-hub","mode":"0755"},` + + `{"id":"unit","type":"file","path":"/etc/systemd/system/hub-filter.service",` + + `"mode":"0644","content":"[Unit]\\nDescription=What the mesh computed for the hub\\n` + + `[Service]\\nType=oneshot\\nRemainAfterExit=yes\\n` + + `ExecStart=/usr/bin/nft -f ${rules}\\n[Install]\\nWantedBy=multi-user.target\\n"},` + + `{"id":"filter","type":"service","unit":"hub-filter.service","state":"running",` + + `"boot":"enabled","restart-on":["filtering"]}]}' > /tmp/hubfilter.json`); + await must("anchor", `docker cp /tmp/hubfilter.json mesh-control:/hubfilter.json`); + await mesh("module add /hubfilter.json"); + await mesh("assign anchor hubfilter"); + await mesh("push anchor"); + await new Promise((r) => setTimeout(r, 20_000)); + + // The hub's own way onto the private network is open, and derived — nothing in that manifest + // mentions a port. + const written = await must("anchor", `cat ${rules}`); + assert.match(written, /udp dport 51820 accept/, + `the hub's rule set closes the private network it is the way onto:\n${written}`); + // The module that provides the private network, not the requirement it answers: `networking` + // is the domain a module offers, and what caused a rule is the module itself. + assert.match(written, /# mesh-wireguard — the private network/, + `the rule does not name what caused it:\n${written}`); + + // Loaded, and the mesh still works: a declaration reaches the other machine, which it cannot if + // the overlay is severed. This is the assertion that matters — a rule file that looks right and + // a mesh that has stopped are exactly what this is guarding against. + assert.match(await must("anchor", `nft list table inet mesh`), /dport 51820/); + + await must("laptop", `rm -f /etc/mesh-still-works`); + await must("anchor", `printf %s '{"module":"stillworks","version":"1",` + + `"resources":[{"id":"marker","type":"file","path":"/etc/mesh-still-works",` + + `"content":"yes","mode":"0644"}]}' > /tmp/stillworks.json`); + await must("anchor", `docker cp /tmp/stillworks.json mesh-control:/stillworks.json`); + await mesh("module add /stillworks.json"); + await mesh("assign laptop stillworks"); + await mesh("push laptop"); + + let arrived = false; + for (let i = 0; i < 20 && !arrived; i++) { + arrived = (await on("laptop", `test -f /etc/mesh-still-works`)).ok; + if (!arrived) await new Promise((r) => setTimeout(r, 3000)); + } + assert.ok(arrived, + "the hub applied its own rule set and the mesh stopped reaching the other machine"); + + // And the other machine still reaches the hub over the private network, which is what the + // opened port is for. + assert.ok((await on("laptop", `ping -c 1 -W 5 anchor.internal`)).ok, + "the private network is down after the hub filtered itself"); + + await mesh("unassign anchor hubfilter"); + await mesh("unassign laptop stillworks"); + await mesh("push"); +}); + +test("a container reaches another machine by the name the mesh gave it", { + skip, timeout: 900_000, +}, async () => { + // Internal names are written to the machine's hosts file, which serves the machine and not what + // the machine runs: a container gets its own hosts file holding only its own hostname. So every + // name the mesh wrote was invisible to the majority of things that need one. + // + // Checked from inside a container rather than on the machine, because on the machine it has + // always worked — and that is exactly what made this easy to miss. + await must("anchor", `printf %s '{"module":"resolves","version":"1",` + + `"capabilities":["container-runtime"],` + + `"resources":[{"id":"idle","type":"container","name":"resolves",` + + `"image":"${pinned("registry")}"}]}' > /tmp/resolves.json`); + await must("anchor", `docker cp /tmp/resolves.json mesh-control:/resolves.json`); + await mesh("module add /resolves.json"); + await mesh("assign laptop resolves"); + await mesh("push laptop"); + await new Promise((r) => setTimeout(r, 15_000)); + + // The names are in the container's own hosts file, written by the runtime. + const inside = await must("laptop", `docker exec resolves cat /etc/hosts`); + assert.match(inside, /anchor\.internal/, + `the container cannot see the other machine's name:\n${inside}`); + assert.match(inside, /laptop\.internal/, + `the container cannot see its own machine's name:\n${inside}`); + + // And the name actually reaches the machine, which is the part that matters: a hosts entry + // pointing at the wrong address resolves perfectly and connects to nothing. + const reached = await on("laptop", + `docker exec resolves sh -c 'getent hosts anchor.internal'`); + assert.ok(reached.ok, `the name does not resolve inside the container:\n${reached.out}`); + const address = reached.out.trim().split(/\s+/)[0]; + const onTheMachine = (await must("laptop", + `getent hosts anchor.internal | head -1 | cut -d' ' -f1`)).trim(); + assert.equal(address, onTheMachine, + "the container and its machine disagree about where the other machine is"); + + await mesh("unassign laptop resolves"); + await mesh("push laptop"); +}); + +// Defends novox/hq ADR 0007: a name under a machine is that machine, without the mesh being +// told each one. +test("every name under a machine resolves to that machine", { + skip, timeout: 900_000, +}, async () => { + // Services are named under the machine they run on — postgres.novox.internal, + // plex.ace.internal. The first label is the service and the rest is the node, so what must + // resolve is anything under a node's name. What routes it once it arrives is a proxy's, and + // stays separate. + // + // The mesh writes the data and runs no daemon: a resolver is third-party software, and the + // mesh has no business choosing one. So what is checked here is the mesh's half — that the + // data is right, complete, and follows the machines. + await mesh("assign anchor mesh-resolver"); + await mesh("assign laptop mesh-resolver"); + await mesh("push"); + await new Promise((r) => setTimeout(r, 15_000)); + + for (const machine of ["anchor", "laptop"]) { + const written = await must(machine, `cat /etc/mesh-resolver/nodes.conf`); + + // A wildcard per machine, matching the name and everything under it. Both machines get the + // whole mesh: a node resolves every other node, and itself. + for (const node of ["anchor", "laptop"]) { + assert.match(written, new RegExp(`address=/${node}\\.internal/10\\.42\\.0\\.\\d+`), + `${machine} cannot resolve names under ${node}:\n${written}`); + } + + // And the addresses agree with what the machine's own hosts file says. Two accounts of where + // a machine is, disagreeing, would be worse than either alone — and this is the one place + // they could drift, because they are generated separately. + const hosts = await must(machine, `getent hosts anchor.internal | head -1 | cut -d' ' -f1`); + assert.match(written, new RegExp(`address=/anchor\\.internal/${hosts.trim().replace(/\./g, "\\.")}`), + `the resolver data and the hosts file disagree about where anchor is:\n${written}`); + } + + // It follows the machines. A node leaving the private network must stop being answered for, + // because a wildcard pointing at nothing resolves and then hangs — where an unresolvable name + // fails at once and says which name it was. + // + // Both, and that is not tidiness: `mesh-resolver` requires name resolution, which requires the + // network, so unassigning the domain module alone leaves the machine on the network — pulled + // back by its own requirement. The mesh was right and this test was wrong the first time. + await mesh("unassign laptop mesh-resolver"); + await mesh("unassign laptop networking"); + await mesh("push anchor"); + await new Promise((r) => setTimeout(r, 15_000)); + + const after = await must("anchor", `cat /etc/mesh-resolver/nodes.conf`); + assert.doesNotMatch(after, /address=\/laptop\.internal\//, + `a machine that left the private network is still answered for:\n${after}`); + assert.match(after, /address=\/anchor\.internal\//, + `the machine that stayed lost its own name:\n${after}`); + + await mesh("assign laptop networking"); + await mesh("unassign anchor mesh-resolver"); + await mesh("push"); + await new Promise((r) => setTimeout(r, 15_000)); +}); + +test("a service is reached by a name under the machine it runs on", { + skip: skip || (!moduleExamples ? "set MESH_LAB_MODULES to mesh-control's examples/modules" : false), + timeout: 900_000, +}, async () => { + // postgres.novox.internal, plex.ace.internal — the first label is the service and the rest is + // the node, so anything under a node's name must resolve to that node. What routes it once it + // arrives is a proxy's concern and stays separate. + // + // The mesh writes the data; a module runs the daemon. Both manifests are read from the + // repository rather than written here, so what is proven is what ships. + // `resolved-split-dns`, not `resolv-conf`: these machines run systemd-resolved, which owns + // /etc/resolv.conf. The two claim the same thing precisely so that assigning the wrong one is a + // refusal rather than a fight over the file — and picking the wrong one here would have been + // testing that fight. + for (const name of ["dnsmasq", "resolved-split-dns"]) { + const manifest = readFileSync(`${moduleExamples}/${name}.json`, "utf8"); + await must("anchor", `cat > /tmp/${name}.json <<'MANIFEST'\n${manifest}\nMANIFEST`); + await must("anchor", `docker cp /tmp/${name}.json mesh-control:/${name}.json`); + await mesh(`module add /${name}.json`); + } + + // Both machines, because a node resolves from its own copy — the same rule as everything else + // it holds. A mesh where one machine answers for all of them stops resolving when that machine + // does, which is the arrangement this design refuses everywhere else. + for (const machine of ["anchor", "laptop"]) { + await mesh(`assign ${machine} dnsmasq`); + await mesh(`assign ${machine} resolved-split-dns`); + } + await mesh("push"); + await new Promise((r) => setTimeout(r, 25_000)); + + // Everything this test could want to know, gathered in one place. + // + // Three times now a diagnostic has not run because the thing before it threw: `must` on a + // command that fails, and then a query that hangs long enough to take the harness's own timeout + // with it. A 30-second test costs fifteen minutes to re-run, so the evidence has to be gathered + // whether the failure is an assertion, an error, or a hang. + const diagnose = async (machine: string) => + `--- ${machine}\n` + + `dnsmasq: ${(await on(machine, `systemctl is-active dnsmasq.service`)).out.trim()}\n` + + `${(await on(machine, `journalctl -u dnsmasq -n 12 --no-pager`)).out}\n` + + `resolved: ${(await on(machine, `resolvectl status | head -30`)).out}\n` + + `resolv.conf:\n${(await on(machine, `cat /etc/resolv.conf`)).out}\n` + + `what the mesh wrote:\n${(await on(machine, `cat /etc/mesh-resolver/nodes.conf`)).out}\n` + + `listening:\n${(await on(machine, `ss -lnup | grep :53 || echo none`)).out}\n` + + `asked directly:\n${(await on(machine, + `timeout 5 resolvectl query postgres.anchor.internal 2>&1 || echo "no answer"`)).out}`; + + // `on`, not `must`: `is-active` exits non-zero for a unit that failed, so `must` would throw + // before the assertion below — taking every diagnostic with it. That happened, and the run said + // only "failed". + for (const machine of ["anchor", "laptop"]) { + const state = await on(machine, `systemctl is-active dnsmasq.service`); + if (state.out.trim() === "active") continue; + assert.fail(`the resolver is not running on ${machine} (${state.out.trim()}):\n\n` + + `its config:\n${(await on(machine, `cat /etc/dnsmasq.conf`)).out}\n` + + `${await diagnose(machine)}`); + } + + // Through the machine's own resolver, by the path an application actually takes: nsswitch, then + // files, then DNS. `dig` would ask a server directly and prove less — the resolv.conf module is + // half of what is being tested, and only this path goes through it. + // + // Bounded on the machine rather than by the harness: a query that hangs is a result, and letting + // it run into the harness's own timeout turns it into an error with no evidence attached. + const resolves = async (machine: string, name: string) => { + const said = await on(machine, + `timeout 5 getent hosts ${name} | head -1 | cut -d' ' -f1`, 20_000); + return said.out.trim(); + }; + const addressOf = async (machine: string, node: string) => + (await must(machine, `getent hosts ${node}.internal | head -1 | cut -d' ' -f1`)).trim(); + + const anchorAt = await addressOf("anchor", "anchor"); + const laptopAt = await addressOf("anchor", "laptop"); + + // A name the mesh was never told about, under a machine it was — from both machines, because a + // node must answer for every machine and not only for itself. + for (const machine of ["anchor", "laptop"]) { + let got = ""; + for (let i = 0; i < 15 && !got; i++) { + got = await resolves(machine, "postgres.anchor.internal"); + if (!got) await new Promise((r) => setTimeout(r, 3000)); + } + assert.equal(got, anchorAt, + `${machine} does not resolve a service named under anchor: ${got || "(nothing)"}\n\n` + + `${await diagnose(machine)}`); + } + + // Any name at all, which is the whole point: the mesh was never told these exist. + for (const [name, expected] of [ + ["postgres-2.anchor.internal", anchorAt], + ["keycloak.anchor.internal", anchorAt], + ["plex.laptop.internal", laptopAt], + ["radarr.laptop.internal", laptopAt], + ] as const) { + assert.equal(await resolves("laptop", name), expected, + `${name} did not resolve to the machine it is named under\n\n${await diagnose("laptop")}`); + } + + // The machine's own name still resolves, and to the same place. Two accounts of where a machine + // is, disagreeing, would be worse than either alone. + assert.equal(await resolves("laptop", "anchor.internal"), anchorAt); + + // And what is not the mesh's is not answered by it. The resolver takes over the mesh's names + // and nothing else, which is what lets a machine keep whatever DNS it already had. + assert.equal(await resolves("anchor", "something.example.com"), "", + "the resolver answered for a name that is not the mesh's"); + + for (const machine of ["anchor", "laptop"]) { + await mesh(`unassign ${machine} resolved-split-dns`); + await mesh(`unassign ${machine} dnsmasq`); + } + await mesh("push"); +}); + +// A real third-party workload, adopted the way the conversion will adopt one. +// +// **Everything before this used modules written to exercise the mesh.** This one is software +// nobody here wrote, taking its credentials the way such software does — from its environment — +// and needing two containers that reach each other by name. It is the first module that could not +// have been declared before today: it needs the `network` shape, and it needs a sealed value to +// reach a container's environment. +// +// Its database password is **accepted rather than generated**, which is the whole shape of an +// adoption: a service that already exists keeps the credential it already has, because minting a +// new one is how a running application stops being able to reach its own database. +test("a third-party workload is adopted, with the credential it already had", { + skip, timeout: 900_000, +}, async () => { + const password = "the-password-it-already-had"; + + await must("anchor", `printf %s ${quote(JSON.stringify({ + module: "umami", + version: "1", + capabilities: ["container-runtime"], + "own-secrets": { + database: "/var/lib/umami/database.env", + app: "/var/lib/umami/app.env", + }, + listens: [{ port: 1212, protocol: "tcp", from: "mesh", why: "the analytics page" }], + resources: [ + { id: "state", type: "directory", path: "/var/lib/umami", mode: "0700" }, + // The two containers must reach each other by name, which is what this shape is for. + { id: "net", type: "network", name: "umami" }, + { + id: "db", type: "container", name: "umami-db", + image: pinned("postgres"), + network: "umami", + env: { POSTGRES_DB: "umami", POSTGRES_USER: "umami" }, + "env-file": ["/var/lib/umami/database.env"], + }, + { + id: "app", type: "container", name: "umami", + image: pinned("ghcr.io/umami-software/umami"), + network: "umami", + env: { DATABASE_TYPE: "postgresql" }, + "env-file": ["/var/lib/umami/app.env"], + ports: ["1212:3000"], + }, + ], + }))} > /umami.json`); + await must("anchor", `docker cp /umami.json mesh-control:/umami.json`); + await mesh("module add /umami.json"); + + // **Accepted, not generated.** The value is what the database already answers to; the mesh + // seals it and cannot read it again. Given whole, as the environment lines the containers read. + await must("anchor", + `printf %s ${quote(`POSTGRES_PASSWORD=${password}`)} | ` + + `docker exec -i mesh-control /mesh-control secret accept anchor umami database --from -`); + await must("anchor", + `printf %s ${quote( + `DATABASE_URL=postgresql://umami:${password}@umami-db:5432/umami`)} | ` + + `docker exec -i mesh-control /mesh-control secret accept anchor umami app --from -`); + + await mesh("assign anchor umami"); + await mesh("push anchor", 300_000); + + // Both containers, and the network they share. + let up = false; + for (let i = 0; i < 60 && !up; i++) { + const running = await on("anchor", `docker ps --format '{{.Names}}'`); + up = running.out.includes("umami-db") && running.out.includes("umami"); + if (!up) await new Promise((r) => setTimeout(r, 5000)); + } + if (!up) { + // Everything that could say why, gathered before asserting. "It did not start" is the one + // thing already known; what is wanted is whether the mesh sent it, whether the host refused + // it, and what the runtime said when it tried. + const said = await mesh("status"); + const containers = await on("anchor", `docker ps -a --format '{{.Names}} {{.Status}}'`); + const applied = await on("anchor", + `${HOST_PATH} owned 2>&1 | head -30 || echo "the host could not say what it owns"`); + const files = await on("anchor", `ls -la /var/lib/umami/ 2>&1; ` + + `for f in /var/lib/umami/*.env; do echo "-- $f"; wc -c "$f"; done 2>&1`); + const tried = await on("anchor", + `docker inspect umami-db --format '{{.State.Status}} {{.State.Error}}' 2>&1; ` + + `docker logs umami-db 2>&1 | tail -15`); + assert.fail( + `the workload never started.\n\n` + + `── what the mesh thinks:\n${said}\n` + + `── containers:\n${containers.out}\n` + + `── what the host owns:\n${applied.out}\n` + + `── what the mesh wrote:\n${files.out}\n` + + `── the database container:\n${tried.out}\n`); + } + + // The environment file the mesh sealed is on the machine and readable only by root. + const mode = await must("anchor", `stat -c %a /var/lib/umami/database.env`); + assert.equal(mode.trim(), "600", "a file holding a credential is readable by more than root"); + + // **The assertion that matters: the credential works.** Not that a file arrived — that the + // database the mesh started answers to the password the mesh was given rather than one it made. + let connected = { out: "", ok: false }; + for (let i = 0; i < 40 && !connected.ok; i++) { + connected = await on("anchor", + `docker exec umami-db psql -U umami -d umami -qAt -c 'select 1'`); + if (!connected.ok) await new Promise((r) => setTimeout(r, 3000)); + } + assert.ok(connected.ok, `the database never came up:\n${connected.out}`); + + const wrong = await on("anchor", + `docker run --rm --network umami -e PGPASSWORD=not-the-password ${pinned("postgres")} ` + + `psql -h umami-db -U umami -d umami -qAt -c 'select 1'`); + assert.ok(!wrong.ok, + "the database accepted a password nobody gave it, so this proves nothing about the one that was"); + + // And the two containers reach each other by name over the module's own network. + const reached = await must("anchor", + `docker run --rm --network umami ${pinned("postgres")} ` + + `sh -c 'getent hosts umami-db || echo unreachable'`); + assert.doesNotMatch(reached, /unreachable/, + "a container could not reach the other by name, so the module's network did nothing"); +}); + +// The real modules, resolved together on one machine. +// +// **What this proves without pulling a gigabyte of images**: that five manifests written from the +// running system resolve as a graph — keycloak's requirement met by postgres's provision, +// capabilities checked, nothing claiming the same singular thing — and that the declaration the +// control plane composes is one the host accepts. `plan --json` exists for exactly this: it is +// the only way to know that what the control plane emits is what the host takes. +// +// Running them needs their images stocked and two provisioners built, which is a separate and +// larger job. This is the half that can be known now, and it is the half where a design fault +// would live. +test("the real modules resolve together, and compose a declaration a host accepts", { + skip, timeout: 300_000, +}, async (t) => { + // Everything the catalogue holds, except two whose names this mesh is already running under: + // `registry` is the artifact store the suite stood up, and `umami` is the adopted workload — + // adding the catalogue's manifests would replace the records of modules that are live and + // assigned, and the adopted umami would suddenly require a database it never asked for. + const modules = ["postgres", "keycloak", "gitea", "minio", "mailu", + "redis", "grafana", "nextcloud", "searxng", "influxdb", "verdaccio"]; + const planned: string[] = []; + for (const name of modules) { + const raw = readFileSync( + `${process.env["MESH_LAB_MODULES"]}/${name}.json`, "utf8"); + // Pointed at this scenario's registry before being added, exactly as the forge is. + // + // **Not cosmetic.** Some of these name an image the mesh builds, whose digest does not exist + // until it is built — so the file legitimately carries a placeholder, and composing a + // declaration from it is refused (novox/hq 04-ISSUES/025). Planning what could never run is + // what this test used to do. + const pinned = pinnedInto(raw, stocked); + // What this scenario does not serve cannot be redirected, and a module still naming a + // placeholder cannot be planned — the refusal is the point (novox/hq 04-ISSUES/025). Skipped + // and said, rather than silently dropped: a planning test quietly covering four modules + // instead of five is the false coverage this suite exists to prevent. + const left = stillUnpinned(pinned); + if (left.length > 0) { + console.log(`skipping ${name}: this scenario serves no ${left.join(", ")}`); + continue; + } + planned.push(name); + await must("anchor", `printf %s ${quote(pinned)} > /${name}.json`); + await must("anchor", `docker cp /${name}.json mesh-control:/${name}.json`); + await mesh(`module add /${name}.json`); + } + + // **Put the machine back whatever happens.** Tests here share one mesh, so what this one + // assigns is what the next one inherits. Written at the end of the body once, it was skipped + // the first time this test failed — and the next test's push was refused by a module this one + // had left behind, which reads as a fault in the test that was actually working. + t.after(async () => { + for (const name of planned) await mesh(`unassign anchor ${name}`).catch(() => {}); + }); + + // Assigned one at a time, because assignment resolves the whole set and says so immediately. + // A refusal here is the graph rejecting something, which is the point of asking. + for (const name of planned) { + await mesh(`assign anchor ${name}`); + } + + const plan = await mesh("plan anchor --json", 120_000); + const declaration = JSON.parse(plan.slice(plan.indexOf("{"))); + const byId = new Map( + (declaration.resources as any[]).map((r) => [r.id, r])); + const ids = [...byId.keys()]; + + // Every module's own network, which only exists because more than one container needs to reach + // another by name. + for (const id of ["postgres.net", "keycloak.net", "minio.net", "mailu.net"]) { + assert.ok(byId.has(id), `${id} is missing; ${ids.length} resources: ${ids.join(", ")}`); + assert.equal(byId.get(id).type, "network"); + } + + // The cross-module edge: keycloak asked for a database and was told where it is and given a + // credential. Neither file is anything keycloak's manifest could have written. + const bound = [...byId.values()].find((r) => + r.type === "file" && r.path === "/var/lib/keycloak/database.json"); + assert.ok(bound, `keycloak was never told where its database is: ${ids.join(", ")}`); + assert.match(JSON.stringify(bound), /postgres/, + "keycloak's binding does not name what answered its requirement"); + + // The password, alone in a file and sealed. It is a password and nothing else, so nothing reads + // it as configuration — novox/hq 04-ISSUES/023 and the playbook both turn on that distinction. + const credential = [...byId.values()].find((r) => + r.type === "file" && r.path === "/var/lib/keycloak/database.secret"); + assert.ok(credential, `keycloak was given no credential for its database: ${ids.join(", ")}`); + assert.ok(credential.sealed, "keycloak's credential is not sealed, so the mesh can read it"); + assert.ok(!credential.content, "a credential arrived as content rather than sealed"); + + // And the connection itself, which keycloak could not have written: the address and port come + // from what the provider serves, and the user name from what the mesh decided both ends would + // call this consumer (novox/hq 04-ISSUES/023). + const connection = [...byId.values()].find((r) => + r.type === "file" && r.path === "/var/lib/keycloak/database.env"); + assert.ok(connection, "keycloak was given no database configuration"); + assert.match(connection.content, /KC_DB_USERNAME=mesh_[a-z0-9_]+_keycloak/, + `keycloak was not told what name to present:\n${connection.content}`); + assert.doesNotMatch(connection.content, /\$\{bound:/, + `a placeholder reached the machine as a value:\n${connection.content}`); + + // The password is the one hole left open, and the sealed value travels beside it. The mesh + // discarded the plaintext, so the host is the only thing that can close it. + assert.match(connection.content, /KC_DB_PASSWORD=\$\{secret:postgres-database\}/, + `the password was not left for the host to fill:\n${connection.content}`); + assert.ok(connection.secrets?.["postgres-database"], + "the sealed credential did not travel with the file that needs it"); + assert.doesNotMatch(JSON.stringify(connection.content), /postgres-database":"[A-Za-z0-9+/]{24,}/, + "the credential was written into the configuration in the clear"); + + // And the provider was told who asked, which is what its provisioner reconciles against. + const grants = [...byId.values()].find((r) => + r.type === "file" && String(r.path).startsWith("/var/lib/postgres/grants")); + assert.ok(grants, "postgres was never told which modules were granted a database"); + assert.match(JSON.stringify(grants), /keycloak|gitea/, + "the grants file names neither module that asked for a database"); + + // Secrets reach containers as files, never as environment in the declaration. + const containers = [...byId.values()].filter((r) => r.type === "container"); + assert.ok(containers.length >= 12, + `only ${containers.length} containers; mailu alone is nine`); + for (const c of containers) { + for (const [key, value] of Object.entries(c.env ?? {})) { + // **An absolute path is a reference to a secret, not a secret**, and naming one is the + // whole design: the mesh delivers a credential as a file and a module says where. + // + // Excluded because `/` is in the base64 alphabet, so any path of 24 characters or more + // matched — `MESH_BROKER_FILE=/var/lib/mesh/builder/broker` was reported as a credential + // the broker would see. A check that fires on the right shape for the wrong reason is + // worse than none: it is the one that gets suppressed, and then it is not there when it + // is right. + if (String(value).startsWith("/")) continue; + assert.doesNotMatch(String(value), /^[A-Za-z0-9+/]{24,}={0,2}$/, + `${c.name} carries something secret-shaped in env.${key}, which the broker would see`); + } + } + +}); + +// The first of the real module descriptions to actually run. +// +// **Everything before this stopped at composing a declaration.** That proves the control plane and +// the host agree, and proves nothing about whether the thing described works — which is how five +// modules sat pinned to images that did not exist, parsing and resolving perfectly +// (novox/hq 04-ISSUES/025). +// +// The forge is the one worth running first. It needs a database from another module, a password it +// did not choose, and a connection string it could not have written itself: the address and port +// come from what the database serves, and the user name from what the mesh decided both ends would +// call it (04-ISSUES/022 and 023). If any of that is wrong it cannot start, and nothing else in +// this file would notice. +test("the forge runs, on a database the mesh gave it", { skip, timeout: 900_000 }, async () => { + for (const name of ["postgres", "gitea"]) { + const raw = readFileSync(`${process.env["MESH_LAB_MODULES"]}/${name}.json`, "utf8"); + // An image the mesh builds has no digest until it is built, and one it does not build belongs + // to whichever registry served it. Both are answered by this scenario's own registry. + const pinned = pinnedInto(raw, stocked); + assert.deepEqual(stillUnpinned(pinned), [], + `${name} still names an image nothing serves, so it could not start`); + await must("anchor", `printf %s ${quote(pinned)} > /run-${name}.json`); + await must("anchor", `docker cp /run-${name}.json mesh-control:/run-${name}.json`); + await mesh(`module add /run-${name}.json`); + await mesh(`assign anchor ${name}`); + } + await mesh("push anchor", 300_000); + + // **What the machine says it did, before asking what it produced.** This test pushed and then + // waited for a database role, so when the containers were never created at all it reported "no + // login was created" — true, and silent about the reason. A push that was accepted and an apply + // that worked are different facts, and the second is the one this depends on. + // + // And waited for, because `push` sends without waiting. Reading `status` the instant it returns + // describes the apply *before* this one, which is how this test came to report a missing + // container while insisting the machine was fine. + await settled("anchor"); + const running = (await on("anchor", `docker ps -a --format '{{.Names}} {{.Status}}'`)).out; + // Named with what the mesh meant to send, not only with what the machine has. A container that + // is absent because the mesh never asked for it and one that is absent because the machine could + // not make it are the same sentence here and different faults entirely, and the plan is the only + // thing that tells them apart. + assert.match(running, /\bpostgres\b/, + `the database module was pushed and no container for it exists:\n${running}\n\n` + + `what the mesh would send anchor:\n${await mesh("plan anchor")}\n\n` + + `${(await on("anchor", `tail -30 /var/log/mesh-host.log`)).out}`); + + // The database first: until the provisioner has made the login, the forge has nothing to + // connect to and its own start would prove only that it retries. + const psql = async (q: string) => + (await on("anchor", + `docker exec postgres psql -U postgres -qAt -c ${quote(q)}`, 60_000)).out.trim(); + + // Both halves in one poll. The provisioner makes the role and then the database, and a test + // that waited for the first and checked the second once was racing the gap between two + // statements — it lost, once, eighteen seconds into a run. + let made = ""; + for (let i = 0; i < 40 && made !== "t"; i++) { + made = await psql("select true from pg_roles where rolname = 'mesh_anchor_gitea'" + + " and exists (select from pg_database where datname = 'gitea')"); + if (made !== "t") await new Promise((r) => setTimeout(r, 3000)); + } + assert.equal(made, "t", + `no login was created for the forge:\n${(await on("anchor", "docker logs mesh-provision-postgres 2>&1 | tail -20")).out}`); + + // And the forge itself, answering. Not that its container exists — that it serves. + // + // On the port the mesh assigned, not the one the module declared (novox/hq ADR 0038): the + // module says 3000 and the machine publishes wherever the mesh put it. Read from the plan, + // because the plan is the same composition a push sends. + const planned = await mesh("plan anchor --json", 120_000); + const mapping = (JSON.parse(planned.slice(planned.indexOf("{"))).resources as any[]) + .find((r) => r.id === "gitea.server")?.ports + ?.map(String).find((p: string) => p.endsWith(":3000")); + assert.ok(mapping, "the plan does not say where the machine publishes the forge"); + const at = mapping.split(":")[0]; + let answered = false; + let said = { out: "", ok: false }; + for (let i = 0; i < 60 && !answered; i++) { + said = await on("anchor", `curl -sf -o /dev/null -w '%{http_code}' --max-time 5 http://127.0.0.1:${at}/`, 30_000); + answered = said.out.trim().startsWith("2") || said.out.trim() === "303"; + if (!answered) await new Promise((r) => setTimeout(r, 5000)); + } + assert.ok(answered, + `the forge never answered (last: ${said.out.trim()}):\n` + + `${(await on("anchor", "docker logs gitea 2>&1 | tail -25")).out}`); + + // **The credential actually worked.** A forge that started and could not reach its database + // would still answer on its port, so the log is where the difference lives. + const log = (await on("anchor", "docker logs gitea 2>&1 | tail -60")).out; + assert.doesNotMatch(log, /password authentication failed|connection refused|does not exist/i, + `the forge started and could not use the database it was given:\n${log}`); + + await mesh("unassign anchor gitea"); + await mesh("unassign anchor postgres"); + await mesh("push anchor", 300_000); +}); + +test("a consumer's cache grant means exactly its own keys", { skip, timeout: 600_000 }, async (t) => { + // The third provision after a database and a bucket, and the first whose tenancy is enforced + // by the store's own ACL rather than by separate namespaces: every consumer shares one + // keyspace, so the grant is a pattern — and the test is that the pattern means what the + // manifest said, in both directions. + const raw = readFileSync(`${process.env["MESH_LAB_MODULES"]}/redis.json`, "utf8"); + const pinned = pinnedInto(raw, stocked); + assert.deepEqual(stillUnpinned(pinned), [], + "redis still names an image nothing serves, so it could not start"); + await must("anchor", `printf %s ${quote(pinned)} > /run-redis.json`); + await must("anchor", `docker cp /run-redis.json mesh-control:/run-redis.json`); + await mesh("module add /run-redis.json"); + + // A consumer with no container: what is under test is the credential's reach, and files on the + // machine are enough to prove it — the same reduction the first credential test makes. + await must("anchor", `printf %s '{"module":"cachetest","version":"1",` + + `"requires":["redis-cache"],` + + `"contributes":{"redis-cache":{"prefix":"cachetest"}},` + + `"binds":{"redis-cache":"/var/lib/cachetest/cache.json"},` + + `"secrets":{"redis-cache":"/var/lib/cachetest/cache.secret"},` + + `"resources":[{"id":"state","type":"directory","path":"/var/lib/cachetest","mode":"0700"}]}' ` + + `> /cachetest.json`); + await must("anchor", `docker cp /cachetest.json mesh-control:/cachetest.json`); + await mesh("module add /cachetest.json"); + + await mesh("assign anchor redis"); + await mesh("assign anchor cachetest"); + await mesh("push anchor", 300_000); + await settled("anchor"); + + t.after(async () => { + for (const name of ["cachetest", "redis"]) { + await mesh(`unassign anchor ${name}`).catch(() => {}); + } + await mesh("push anchor", 300_000).catch(() => {}); + }); + + // What the mesh told each end. The consumer's user name comes from its binding; the user's + // password from the sealed file beside it — both written by the host, neither invented here. + const bound = JSON.parse(await must("anchor", `cat /var/lib/cachetest/cache.json`)); + const user = bound.as; + assert.ok(user?.startsWith("mesh_"), `the binding does not carry a usable user: ${user}`); + const secret = (await must("anchor", `cat /var/lib/cachetest/cache.secret`)).trim(); + + // The provisioner has to have run before anything can authenticate. Waited for via the store + // itself: the user list, asked with the server's own password, which the conf file the host + // wrote holds on the machine. + const admin = (await must("anchor", + `awk '/^requirepass/ {print $2}' /var/lib/redis-module/redis.conf`)).trim(); + let granted = false; + for (let i = 0; i < 40 && !granted; i++) { + const users = (await on("anchor", + `docker exec redis redis-cli --no-auth-warning -a ${quote(admin)} ACL USERS`)).out; + granted = users.includes(user); + if (!granted) await new Promise((r) => setTimeout(r, 3000)); + } + assert.ok(granted, `no user was created for the consumer: +` + + `containers:\n${(await on("anchor", "docker ps -a --format '{{.Names}} {{.Status}}' | head -20")).out}\n` + + `the store:\n${(await on("anchor", "docker logs redis 2>&1 | tail -15")).out}\n` + + `the provisioner:\n${(await on("anchor", "docker logs mesh-provision-redis 2>&1 | tail -15")).out}`); + + const asConsumer = (command: string) => + on("anchor", `docker exec redis redis-cli --no-auth-warning ` + + `--user ${quote(user)} --pass ${quote(secret)} ${command}`); + + // Its own keys: usable. + assert.match((await asConsumer("SET cachetest:proof yes")).out, /OK/, + "the consumer cannot write under the prefix it was granted"); + assert.match((await asConsumer("GET cachetest:proof")).out, /yes/, + "the consumer cannot read back what it wrote"); + + // Anyone else's: refused by the store itself, which is the entire point of the grant. + assert.match((await asConsumer("SET other:proof no")).out, /NOPERM|no permissions/i, + "the consumer wrote outside its prefix — the grant means more than the manifest said"); + assert.match((await asConsumer("FLUSHALL")).out, /NOPERM|no permissions/i, + "the consumer can flush the store, which no tenant may"); +}); diff --git a/test/integration/minio-grant-end-to-end.test.ts b/test/integration/minio-grant-end-to-end.test.ts new file mode 100644 index 0000000..fbc860f --- /dev/null +++ b/test/integration/minio-grant-end-to-end.test.ts @@ -0,0 +1,258 @@ +/** + * The whole grant for an S3 bucket, mesh-driven — novox/hq ADR 0052/0053, the minio case. + * + * The postgres bed proves the provider/consumer contract for a database. This proves it for object + * storage, on a provider whose code drives the `mc` CLI (so the runtime image carries it): minio is + * assigned, a consumer that requires s3-bucket is assigned, and the mesh mints one secret key, sealing + * a copy to each end. minio's provisioner — reading only the mesh's contributions — creates a bucket + * and a service account under the access key the mesh derived, with the secret it minted. The proof is + * the consumer reaching its bucket with the access key and secret the mesh delivered it. Nothing is + * placed by the test. + * + * MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-module-runtime.sh minio builds mesh-runtime-minio:development (with mc), which + * scenarios/minio-node.yml stocks. minio/minio:latest must be in the local daemon. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +// 04-ISSUES/010 is fixed by ADR 0054: an S3 access key is capped at 20, and the mesh derives +// `mesh__`, so `bucketuser` on `anchor` (22) would overflow — but a consumer declares a +// short `slug` and its identity fits. This bed's consumer does exactly that. +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "minio-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +/** Same rule minio's client uses to name a bucket for a consumer — recomputed so the test knows it. */ +function bucketFor(as: string): string { + const name = as.toLowerCase().replace(/[^a-z0-9-]+/g, "-").replace(/^-+|-+$/g, "").slice(0, 63); + return name.length >= 3 ? name : `mesh-${name}`; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("the mesh grants a consumer an S3 bucket, and the credential it delivers reaches it", { + skip, timeout: 900_000, +}, async () => { + // The PROVIDER: minio in its committed shape — server and a broker-bound runtime (carrying mc) on + // the private minio network, the runtime running the provisioner. No published port on this + // single-node bed; the consumer reaches minio over the private network by name. + const minioManifest = JSON.stringify({ + module: "minio", + version: "1", + provides: [{ name: "s3-bucket", scope: "mesh" }], + serves: { "s3-bucket": { scheme: "http", region: "us-east-1" } }, + emits: ["module.minio.bucket.created", "module.minio.bucket.removed"], + receives: { "s3-bucket": "/var/lib/minio/grants/mesh.json" }, + grants: { "s3-bucket": "/var/lib/minio/grants" }, + "own-secrets": { root: "/var/lib/minio/root.secret", broker: "/var/lib/mesh/minio/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/minio", mode: "0700" }, + { id: "state", type: "directory", path: "/var/lib/minio", mode: "0700" }, + { id: "grants", type: "directory", path: "/var/lib/minio/grants", mode: "0700" }, + { id: "root-env", type: "file", path: "/var/lib/minio/root.env", mode: "0600", content: "MINIO_ROOT_USER=meshroot\nMINIO_ROOT_PASSWORD=${secret:root}\n" }, + { id: "data", type: "directory", path: "/services/minio/data/data1-1", mode: "0700" }, + { id: "net", type: "network", name: "minio" }, + { + id: "server", type: "container", name: "minio", image: pinned("minio/minio"), network: "minio", + args: ["server", "/data", "--console-address", ":9001"], + "env-file": ["/var/lib/minio/root.env"], + volumes: ["/services/minio/data/data1-1:/data"], + }, + { + id: "runtime", type: "container", name: "mesh-minio", image: pinned("mesh-runtime-minio"), + network: "minio", + volumes: [ + "/var/lib/mesh/minio/broker:/run/secrets/broker:ro", + "/var/lib/minio/grants:/var/lib/minio/grants:ro", + "/var/lib/minio/root.secret:/run/secrets/root:ro", + ], + env: { + MESH_MINIO_ENDPOINT: "http://minio:9000", + MESH_MINIO_ROOT_USER: "meshroot", + MESH_MINIO_ROOT_PASSWORD_FILE: "/run/secrets/root", + MESH_BROKER_FILE: "/run/secrets/broker", + MESH_RECEIVES: "/var/lib/minio/grants/mesh.json", + }, + }, + ], + }); + + const consumerManifest = JSON.stringify({ + module: "bucketuser", + version: "1", + // A short slug, so the derived identity `mesh_anchor_bkt` fits an S3 access key's 20 chars where + // `mesh_anchor_bucketuser` (22) would not (novox/hq ADR 0054, 04-ISSUES/010). + slug: "bkt", + requires: ["s3-bucket"], + contributes: { "s3-bucket": { name: "bucketuser" } }, + binds: { "s3-bucket": "/var/lib/bucketuser/s3.json" }, + secrets: { "s3-bucket": "/var/lib/bucketuser/s3.secret" }, + resources: [{ id: "state", type: "directory", path: "/var/lib/bucketuser", mode: "0700" }], + }); + + await must(`printf %s ${quote(minioManifest)} > /tmp/minio.json && docker cp /tmp/minio.json mesh-control:/minio.json`); + await mesh("module add /minio.json"); + await mesh(`module issue minio --node ${MACHINE}`); + await mesh(`assign ${MACHINE} minio`); + + await must(`printf %s ${quote(consumerManifest)} > /tmp/bucketuser.json && docker cp /tmp/bucketuser.json mesh-control:/bucketuser.json`); + await mesh("module add /bucketuser.json"); + await mesh(`assign ${MACHINE} bucketuser`); + + await mesh(`push ${MACHINE}`); + await settled(); + + // The mesh delivered the consumer its bound file and its unsealed secret. + let boundRaw = ""; + const untilBound = Date.now() + 60_000; + while (Date.now() < untilBound) { + const got = await on(`cat /var/lib/bucketuser/s3.json 2>/dev/null`); + if (got.ok && /"as"/.test(got.out)) { boundRaw = got.out; break; } + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(boundRaw, /"as"/, `the consumer was never told about its bucket:\n${boundRaw}`); + const bound = JSON.parse(boundRaw) as { as: string; provision: string }; + assert.equal(bound.provision, "s3-bucket"); + const accessKey = bound.as; + const secretKey = (await must(`cat /var/lib/bucketuser/s3.secret`)).trim(); + assert.ok(accessKey && secretKey, `the consumer's access key or secret was empty (as=${accessKey})`); + const bucket = bucketFor(accessKey); + + // THE PROOF: reach the bucket as the consumer, with the access key and secret the mesh delivered + // it. mc listing the consumer's own bucket means the service account, the bucket, and the secret all + // line up across the two ends. A provisioner that set a different secret answers "Access Denied". + const probe = + `mc alias set probe http://minio:9000 ${quote(accessKey)} ${quote(secretKey)} >/dev/null 2>&1 && ` + + `mc ls probe/${quote(bucket)}/`; + let out = { out: "", ok: false }; + const untilReach = Date.now() + 90_000; + while (Date.now() < untilReach) { + out = await on(`docker exec mesh-minio sh -c ${quote(probe)} 2>&1`); + if (out.ok) break; + if (/denied/i.test(out.out)) break; // fast-fail: the credential is wrong + await new Promise((r) => setTimeout(r, 3000)); + } + assert.doesNotMatch(out.out, /denied/i, + `the consumer could not reach its bucket with the mesh's secret — the two ends do not agree:\n${out.out}`); + assert.ok(out.ok, + `the consumer could not list its granted bucket ${bucket} as ${accessKey}:\n${out.out}\n---\n${(await on(`docker logs mesh-minio 2>&1 | tail -30`)).out}`); +}); diff --git a/test/integration/objectstore.test.ts b/test/integration/objectstore.test.ts new file mode 100644 index 0000000..72f3adc --- /dev/null +++ b/test/integration/objectstore.test.ts @@ -0,0 +1,305 @@ +/** + * The last step of a credential, against a real object store. + * + * The mesh generates a secret, seals it to the machine that must accept it, and discards the + * plaintext — so it cannot tell the store to start accepting it. Something on that machine reads + * what the host wrote and makes it true. This is the step where a secret either becomes a working + * key or does not. + * + * **Phase 1.1 of the work breakdown**, and the finding that shaped it: the control plane + * special-cases nothing. `provides`, `requires`, `contributes` and `grants` are name-agnostic, so + * asking for a bucket needed no change to the mesh at all — only a provider that answers. What is + * proven here is that half. + * + * **And the half a database does not have.** One PostgreSQL server holds separate databases, and + * a role that cannot reach another's is a boundary the product enforces. One object store holds + * everybody's buckets behind one endpoint, so a consumer being unable to reach another's is a + * policy somebody wrote — which means it is a thing that can be written wrongly, and therefore a + * thing to assert rather than assume. + */ + +import { test, after, before } from "node:test"; +import assert from "node:assert/strict"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; +import { incus } from "../../src/incus/client.ts"; +import { machineName } from "../../src/lifecycle/names.ts"; + +const capability = await labIsUsable(); +const provisioner = process.env["MESH_LAB_OBJECTSTORE_PROVISIONER"] ?? ""; +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !provisioner + ? "set MESH_LAB_OBJECTSTORE_PROVISIONER to a built provisioner " + + "(mesh-control: go build ./examples/objectstore-provisioner)" + : false; + +const SCENARIO = "an-object-store"; +const MACHINE = "anchor"; +const GRANTS = "/var/lib/objectstore/grants"; +const ROOT_USER = "meshroot"; +const ROOT_PASSWORD = "meshroot-super-secret"; +const ROOT_PASSWORD_FILE = "/var/lib/objectstore/root.secret"; +const ENDPOINT = "http://127.0.0.1:9000"; + +let instanceId = ""; +/** The store's image, by digest, from the registry the scenario raised. */ +let storeImage = ""; + +function shellQuote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `${command} 2>&1; echo "__exit=$?"`, + ]); + const marker = stdout.lastIndexOf("__exit="); + const status = Number(stdout.slice(marker + 7).trim()); + return { out: stdout.slice(0, marker), ok: status === 0 }; +} + +/** The same, refusing to continue past a failure nobody would otherwise see. */ +async function must(command: string): Promise { + const { out, ok } = await on(command); + if (!ok) throw new Error(`${command}\n${out}`); + return out; +} + +/** `mc` on the machine, against the store as root. */ +async function admin(args: string): Promise<{ out: string; ok: boolean }> { + return on(`mc --config-dir /tmp/root-mc ${args}`); +} + +/** + * Write what the host would have written from a declaration: the manifest of who asked, and one + * file per consumer holding its secret alone. + * + * Written here rather than by running the host, because what is under test is the step *after* + * the host — and that the host writes these exact shapes is asserted in its own suite. + */ +async function meshWrote( + consumers: { node: string; module: string; bucket: string; secret: string }[], +): Promise { + const manifest = { + contributions: consumers.length, + requirement: "s3-bucket", + generated: "by the mesh", + given: consumers.map((c) => ({ + from: c.module, + node: c.node, + secret: `${GRANTS}/${c.node}.${c.module}.secret`, + values: { bucket: c.bucket }, + })), + }; + await must(`mkdir -p ${GRANTS}`); + await must(`printf %s ${shellQuote(JSON.stringify(manifest))} > ${GRANTS}/mesh.json`); + // Every credential file rewritten from nothing, so a removed consumer's does not linger and + // make the revocation test pass for a reason that is not the one being tested. + await must(`find ${GRANTS} -name '*.secret' -delete`); + for (const c of consumers) { + await must(`printf %s ${shellQuote(c.secret)} > ${GRANTS}/${c.node}.${c.module}.secret`); + await must(`chmod 600 ${GRANTS}/${c.node}.${c.module}.secret`); + } +} + +/** The provisioner, as the module shipping the store would run it. */ +async function provision(): Promise<{ out: string; ok: boolean }> { + return on( + `GRANTS=${GRANTS} ` + + `MESH_OBJECTSTORE_URL=${ENDPOINT} ` + + `MESH_OBJECTSTORE_ROOT_USER=${ROOT_USER} ` + + `MESH_OBJECTSTORE_ROOT_PASSWORD_FILE=${ROOT_PASSWORD_FILE} ` + + `/usr/local/bin/mesh-provision-objectstore`, + ); +} + +/** + * Can this key write to and read from this bucket? + * + * As the consumer, with its own `mc` configuration directory — never the root one. A check made + * with the root alias still in scope would pass for any key at all, which is the object-store + * shape of the mistake the database suite records: two of its tests once passed without verifying + * a password, because they ran where PostgreSQL trusts the caller. + */ +async function canUse(key: string, secret: string, bucket: string): Promise<{ ok: boolean; out: string }> { + const dir = `/tmp/as-${key}`; + const { out, ok } = await on( + `rm -rf ${dir} && mc --config-dir ${dir} alias set probe ${ENDPOINT} ${shellQuote(key)} ${shellQuote(secret)} && ` + + `echo hello > /tmp/probe.txt && ` + + `mc --config-dir ${dir} cp /tmp/probe.txt probe/${bucket}/probe.txt && ` + + `mc --config-dir ${dir} cat probe/${bucket}/probe.txt`, + ); + return { ok: ok && out.includes("hello"), out }; +} + +before(async () => { + if (skip) return; + const scenario = loadScenario(`scenarios/${SCENARIO}.yml`); + const instance = await raise(scenario, {}); + instanceId = instance.instanceId; + + // From the registry the scenario raised, by digest. There is no route to a public registry from + // a documentation range, which is the point of the lab having its own. + const store = instance.images.find((r) => r.includes("minio/minio")); + const client = instance.images.find((r) => r.includes("minio/mc")); + assert.ok(store, `the scenario stocked no store image: ${instance.images.join(", ")}`); + assert.ok(client, `the scenario stocked no client image: ${instance.images.join(", ")}`); + storeImage = store; + + // The client, taken out of the vendor's own image onto the machine. The provisioner drives it, + // so it has to be here — and taking it from the stocked image is what keeps this test off any + // public network. + await must(`docker create --name mc-source ${client}`); + await must(`docker cp mc-source:/usr/bin/mc /usr/local/bin/mc && chmod 755 /usr/local/bin/mc`); + await must(`docker rm mc-source`); + + await must(`mkdir -p ${GRANTS}`); + await must(`printf %s ${shellQuote(ROOT_PASSWORD)} > ${ROOT_PASSWORD_FILE} && chmod 600 ${ROOT_PASSWORD_FILE}`); + + await must( + `docker run -d --name mesh-store ` + + `-e MINIO_ROOT_USER=${ROOT_USER} -e MINIO_ROOT_PASSWORD=${shellQuote(ROOT_PASSWORD)} ` + + `-p 127.0.0.1:9000:9000 ${storeImage} server /data`, + ); + + // Ready over the endpoint the provisioner will use, not by the container being up. A store that + // is starting answers the port and refuses every operation, which is indistinguishable from a + // wrong credential if it is not waited for. + let ready = false; + for (let i = 0; i < 90 && !ready; i++) { + ({ ok: ready } = await on( + `mc --config-dir /tmp/root-mc alias set root ${ENDPOINT} ${ROOT_USER} ${shellQuote(ROOT_PASSWORD)}`, + )); + if (!ready) await new Promise((r) => setTimeout(r, 1000)); + } + assert.ok(ready, "the store never became ready"); + + await incus([ + "file", "push", provisioner, + `${machineName(instanceId, MACHINE)}/usr/local/bin/mesh-provision-objectstore`, + "--mode", "0755", + ], 180_000); +}, { timeout: 1_200_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("a secret the mesh generated becomes a key that works", { skip, timeout: 300_000 }, async () => { + await meshWrote([ + { node: "workstation", module: "photos", bucket: "photos", secret: "first-secret-aaaaaaaa" }, + ]); + const { out, ok } = await provision(); + assert.ok(ok, out); + + const listed = await admin(`admin user list root --json`); + assert.ok(listed.out.includes("mesh_workstation_photos"), `no key was made for the consumer:\n${listed.out}`); + + const used = await canUse("mesh_workstation_photos", "first-secret-aaaaaaaa", "photos"); + assert.ok(used.ok, `the consumer cannot use the bucket the mesh gave it:\n${used.out}`); +}); + +test("a consumer cannot reach another consumer's bucket", { skip, timeout: 300_000 }, async () => { + // **The assertion this whole scenario exists for.** One store holds every bucket behind one + // endpoint, so isolation is a policy rather than a property, and a policy granting + // `arn:aws:s3:::*` would pass every other test in this file. + await meshWrote([ + { node: "workstation", module: "photos", bucket: "photos", secret: "first-secret-aaaaaaaa" }, + { node: "laptop", module: "invoices", bucket: "invoices", secret: "second-secret-bbbbbbbb" }, + ]); + const { out, ok } = await provision(); + assert.ok(ok, out); + + const own = await canUse("mesh_laptop_invoices", "second-secret-bbbbbbbb", "invoices"); + assert.ok(own.ok, `a consumer cannot use its own bucket:\n${own.out}`); + + const other = await canUse("mesh_laptop_invoices", "second-secret-bbbbbbbb", "photos"); + assert.ok(!other.ok, `a consumer reached another consumer's bucket:\n${other.out}`); +}); + +test("rotating the secret makes the new one work and the old one stop", { skip, timeout: 300_000 }, async () => { + // The failure this guards is a provisioner that only ever creates: the mesh replaces the file, + // the user exists, nothing happens, and a rotation reports success while changing nothing. + await meshWrote([ + { node: "workstation", module: "photos", bucket: "photos", secret: "rotated-secret-cccccccc" }, + ]); + const { out, ok } = await provision(); + assert.ok(ok, out); + + const now = await canUse("mesh_workstation_photos", "rotated-secret-cccccccc", "photos"); + assert.ok(now.ok, `the rotated secret does not work:\n${now.out}`); + + const before = await canUse("mesh_workstation_photos", "first-secret-aaaaaaaa", "photos"); + assert.ok(!before.ok, "the secret that was rotated away still works"); +}); + +test("a consumer that goes away loses its key", { skip, timeout: 300_000 }, async () => { + // The half usually missing. Nothing reports a key that outlives its consumer, and it keeps + // working for as long as nobody looks. + // + // **Stages its own precondition rather than inheriting one.** The first version asserted that + // `mesh_laptop` was present, having been left by an earlier test — and by then the rotation + // test had already rewritten the manifest without it, so revocation had happened for the right + // reason two tests too early. The behaviour was correct and the test was measuring residue. + await meshWrote([ + { node: "workstation", module: "photos", bucket: "photos", secret: "rotated-secret-cccccccc" }, + { node: "laptop", module: "invoices", bucket: "invoices", secret: "second-secret-bbbbbbbb" }, + ]); + const staged = await provision(); + assert.ok(staged.ok, staged.out); + const present = await admin(`admin user list root --json`); + assert.ok(present.out.includes("mesh_laptop_invoices"), `the consumer to be removed was never made:\n${present.out}`); + + await meshWrote([ + { node: "workstation", module: "photos", bucket: "photos", secret: "rotated-secret-cccccccc" }, + ]); + const { out, ok } = await provision(); + assert.ok(ok, out); + + const after = await admin(`admin user list root --json`); + assert.ok(!after.out.includes("mesh_laptop_invoices"), `a key nobody asks for survived:\n${after.out}`); + const still = await canUse("mesh_laptop_invoices", "second-secret-bbbbbbbb", "invoices"); + assert.ok(!still.ok, "a revoked key still works"); +}); + +test("a key nobody here made is left alone", { skip, timeout: 300_000 }, async () => { + // A provisioner that removed every key it did not recognise would be one nobody could safely + // run against a store that predates it (novox/hq 04-ISSUES/010). + await must( + `mc --config-dir /tmp/root-mc admin user add root somebody-elses-key somebody-elses-secret`, + ); + const { out, ok } = await provision(); + assert.ok(ok, out); + + const listed = await admin(`admin user list root --json`); + assert.ok(listed.out.includes("somebody-elses-key"), + `a key this provisioner did not make was removed:\n${listed.out}`); +}); + +test("a manifest naming a credential that was never written is refused", { skip, timeout: 300_000 }, async () => { + // Refused rather than creating a user with no secret — a login nothing can use, which nothing + // would report until something tried to connect. + await meshWrote([ + { node: "workstation", module: "photos", bucket: "photos", secret: "rotated-secret-cccccccc" }, + ]); + await must(`rm -f ${GRANTS}/workstation.photos.secret`); + + const { out, ok } = await provision(); + assert.ok(!ok, `it carried on without the credential:\n${out}`); + assert.match(out, /workstation's credential/); +}); + +test("a bucket name that would not work is refused by name", { skip, timeout: 300_000 }, async () => { + // The refusal names the consumer that asked. The store would refuse it too, as an error inside + // a provisioner log with nothing saying whose manifest caused it. + await meshWrote([ + { node: "workstation", module: "photos", bucket: "Photos_2026", secret: "rotated-secret-cccccccc" }, + ]); + const { out, ok } = await provision(); + assert.ok(!ok, `an unusable bucket name was accepted:\n${out}`); + assert.match(out, /workstation asked for a bucket named/); +}); diff --git a/test/integration/placement.test.ts b/test/integration/placement.test.ts index ff0ed53..f0f6ca1 100644 --- a/test/integration/placement.test.ts +++ b/test/integration/placement.test.ts @@ -61,7 +61,7 @@ test("the host reports the MACHINE, not the workstation that placed it", { skip, } }); -test("ADR 0031 — the host confirms the machine carries no overlay", { skip, timeout: 120_000 }, async () => { +test("ADR 0016 — the host confirms the machine carries no overlay", { skip, timeout: 120_000 }, async () => { // The lab provides the underlay and NOTHING of the overlay. Asserted elsewhere by looking // for wireguard interfaces; here the placed host reports it independently, which is a // second witness rather than the same check twice. diff --git a/test/integration/postgres-grant-end-to-end.test.ts b/test/integration/postgres-grant-end-to-end.test.ts new file mode 100644 index 0000000..240bfdb --- /dev/null +++ b/test/integration/postgres-grant-end-to-end.test.ts @@ -0,0 +1,247 @@ +/** + * The whole grant for a database, mesh-driven — novox/hq ADR 0052/0053, the postgres case. + * + * The redis bed proves the provider/consumer contract for a cache. This proves it for a database, on + * a provider whose code shells out to `psql` (so the runtime image carries it): postgres is assigned, + * a consumer that requires postgres-database is assigned, and the mesh mints one password, seals a + * copy to each end, and writes each its file. postgres's provisioner — reading only the mesh's + * contributions — creates a role and a database under the login the mesh derived, with the password + * the mesh minted. The proof is the consumer connecting to its database with the credential the mesh + * delivered it. Nothing is placed by the test. + * + * MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-module-runtime.sh postgres builds mesh-runtime-postgres:development (with psql), + * which scenarios/postgres-node.yml stocks. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "postgres-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("the mesh grants a consumer a postgres database, and the credential it delivers connects", { + skip, timeout: 900_000, +}, async () => { + // The PROVIDER: postgres in its committed shape — server and a broker-bound runtime (carrying psql) + // on the private postgres network, the runtime running the provisioner. + const postgresManifest = JSON.stringify({ + module: "postgres", + version: "1", + provides: [{ name: "postgres-database", scope: "mesh" }], + serves: { "postgres-database": {} }, + emits: ["module.postgres.database.provisioned", "module.postgres.database.deprovisioned"], + consumes: ["module.postgres.database.provisioned", "module.postgres.database.deprovisioned"], + receives: { "postgres-database": "/var/lib/postgres/grants/mesh.json" }, + grants: { "postgres-database": "/var/lib/postgres/grants" }, + "own-secrets": { superuser: "/var/lib/postgres/superuser.secret", broker: "/var/lib/mesh/postgres/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/postgres", mode: "0700" }, + { id: "state", type: "directory", path: "/var/lib/postgres", mode: "0700" }, + { id: "grants", type: "directory", path: "/var/lib/postgres/grants", mode: "0700" }, + { id: "superuser-env", type: "file", path: "/var/lib/postgres/superuser.env", mode: "0600", content: "POSTGRES_PASSWORD=${secret:superuser}\n" }, + { id: "data", type: "directory", path: "/services/postgres/db-data", mode: "0700" }, + { id: "net", type: "network", name: "postgres" }, + { + // No published port here: the substrate's own store already holds host :5432 on this + // single-node bed, and the consumer reaches postgres over the private network by name. The + // committed manifest publishes it for cross-node consumers, which is a different node. + id: "server", type: "container", name: "postgres", image: pinned("postgres"), network: "postgres", + env: { POSTGRES_USER: "postgres", POSTGRES_DB: "postgres" }, + "env-file": ["/var/lib/postgres/superuser.env"], + volumes: ["/services/postgres/db-data:/var/lib/postgresql/data"], + }, + { + id: "runtime", type: "container", name: "mesh-postgres", image: pinned("mesh-runtime-postgres"), + network: "postgres", + volumes: [ + "/var/lib/mesh/postgres/broker:/run/secrets/broker:ro", + "/var/lib/postgres/grants:/var/lib/postgres/grants:ro", + "/var/lib/postgres/superuser.secret:/run/secrets/superuser:ro", + ], + env: { + MESH_BROKER_FILE: "/run/secrets/broker", + MESH_RECEIVES: "/var/lib/postgres/grants/mesh.json", + MESH_PROVISION_POSTGRES: "postgres://postgres@postgres:5432/postgres?sslmode=disable", + MESH_PROVISION_PASSWORD_FILE: "/run/secrets/superuser", + }, + }, + ], + }); + + // The CONSUMER: a module that requires postgres-database and contributes a name so it asks. + const consumerManifest = JSON.stringify({ + module: "dbuser", + version: "1", + requires: ["postgres-database"], + contributes: { "postgres-database": { name: "dbuser" } }, + binds: { "postgres-database": "/var/lib/dbuser/db.json" }, + secrets: { "postgres-database": "/var/lib/dbuser/db.secret" }, + resources: [{ id: "state", type: "directory", path: "/var/lib/dbuser", mode: "0700" }], + }); + + await must(`printf %s ${quote(postgresManifest)} > /tmp/postgres.json && docker cp /tmp/postgres.json mesh-control:/postgres.json`); + await mesh("module add /postgres.json"); + await mesh(`module issue postgres --node ${MACHINE}`); + await mesh(`assign ${MACHINE} postgres`); + + await must(`printf %s ${quote(consumerManifest)} > /tmp/dbuser.json && docker cp /tmp/dbuser.json mesh-control:/dbuser.json`); + await mesh("module add /dbuser.json"); + await mesh(`assign ${MACHINE} dbuser`); + + await mesh(`push ${MACHINE}`); + await settled(); + + // The mesh delivered the consumer its bound file and its unsealed password. + let boundRaw = ""; + const untilBound = Date.now() + 60_000; + while (Date.now() < untilBound) { + const got = await on(`cat /var/lib/dbuser/db.json 2>/dev/null`); + if (got.ok && /"as"/.test(got.out)) { boundRaw = got.out; break; } + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(boundRaw, /"as"/, `the consumer was never told about its database:\n${boundRaw}`); + const bound = JSON.parse(boundRaw) as { as: string; provision: string }; + assert.equal(bound.provision, "postgres-database"); + const as = bound.as; + const password = (await must(`cat /var/lib/dbuser/db.secret`)).trim(); + assert.ok(as && password, `the consumer's login or password was empty (as=${as})`); + + // THE PROOF: connect to postgres as the consumer, with the login and password the mesh delivered + // it, to the database postgres's provisioner created — a real password-checked TCP connection (the + // runtime carries psql). A `1` back means the role, the database, and the password all line up + // across the two ends. A provisioner that set a different password answers "authentication failed". + const conn = `postgresql://${as}:${encodeURIComponent(password)}@postgres:5432/${as}?sslmode=disable`; + let out = { out: "", ok: false }; + const untilConn = Date.now() + 90_000; + while (Date.now() < untilConn) { + out = await on(`docker exec mesh-postgres psql ${quote(conn)} -tAc 'select 1' 2>&1`); + if (out.ok && /^1$/m.test(out.out)) break; + if (/authentication failed/i.test(out.out)) break; // fast-fail: the credential is wrong + await new Promise((r) => setTimeout(r, 3000)); + } + assert.doesNotMatch(out.out, /authentication failed/i, + `the consumer could not authenticate with the mesh's password — the two ends do not agree:\n${out.out}`); + assert.match(out.out, /^1$/m, + `the consumer could not connect to its granted database as ${as}:\n${out.out}\n---\n${(await on(`docker logs mesh-postgres 2>&1 | tail -30`)).out}`); +}); diff --git a/test/integration/provider-on-backend-network.test.ts b/test/integration/provider-on-backend-network.test.ts new file mode 100644 index 0000000..11fb8a8 --- /dev/null +++ b/test/integration/provider-on-backend-network.test.ts @@ -0,0 +1,232 @@ +/** + * A provider's runtime, on the backend's own private network, still binds the mesh broker. + * + * The committed provider manifests keep the backend server on a private container network and put + * the tool/provisioner runtime on it too, so the runtime reaches the backend by name (redis:6379). + * The runtime also has to bind the mesh broker for its scoped account — reached not on that private + * network but via NAT to the broker's node address. This proves that shape: redis's runtime, on the + * `redis` network, comes up, provisions a consumer with the mesh's credential, and its scoped account + * is on the broker — none of which happens if it could not reach the broker from the bridge. + * + * This mirrors the redis committed manifest exactly (server on `redis` with a published port, runtime + * on `redis`), where provider-uses-mesh-credential used host networking. If this is green, the + * private-network shape is the one to roll out to every provider. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "redis-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("redis's runtime, on the backend's private network, binds the broker and provisions with the mesh's credential", { + skip, timeout: 900_000, +}, async () => { + // Exactly the committed redis shape: a private `redis` network, the server on it with a published + // port, and the runtime on it too — reaching redis by name and the broker by NAT. + const manifest = JSON.stringify({ + module: "redis", + version: "1", + provides: [{ name: "redis-cache", scope: "mesh" }], + serves: { "redis-cache": {} }, + emits: ["module.redis.cache.provisioned", "module.redis.cache.deprovisioned"], + consumes: ["module.redis.cache.provisioned", "module.redis.cache.deprovisioned"], + receives: { "redis-cache": "/var/lib/redis-module/grants/mesh.json" }, + grants: { "redis-cache": "/var/lib/redis-module/grants" }, + "own-secrets": { default: "/var/lib/redis-module/default.secret", broker: "/var/lib/mesh/redis/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/redis", mode: "0700" }, + { id: "state", type: "directory", path: "/var/lib/redis-module", mode: "0700" }, + { id: "grants-dir", type: "directory", path: "/var/lib/redis-module/grants", mode: "0700" }, + { id: "data", type: "directory", path: "/services/redis/data", mode: "0700", owner: "999:999" }, + { + id: "server-conf", type: "file", path: "/var/lib/redis-module/redis.conf", mode: "0644", + content: "requirepass ${secret:default}\nappendonly no\ndir /data\n", + }, + { id: "net", type: "network", name: "redis" }, + { + id: "server", type: "container", name: "redis", image: pinned("redis"), network: "redis", + ports: ["6379"], + volumes: [ + "/services/redis/data:/data", + "/var/lib/redis-module/redis.conf:/etc/redis/redis.conf:ro", + ], + args: ["/etc/redis/redis.conf"], + }, + { + id: "runtime", type: "container", name: "mesh-redis", image: pinned("mesh-runtime-redis"), + network: "redis", + volumes: [ + "/var/lib/mesh/redis/broker:/run/secrets/broker:ro", + "/var/lib/redis-module/grants:/var/lib/redis-module/grants:ro", + "/var/lib/redis-module/default.secret:/run/secrets/default:ro", + ], + env: { + MESH_BROKER_FILE: "/run/secrets/broker", + MESH_RECEIVES: "/var/lib/redis-module/grants/mesh.json", + MESH_PROVISION_REDIS: "redis:6379", + MESH_PROVISION_PASSWORD_FILE: "/run/secrets/default", + }, + }, + ], + }); + await must(`printf %s ${quote(manifest)} > /tmp/redis.json && docker cp /tmp/redis.json mesh-control:/redis.json`); + await mesh("module add /redis.json"); + await mesh(`module issue redis --node ${MACHINE}`); + await mesh(`assign ${MACHINE} redis`); + await mesh(`push ${MACHINE}`); + await settled(); + + const running = await must(`docker ps --format '{{.Names}}'`); + assert.match(running, /mesh-redis/, `redis's runtime is not running:\n${(await on(`docker logs mesh-redis 2>&1 | tail -30`)).out}`); + + // It bound the broker from the private network: its scoped account is on the broker. If NAT to the + // broker had failed, the runtime would have crashed and never authenticated. + const users = await must(`docker exec mesh-broker lavinmqctl list_users 2>&1`); + assert.match(users, /anchor-redis/, + `the runtime's scoped account is not on the broker — it did not reach the broker from the bridge:\n` + + `${(await on(`docker logs mesh-redis 2>&1 | tail -30`)).out}`); + + // The mesh delivers the contribution and the unsealed password; the provisioner creates the login. + const password = "mesh-minted-bridge-7c1"; + await must(`printf %s ${quote(password)} > /var/lib/redis-module/grants/app.secret`); + const contributions = JSON.stringify({ + contributions: 1, requirement: "redis-cache", generated: "by the mesh", + given: [{ from: "app", node: "app-node", at: "192.0.2.20:6379", as: "app-one", secret: "/var/lib/redis-module/grants/app.secret", values: {} }], + }); + await must(`printf %s ${quote(contributions)} > /var/lib/redis-module/grants/mesh.json`); + + const adminPw = await must(`cat /var/lib/redis-module/default.secret`); + let acl = ""; + const until = Date.now() + 60_000; + while (Date.now() < until) { + acl = (await on(`docker exec redis redis-cli -a ${quote(adminPw)} --no-auth-warning ACL LIST 2>/dev/null`)).out; + if (/app-one/.test(acl)) break; + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(acl, /app-one/, `the provisioner never created the login:\n${(await on(`docker logs mesh-redis 2>&1 | tail -30`)).out}\n---\n${acl}`); + + const authed = await on(`docker exec redis redis-cli --user app-one --pass ${quote(password)} --no-auth-warning PING 2>&1`); + assert.doesNotMatch(authed.out, /WRONGPASS/, `the consumer could not authenticate with the mesh's password:\n${authed.out}`); + assert.match(authed.out, /PONG/, `expected PONG:\n${authed.out}`); +}); diff --git a/test/integration/provider-uses-mesh-credential.test.ts b/test/integration/provider-uses-mesh-credential.test.ts new file mode 100644 index 0000000..8724872 --- /dev/null +++ b/test/integration/provider-uses-mesh-credential.test.ts @@ -0,0 +1,242 @@ +/** + * A provider creates the resource with the credential the mesh minted — novox/hq ADR 0053. + * + * The old provisioner generated its own password, sealed it with a key nothing delivered, and + * handed it back. This proves the corrected contract: redis's provisioner reads the mesh's + * contributions file and, for each consumer, the password the mesh minted and the host unsealed, and + * creates the ACL user under the login the mesh derived, with that exact password. No $MESH_SEAL_KEY + * is set anywhere. The proof is authentication: a client logging in as that consumer with the mesh's + * password gets PONG — where a provisioner that invented its own password would answer WRONGPASS. + * + * A hand-written contributions file and secret stand in for the control plane here (a full grant + * from a second module is a heavier bed); their SHAPE is exactly what mesh-control writes — a + * `receives` doc with `as`/`secret`, and the secret file the host leaves after unsealing. + * + * MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-module-runtime.sh redis builds mesh-runtime-redis:development, which + * scenarios/redis-node.yml stocks. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "redis-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("redis creates a consumer's login with the password the mesh minted, sealing nothing", { + skip, timeout: 900_000, +}, async () => { + // redis as a provider: the server, and a broker-bound runtime that serves its tools AND runs its + // provisioner. The provisioner is pointed at the contributions file the mesh would write + // (MESH_RECEIVES). There is NO MESH_SEAL_KEY — the whole point of ADR 0053 is that a provider + // needs none. + const manifest = JSON.stringify({ + module: "redis", + version: "1", + emits: ["module.redis.cache.provisioned", "module.redis.cache.deprovisioned"], + consumes: ["module.redis.cache.provisioned", "module.redis.cache.deprovisioned"], + "own-secrets": { default: "/var/lib/redis-module/default.secret", broker: "/var/lib/mesh/redis/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/redis", mode: "0700" }, + { id: "state", type: "directory", path: "/var/lib/redis-module", mode: "0700" }, + { id: "grants-dir", type: "directory", path: "/var/lib/redis-module/grants", mode: "0700" }, + { id: "data", type: "directory", path: "/services/redis/data", mode: "0700", owner: "999:999" }, + { + id: "server-conf", type: "file", path: "/var/lib/redis-module/redis.conf", mode: "0644", + content: "requirepass ${secret:default}\nappendonly no\ndir /data\n", + }, + { + id: "server", type: "container", name: "redis", image: pinned("redis"), network: "host", + volumes: [ + "/services/redis/data:/data", + "/var/lib/redis-module/redis.conf:/etc/redis/redis.conf:ro", + ], + args: ["/etc/redis/redis.conf"], + }, + { + id: "runtime", type: "container", name: "mesh-redis", image: pinned("mesh-runtime-redis"), + network: "host", + volumes: [ + "/var/lib/mesh/redis/broker:/run/secrets/broker:ro", + "/var/lib/redis-module/grants:/var/lib/redis-module/grants", + "/var/lib/redis-module/default.secret:/run/secrets/default:ro", + ], + env: { + MESH_BROKER_FILE: "/run/secrets/broker", + MESH_RECEIVES: "/var/lib/redis-module/grants/redis-cache.json", + MESH_PROVISION_REDIS: "127.0.0.1:6379", + MESH_PROVISION_PASSWORD_FILE: "/run/secrets/default", + }, + }, + ], + }); + await must(`printf %s ${quote(manifest)} > /tmp/redis.json && docker cp /tmp/redis.json mesh-control:/redis.json`); + await mesh("module add /redis.json"); + await mesh(`module issue redis --node ${MACHINE}`); + await mesh(`assign ${MACHINE} redis`); + await mesh(`push ${MACHINE}`); + await settled(); + + const running = await must(`docker ps --format '{{.Names}}'`); + assert.match(running, /mesh-redis/, `redis's runtime is not running:\n${(await on(`docker logs mesh-redis 2>&1 | tail -20`)).out}`); + + // What the mesh delivers to the provider: a contributions file naming the consumer's login and + // where its password is, and the password itself as the file the host leaves after unsealing. + const password = "mesh-minted-9f3c2a"; + await must(`printf %s ${quote(password)} > /var/lib/redis-module/grants/app.secret`); + const contributions = JSON.stringify({ + contributions: 1, + requirement: "redis-cache", + generated: "by the mesh — do not edit", + given: [ + { from: "app", node: "app-node", at: "192.0.2.20:6379", as: "app-one", secret: "/var/lib/redis-module/grants/app.secret", values: {} }, + ], + }); + await must(`printf %s ${quote(contributions)} > /var/lib/redis-module/grants/redis-cache.json`); + + // Within a reconcile tick the provisioner creates the ACL user. It exists on the server. + let acl = ""; + const until = Date.now() + 60_000; + while (Date.now() < until) { + acl = (await on(`docker exec redis redis-cli -a ${quote(await must(`cat /var/lib/redis-module/default.secret`))} --no-auth-warning ACL LIST 2>/dev/null`)).out; + if (/app-one/.test(acl)) break; + await new Promise((r) => setTimeout(r, 3000)); + } + assert.match(acl, /app-one/, `the provisioner never created the consumer's login:\n${(await on(`docker logs mesh-redis 2>&1 | tail -30`)).out}\n---\n${acl}`); + + // The proof: authenticate as that consumer with the password the MESH minted. PONG means the + // provisioner created the login with exactly that password. A provisioner that invented its own + // (the old behaviour) would answer WRONGPASS here. + const authed = await on(`docker exec redis redis-cli --user app-one --pass ${quote(password)} --no-auth-warning PING 2>&1`); + assert.doesNotMatch(authed.out, /WRONGPASS/, + `the consumer could not authenticate with the mesh's password — the provider used a different one:\n${authed.out}`); + assert.match(authed.out, /PONG/, `expected PONG authenticating as the consumer:\n${authed.out}`); + + // And it needed no seal key: the runtime came up and provisioned with MESH_SEAL_KEY set nowhere. + const env = await must(`docker inspect mesh-redis --format '{{json .Config.Env}}'`); + assert.doesNotMatch(env, /MESH_SEAL_KEY/, `a seal key was set after all — ADR 0053 is not what ran:\n${env}`); + + // The provisioner emitted its lifecycle event under the bound account, and no emit was refused. + const log = (await on(`docker logs mesh-redis 2>&1`)).out; + assert.doesNotMatch(log, /emit .*failed/, `the provisioned event was refused:\n${log}`); +}); diff --git a/test/integration/provisioner.test.ts b/test/integration/provisioner.test.ts new file mode 100644 index 0000000..715fd42 --- /dev/null +++ b/test/integration/provisioner.test.ts @@ -0,0 +1,258 @@ +/** + * The last step of a credential, against a real database. + * + * The mesh generates a password, seals it to the machine that must accept it, and discards the + * plaintext — so it cannot tell PostgreSQL to start accepting it. Something on that machine reads + * what the host wrote and makes it true. Everything up to that point is proven elsewhere; this is + * the step where a password either becomes a login or does not. + * + * Against a real PostgreSQL because there is no version of this worth asserting against a fake: + * what is under test is whether `create role ... password` and a connection agree, which is + * exactly what a fake would be told to agree about (novox/hq ADR 0017). + */ + +import { test, after, before } from "node:test"; +import assert from "node:assert/strict"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; +import { incus } from "../../src/incus/client.ts"; +import { machineName } from "../../src/lifecycle/names.ts"; + +const capability = await labIsUsable(); +const provisioner = process.env["MESH_LAB_PROVISIONER"] ?? ""; +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !provisioner + ? "set MESH_LAB_PROVISIONER to a built provisioner (mesh-control: go build ./examples/postgres-provisioner)" + : false; + +const SCENARIO = "a-provider"; +const MACHINE = "anchor"; +const GRANTS = "/var/lib/postgres/grants"; +const SUPER = "postgres://postgres:super@127.0.0.1:5432/postgres?sslmode=disable"; + +let instanceId = ""; +/** The postgres image, by digest, from the registry the scenario raised. */ +let image = ""; + +function shellQuote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +/** Run something on the machine and return what it said, with its exit status. */ +async function on(command: string): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `${command} 2>&1; echo "__exit=$?"`, + ]); + const marker = stdout.lastIndexOf("__exit="); + const status = Number(stdout.slice(marker + 7).trim()); + return { out: stdout.slice(0, marker), ok: status === 0 }; +} + +/** The same, refusing to continue past a failure nobody would otherwise see. */ +async function must(command: string): Promise { + const { out, ok } = await on(command); + if (!ok) throw new Error(`${command}\n${out}`); + return out; +} + +/** psql as the superuser, inside the database container. */ +async function sql(query: string): Promise { + return (await must(`docker exec mesh-db psql -U postgres -qAt -c ${shellQuote(query)}`)).trim(); +} + +/** + * Write what the host would have written from a declaration: the manifest of who asked, and one + * file per consumer holding its password alone. + * + * Written here rather than by running the host, because what is under test is the step *after* + * the host — and that the host writes these exact shapes is asserted in its own suite. + */ +async function meshWrote( + consumers: { node: string; module: string; name: string; password: string }[], +): Promise { + const manifest = { + contributions: 1, + requirement: "postgres-database", + generated: "by the mesh", + given: consumers.map((c) => ({ + from: c.module, + node: c.node, + secret: `${GRANTS}/${c.node}.${c.module}.secret`, + values: { name: c.name }, + })), + }; + await must(`mkdir -p ${GRANTS}`); + await must(`printf %s ${shellQuote(JSON.stringify(manifest))} > ${GRANTS}/mesh.json`); + // Every credential file rewritten from nothing, so a removed consumer's does not linger and + // make the revocation test pass for a reason that is not the one being tested. + await must(`find ${GRANTS} -name '*.secret' -delete`); + for (const c of consumers) { + await must(`printf %s ${shellQuote(c.password)} > ${GRANTS}/${c.node}.${c.module}.secret`); + await must(`chmod 600 ${GRANTS}/${c.node}.${c.module}.secret`); + } +} + +/** The provisioner, as the module shipping PostgreSQL would run it. */ +async function provision(): Promise<{ out: string; ok: boolean }> { + return on( + `GRANTS=${GRANTS} MESH_PROVISION_POSTGRES=${shellQuote(SUPER)} /usr/local/bin/mesh-provision-postgres`, + ); +} + +/** + * Can this role log in with this password? + * + * Over the bridge, from a container of its own. `--network container:mesh-db` would share the + * database's namespace and put us back on its loopback, which is the very thing being avoided. + * + * The address comes from `.NetworkSettings.Networks.bridge.IPAddress` rather than the top-level + * `.NetworkSettings.IPAddress`, which docker 29 no longer populates — it templates to empty, psql + * silently falls back to a unix socket that is not there, and every login looks impossible. + * + * From a separate container, reaching the database over the bridge — **not** from inside it over + * loopback. PostgreSQL's default `pg_hba.conf` trusts `127.0.0.1`, so a check made from inside + * the container authenticates nothing and returns true for any password at all. Which is what the + * first version of this did: two tests passed without ever verifying a password, and only the + * rotation test noticed, by asserting that an old password had *stopped* working. + */ +async function canLogIn(role: string, password: string, database: string): Promise { + return (await tryLogIn(role, password, database)).ok; +} + +/** The same, keeping what the database said — so a failure says why rather than only that. */ +async function tryLogIn( + role: string, + password: string, + database: string, +): Promise<{ ok: boolean; out: string }> { + const { out } = await on( + `docker run --rm -e PGPASSWORD=${shellQuote(password)} ${image} ` + + `psql -h "$(docker inspect -f '{{.NetworkSettings.Networks.bridge.IPAddress}}' mesh-db)" ` + + `-U ${role} -d ${database} -qAt -c 'select 1'`, + ); + return { ok: out.trim() === "1", out }; +} + +before(async () => { + if (skip) return; + const scenario = loadScenario(`scenarios/${SCENARIO}.yml`); + const instance = await raise(scenario, {}); + instanceId = instance.instanceId; + + // From the registry the scenario raised, by digest. There is no route to a public registry from + // a documentation range, which is the point of the lab having its own. + const stocked = instance.images.find((r) => r.includes("postgres")); + assert.ok(stocked, `the scenario stocked no postgres image: ${instance.images.join(", ")}`); + image = stocked; + + await must( + `docker run -d --name mesh-db -e POSTGRES_PASSWORD=super ` + + `-p 127.0.0.1:5432:5432 ${image}`, + ); + let ready = false; + for (let i = 0; i < 90 && !ready; i++) { + ({ ok: ready } = await on(`docker exec mesh-db pg_isready -U postgres`)); + if (!ready) await new Promise((r) => setTimeout(r, 1000)); + } + assert.ok(ready, "the database never became ready"); + + await incus([ + "file", "push", provisioner, + `${machineName(instanceId, MACHINE)}/usr/local/bin/mesh-provision-postgres`, + "--mode", "0755", + ], 180_000); +}, { timeout: 1_200_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("a password the mesh generated becomes a login that works", { skip, timeout: 300_000 }, async () => { + await meshWrote([ + { node: "workstation", module: "meshboard", name: "meshboard", password: "first-password-aaa" }, + ]); + const { out, ok } = await provision(); + assert.ok(ok, out); + + assert.equal(await sql(`select rolcanlogin from pg_roles where rolname = 'mesh_workstation_meshboard'`), "t"); + assert.equal(await sql(`select 1 from pg_database where datname = 'meshboard'`), "1"); + const attempt = await tryLogIn("mesh_workstation_meshboard", "first-password-aaa", "meshboard"); + assert.ok(attempt.ok, `the consumer cannot log in with the password the mesh gave it:\n${attempt.out}`); +}); + +test("running it again reaches the same state and says nothing", { skip, timeout: 300_000 }, async () => { + // It runs after every declaration and is never told what changed, so arriving at an already + // correct state is the ordinary case rather than an edge one. + const { out, ok } = await provision(); + assert.ok(ok, out); + assert.equal(out.trim(), "", `it did work on a second run: ${out}`); + assert.ok(await canLogIn("mesh_workstation_meshboard", "first-password-aaa", "meshboard")); +}); + +test("rotating the password makes the new one work and the old one stop", { skip, timeout: 300_000 }, async () => { + // The failure this guards is a provisioner that only ever creates: the mesh replaces the file, + // the role exists, nothing happens, and a rotation reports success while changing nothing. + await meshWrote([ + { node: "workstation", module: "meshboard", name: "meshboard", password: "second-password-bbb" }, + ]); + const { out, ok } = await provision(); + assert.ok(ok, out); + + assert.ok( + await canLogIn("mesh_workstation_meshboard", "second-password-bbb", "meshboard"), + "the rotated password does not work", + ); + assert.equal( + await canLogIn("mesh_workstation_meshboard", "first-password-aaa", "meshboard"), + false, + "the old password still works, so the rotation changed nothing", + ); +}); + +test("a consumer that goes away loses its login", { skip, timeout: 300_000 }, async () => { + // The half usually missing. A consumer removed from the mesh otherwise keeps a working login + // for ever and nothing says so — the same rule the host follows about removing what it declared + // and no longer declares. + await meshWrote([]); + const { out, ok } = await provision(); + assert.ok(ok, out); + assert.match(out, /revoked mesh_workstation_meshboard/); + + assert.equal(await sql(`select rolcanlogin from pg_roles where rolname = 'mesh_workstation_meshboard'`), "f"); + assert.equal( + await canLogIn("mesh_workstation_meshboard", "second-password-bbb", "meshboard"), + false, + "a consumer nobody asks for any more can still log in", + ); +}); + +test("a role nobody here made is left alone", { skip, timeout: 300_000 }, async () => { + // A provisioner that removed every role it did not recognise could not safely be run on a + // database that predates it — which is every database anybody would want to adopt. + await sql(`create role someone_elses with login password 'theirs'`); + await sql(`create database theirs owner someone_elses`); + await meshWrote([]); + const { ok } = await provision(); + assert.ok(ok); + assert.equal(await sql(`select rolcanlogin from pg_roles where rolname = 'someone_elses'`), "t"); + assert.ok(await canLogIn("someone_elses", "theirs", "theirs")); +}); + +test("a manifest naming a credential that was never written is refused", { skip, timeout: 300_000 }, async () => { + // Rather than creating a role with no password — a login nothing can use, which nothing would + // report until something tried to connect. + await meshWrote([]); + await must( + `printf %s '{"contributions":1,"requirement":"postgres-database","given":[` + + `{"from":"meshboard","node":"ghost","secret":"${GRANTS}/ghost.meshboard.secret","values":{"name":"ghost"}}` + + `]}' > ${GRANTS}/mesh.json`, + ); + const { out, ok } = await provision(); + assert.equal(ok, false, "it carried on past a missing credential"); + assert.match(out, /should be at .*ghost\.meshboard\.secret/); + assert.equal(await sql(`select count(*) from pg_roles where rolname = 'mesh_ghost_meshboard'`), "0"); +}); diff --git a/test/integration/runtime-restart-on-config.test.ts b/test/integration/runtime-restart-on-config.test.ts new file mode 100644 index 0000000..fc6a155 --- /dev/null +++ b/test/integration/runtime-restart-on-config.test.ts @@ -0,0 +1,209 @@ +/** + * A running tool runtime picks up a settings change — novox/hq 04-ISSUES/009, and its fix. + * + * A runtime reads its settings-merged config file once, at start. Change the settings on an + * already-running runtime and, without this, the container keeps the value it read: its spec did + * not move (a mounted file's content is not part of it) so the host left it alone, and every check + * passed while the mesh did the old thing. The fix gives a container `restart-on`, the same field a + * service has: the runtime names its config resource, and the host recreates the container when that + * resource changed this pass. + * + * This assigns grafana configured by settings, then changes the token and pushes again, and asserts + * the container was replaced (a new container id) and the config on disk carries the new value. + * It builds the host from source (no --no-build), because the behaviour under test is the host's. + * + * MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock + * scripts/build-module-runtime.sh grafana builds mesh-runtime-grafana:development, which + * scenarios/grafana-node.yml stocks. + */ + +import { test, before, after } from "node:test"; +import assert from "node:assert/strict"; +import { existsSync, readFileSync } from "node:fs"; +import { loadScenario } from "../../src/declaration/parse.ts"; +import { raise } from "../../src/lifecycle/raise.ts"; +import { destroy, exec } from "../../src/lifecycle/operate.ts"; +import { hostBinaryPath, HOST_PATH } from "../../src/lifecycle/place.ts"; +import { labIsUsable, destroyAll } from "./harness.ts"; + +const capability = await labIsUsable(); +const binary = hostBinaryPath(); +const bundle = process.env["MESH_LAB_BUNDLE"] ?? ""; + +const skip = !capability.usable + ? `lab not usable: ${capability.why}` + : !binary || !existsSync(binary) + ? "MESH_LAB_HOST_BINARY is not set to a built mesh-host" + : !bundle || !existsSync(bundle) + ? "MESH_LAB_BUNDLE is not set to a substrate bundle (mesh-host examples/)" + : false; + +const SCENARIO = "grafana-node"; +const MACHINE = "anchor"; + +let instanceId = ""; +let stocked: string[] = []; + +function quote(s: string): string { + return `'${s.replaceAll("'", `'\\''`)}'`; +} + +async function on(command: string, timeoutMs?: number): Promise<{ out: string; ok: boolean }> { + const { stdout } = await exec(instanceId, MACHINE, [ + "sh", "-c", `exec 2>&1\n${command}\necho "__exit=$?"`, + ], timeoutMs); + const marker = stdout.lastIndexOf("__exit="); + if (marker < 0) return { out: stdout, ok: false }; + return { out: stdout.slice(0, marker), ok: stdout.slice(marker + 7).trim() === "0" }; +} + +async function must(command: string, timeoutMs?: number): Promise { + const { out, ok } = await on(command, timeoutMs); + if (!ok) throw new Error(`${MACHINE}: ${command}\n${out}`); + return out; +} + +async function mesh(command: string, timeoutMs?: number): Promise { + return must(`docker exec mesh-control /mesh-control ${command}`, timeoutMs); +} + +function pinned(repository: string): string { + const found = stocked.find((r) => r.slice(r.indexOf("/") + 1, r.indexOf("@")) === repository); + assert.ok(found, `the scenario stocks no ${repository}; it serves ${stocked.join(", ")}`); + return found; +} + +function bundleFor(images: string[]): string { + let text = readFileSync(bundle, "utf8"); + for (const ref of images) { + const repository = ref.slice(ref.indexOf("/") + 1, ref.indexOf("@")); + const escaped = repository.replaceAll("/", "\\/").replaceAll(".", "\\."); + text = text.replaceAll(new RegExp(`[A-Za-z0-9_.:-]+\\/${escaped}@sha256:[0-9a-f]+`, "g"), ref); + } + return text; +} + +function tokenFrom(said: string): string { + const found = said.split("\n").map((l) => l.trim()).find((l) => l.length > 100 && !l.includes(" ")); + assert.ok(found, `no token in:\n${said}`); + return found; +} + +async function settled(withinMs = 480_000): Promise { + const until = Date.now() + withinMs; + let last = ""; + while (Date.now() < until) { + const asked = await on(`docker exec mesh-control /mesh-control status --json`); + if (asked.ok) { + try { + const state = JSON.parse(asked.out) as { + wrong: { node: string; outcome: string }[]; + waiting: { node: string }[]; + reported: { node: string; outcome: string; current: boolean }[]; + }; + const bad = state.wrong.find((w) => w.node === MACHINE); + if (bad) throw new Error(`${MACHINE} did not apply what it was sent: ${bad.outcome}\n${asked.out}`); + const word = state.reported.find((r) => r.node === MACHINE); + if (!state.waiting.some((w) => w.node === MACHINE) && word?.outcome === "applied" && word.current) return; + last = asked.out; + } catch (err) { + if (err instanceof Error && err.message.includes("did not apply")) throw err; + last = asked.out; + } + } + await new Promise((r) => setTimeout(r, 5000)); + } + throw new Error(`${MACHINE} never caught up within ${Math.round(withinMs / 1000)}s. Last:\n${last}`); +} + +async function setToken(token: string): Promise { + const settings = JSON.stringify({ url: "http://127.0.0.1:3000", token }); + await must(`printf %s ${quote(settings)} > /tmp/s.json && docker cp /tmp/s.json mesh-control:/s.json`); + await mesh(`settings set grafana /s.json --node ${MACHINE}`); +} + +async function containerId(): Promise { + return (await must(`docker inspect mesh-grafana --format '{{.Id}}'`)).trim(); +} + +before(async () => { + if (skip) return; + + const raised = await raise(loadScenario(`scenarios/${SCENARIO}.yml`), { + onProgress: (m) => console.log(`raise: ${m}`), + }); + instanceId = raised.instanceId; + stocked = raised.images; + + await must(`cat > /tmp/substrate.lock <<'MESHBUNDLE'\n${bundleFor(raised.images)}\nMESHBUNDLE`); + await must(`${HOST_PATH} apply /tmp/substrate.lock`, 600_000); + const up = await must(`docker ps --format '{{.Names}}'`); + for (const c of ["mesh-store", "mesh-broker", "mesh-control"]) { + assert.match(up, new RegExp(c), `the substrate did not raise ${c}:\n${up}`); + } + + await mesh(`node add ${MACHINE}`); + const token = tokenFrom(await mesh(`token issue --node ${MACHINE}`)); + await must(`${HOST_PATH} enrol --token ${quote(token)}`); + await must(`nohup ${HOST_PATH} run > /var/log/mesh-host.log 2>&1 & sleep 3`); +}, { timeout: 1_800_000 }); + +after(async () => { + if (instanceId) await destroy(instanceId); + await destroyAll(`${SCENARIO}-`); +}, { timeout: 600_000 }); + +test("a running runtime is recreated when its settings change, and reads the new value", { + skip, timeout: 900_000, +}, async () => { + const manifest = JSON.stringify({ + module: "grafana", + version: "1", + emits: ["module.grafana.alert.firing"], + "own-secrets": { broker: "/var/lib/mesh/grafana/broker" }, + resources: [ + { id: "mesh-state", type: "directory", path: "/var/lib/mesh/grafana", mode: "0700" }, + { id: "runtime-config", type: "file", path: "/var/lib/mesh/grafana/config.json", mode: "0600", content: "{}\n", merge: "json" }, + { + id: "runtime", type: "container", name: "mesh-grafana", image: pinned("mesh-runtime-grafana"), + network: "host", + "restart-on": ["runtime-config"], + volumes: [ + "/var/lib/mesh/grafana/broker:/run/secrets/broker:ro", + "/var/lib/mesh/grafana/config.json:/run/config/config.json:ro", + ], + env: { MESH_BROKER_FILE: "/run/secrets/broker", MESH_GRAFANA_CONFIG_FILE: "/run/config/config.json" }, + }, + ], + }); + await must(`printf %s ${quote(manifest)} > /tmp/grafana.json && docker cp /tmp/grafana.json mesh-control:/grafana.json`); + await mesh("module add /grafana.json"); + + await setToken("token-alpha"); + await mesh(`module issue grafana --node ${MACHINE}`); + await mesh(`assign ${MACHINE} grafana`); + await mesh(`push ${MACHINE}`); + await settled(); + + const before = await containerId(); + const configBefore = await must(`cat /var/lib/mesh/grafana/config.json`); + assert.match(configBefore, /token-alpha/, `first settings not rendered:\n${configBefore}`); + + // Change the setting on the already-running runtime, and push. Nothing about the container's + // spec changes — only the content of the file it mounts. + await setToken("token-bravo"); + await mesh(`push ${MACHINE}`); + await settled(); + + const configAfter = await must(`cat /var/lib/mesh/grafana/config.json`); + assert.match(configAfter, /token-bravo/, `the settings change did not re-render the config:\n${configAfter}`); + + const after = await containerId(); + assert.notEqual(after, before, + `the runtime was NOT recreated on a config change (issue 009 not fixed): id stayed ${before}\n` + + `host log:\n${(await on(`grep -i grafana /var/log/mesh-host.log | tail -10`)).out}`); + + // And it is running on the new container, so the process re-read the new config. + const running = await must(`docker ps --format '{{.Names}}'`); + assert.match(running, /mesh-grafana/, "the recreated runtime is not running"); +}); diff --git a/test/integration/underlay.test.ts b/test/integration/underlay.test.ts index c9f3f8d..fbe5b03 100644 --- a/test/integration/underlay.test.ts +++ b/test/integration/underlay.test.ts @@ -1,6 +1,6 @@ /** * Each test names the decision it defends. A decision with no test is one that will quietly - * stop being true (novox/hq ADR 0034). + * stop being true (novox/hq ADR 0017). */ import { test, before, after } from "node:test"; @@ -30,7 +30,7 @@ after(async () => { if (instanceId) await destroy(instanceId); }); -test("ADR 0031 — the lab provides the underlay and NOTHING of the overlay", { skip, timeout: 120_000 }, async () => { +test("ADR 0016 — the lab provides the underlay and NOTHING of the overlay", { skip, timeout: 120_000 }, async () => { // A scenario that pre-built peering would certify its own work. Whatever the mesh is // responsible for must be absent from a freshly raised machine. const { stdout } = await exec(instanceId, "home-server", [ @@ -43,7 +43,7 @@ test("ADR 0031 — the lab provides the underlay and NOTHING of the overlay", { assert.deepEqual(counts, [0, 0, 0], "a raised machine carries no overlay, no mesh config"); }); -test("ADR 0031 — the declared address IS what the machine holds", { skip }, async () => { +test("ADR 0016 — the declared address IS what the machine holds", { skip }, async () => { const { stdout } = await exec(instanceId, "home-server", ["ip", "-o", "-4", "addr", "show"]); assert.match(stdout, /192\.168\.1\.135\/24/); }); @@ -57,7 +57,7 @@ test("design — raise waits for USABLE, not for the call to return", { skip, ti } }); -test("ADR 0033 — a router is scenery: containers, while machines are virtual machines", { skip, timeout: 120_000 }, async () => { +test("ADR 0016 — a router is scenery: containers, while machines are virtual machines", { skip, timeout: 120_000 }, async () => { const json = (await incusOk(["list", "--format", "json"], 30_000)) ?? "[]"; const all = JSON.parse(json) as { name?: string; type?: string; config?: Record }[]; const mine = all.filter((i) => i.config?.["user.mesh-lab.instance"] === instanceId); @@ -114,7 +114,7 @@ test("design — restore leaves the scenario USABLE, not merely running", { skip assert.equal(stdout.trim(), "alive"); }); -test("ADR 0032 — the workstation has no route into the scenario", { skip, timeout: 60_000 }, async () => { +test("ADR 0016 — the workstation has no route into the scenario", { skip, timeout: 60_000 }, async () => { // Reachability is asked from INSIDE. If the workstation could reach a scenario address, // two scenarios carrying the same prefix would put one's traffic in the other. const { stdout } = await incus(["exec", `mlab-${instanceId}-anchor`, "--", "echo", "inside"]); @@ -171,7 +171,7 @@ test("the live diagram draws what exists, never what was asked for", { skip, tim } }); -test("ADR 0033 — the live diagram distinguishes scenery from a node", { skip }, async () => { +test("ADR 0016 — the live diagram distinguishes scenery from a node", { skip }, async () => { // The router is drawn as a router because the hypervisor says it is a container tagged as // a gateway — not because the diagram re-read the scenario and inferred it. const drawn = await diagramFromLive(instanceId); diff --git a/test/lastrun.test.ts b/test/lastrun.test.ts new file mode 100644 index 0000000..3e6d0b9 --- /dev/null +++ b/test/lastrun.test.ts @@ -0,0 +1,167 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { endToEnd, headOf, judge, type Receipt } from "../src/lastrun.ts"; + +const now = new Date("2026-08-31T12:00:00Z"); +const passing = (at: string, against: Record): Receipt => + ({ at, passed: 22, failed: 0, against, ran: [endToEnd] }); + +// A machine that has never run it is told so, rather than told nothing. +// +// novox/hq 04-ISSUES/005: the harness it replaces had not built for two and a half months and +// nothing said so. Silence and success must never look alike. +test("a machine that has never run the suite is told so", () => { + const said = judge(null, now, { "mesh-lab": "aaa" }); + assert.equal(said.current, false); + assert.match(said.lines.join("\n"), /never run/); +}); + +// The one that matters: it passed, and against code nobody runs any more. +test("a run against code that has since changed is not current", () => { + const said = judge( + passing("2026-08-31T11:00:00Z", { "mesh-lab": "aaa", "mesh-control": "bbb" }), + now, + { "mesh-lab": "aaa", "mesh-control": "ccc" }, + ); + assert.equal(said.current, false, "a run against changed code was reported as current"); + const text = said.lines.join("\n"); + assert.match(text, /mesh-control\s+at bbb, now at ccc/, text); + assert.match(text, /code that has since changed/, text); +}); + +// Passing, recent, and against exactly this code is the only thing that counts. +test("a recent run against this code is current", () => { + const said = judge( + passing("2026-08-31T11:00:00Z", { "mesh-lab": "aaa" }), + now, + { "mesh-lab": "aaa" }, + ); + assert.equal(said.current, true, said.lines.join("\n")); + assert.match(said.lines.join("\n"), /unchanged/); +}); + +// Old is a different complaint from moved, and says so — otherwise somebody goes looking for a +// change that did not happen. +test("a run that is merely old says that, not that something changed", () => { + const said = judge( + passing("2026-08-01T11:00:00Z", { "mesh-lab": "aaa" }), + now, + { "mesh-lab": "aaa" }, + ); + assert.equal(said.current, false); + const text = said.lines.join("\n"); + assert.match(text, /Nothing has changed since/, text); + assert.doesNotMatch(text, /has since changed/, text); +}); + +// A failed run is recorded, and does not count as coverage. +test("a run that failed is not coverage", () => { + const said = judge( + { + at: "2026-08-31T11:00:00Z", + passed: 21, + failed: 1, + against: { "mesh-lab": "aaa" }, + ran: [endToEnd], + }, + now, + { "mesh-lab": "aaa" }, + ); + assert.equal(said.current, false); + assert.match(said.lines.join("\n"), /1 test\(s\) failed/); + assert.match(said.lines.join("\n"), /Nothing has been proven end to end since/); +}); + +// A repository the run never accounted for is named, rather than passing silently: a receipt that +// says nothing about something is not a receipt that clears it. +test("a repository the run did not account for is named", () => { + const said = judge( + passing("2026-08-31T11:00:00Z", { "mesh-lab": "aaa" }), + now, + { "mesh-lab": "aaa", "mesh-host": "ddd" }, + ); + assert.equal(said.current, false); + assert.match(said.lines.join("\n"), /mesh-host\s+was not accounted for/); +}); + +// A green run of something else is not a green run of this. +// +// The suite takes paths, so it can be pointed at one quick unit file. Without recording what it +// ran, that receipt and a receipt for the real thing are the same document — which is the whole +// fault of novox/hq 04-ISSUES/005, reintroduced by the fix for it. +test("a run that raised no machines is not end-to-end coverage", () => { + const said = judge( + { + at: "2026-08-31T11:00:00Z", + passed: 6, + failed: 0, + against: { "mesh-lab": "aaa" }, + ran: ["test/lastrun.test.ts"], + }, + now, + { "mesh-lab": "aaa" }, + ); + assert.equal(said.current, false, "a unit run was accepted as end-to-end coverage"); + assert.match(said.lines.join("\n"), /raised no machines/); +}); + +// A receipt written before the mesh recorded what it ran claims nothing, and is read as claiming +// nothing — not as claiming everything. +test("a receipt from before this was recorded is not read as covering everything", () => { + const old = { at: "2026-08-31T11:00:00Z", passed: 22, failed: 0, against: { "mesh-lab": "aaa" } }; + const said = judge(old as unknown as Receipt, now, { "mesh-lab": "aaa" }); + assert.equal(said.current, false); +}); + +// A dirty tree is never equal to the clean commit it sits on. +// +// The run tested what was on disk. Naming the bare hash would claim coverage of code nobody can +// check out — and nothing else could tell, because the hash is identical either way. +test("a run taken against uncommitted work does not count as covering the commit", () => { + const said = judge(passing("2026-08-31T11:00:00Z", { "mesh-lab": "aaa+uncommitted" }), now, { + "mesh-lab": "aaa", + }); + assert.equal(said.current, false, "a run against uncommitted work was read as covering the commit"); + assert.match(said.lines.join("\n"), /aaa\+uncommitted, now at aaa/); +}); + +// headOf against a real repository, because the rule lives in headOf and not in judge. +// +// The first test written for this marked a hand-built receipt and passed with the marking removed +// — it checked how judge reads the value, never that anything produces it. A test that cannot fail +// when the behaviour is deleted is not defending the behaviour. +test("a repository with uncommitted work reports a commit that is marked as such", () => { + const repo = mkdtempSync(join(tmpdir(), "mesh-lab-headof-")); + try { + const git = (...args: string[]) => + execFileSync("git", ["-C", repo, ...args], { stdio: ["ignore", "pipe", "ignore"] }); + git("init", "-q"); + git("config", "user.email", "test@example.invalid"); + git("config", "user.name", "test"); + writeFileSync(join(repo, "a"), "one\n"); + git("add", "a"); + git("commit", "-qm", "first"); + + const clean = headOf(repo); + assert.match(clean, /^[0-9a-f]+$/, `a clean tree was reported as ${clean}`); + + writeFileSync(join(repo, "a"), "two\n"); + assert.equal(headOf(repo), `${clean}+uncommitted`, "an uncommitted change was not marked"); + } finally { + rmSync(repo, { recursive: true, force: true }); + } +}); + +// A directory that is not a checkout is absent from the receipt, not guessed at. +test("a directory that is not a repository reports nothing", () => { + const plain = mkdtempSync(join(tmpdir(), "mesh-lab-plain-")); + try { + assert.equal(headOf(plain), ""); + } finally { + rmSync(plain, { recursive: true, force: true }); + } +}); diff --git a/test/log.test.ts b/test/log.test.ts new file mode 100644 index 0000000..5e68dbf --- /dev/null +++ b/test/log.test.ts @@ -0,0 +1,172 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readFileSync, mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { around, log, logTo, shorten } from "../src/log.ts"; + +function intoAFile(): string { + const path = join(mkdtempSync(join(tmpdir(), "mesh-lab-log-")), "run.log"); + logTo(path, "debug"); + return path; +} + +function read(path: string): string { + try { + return readFileSync(path, "utf8"); + } catch { + return ""; + } +} + +// Nothing at all unless asked. A suite that writes a debug log nobody reads is a suite that +// writes a debug log nobody reads. +test("it is off until it is turned on", () => { + const path = join(mkdtempSync(join(tmpdir(), "mesh-lab-log-")), "run.log"); + logTo(null); + log.info("this should go nowhere"); + assert.equal(read(path), ""); + assert.equal(log.on(), false); +}); + +test("what it records says when, and how far into the run", () => { + const path = intoAFile(); + log.info("a thing happened"); + const written = read(path); + assert.match(written, /a thing happened/); + assert.match(written, /^\d{4}-\d{2}-\d{2}T/, `no timestamp:\n${written}`); + assert.match(written, /\d+\.\ds/, `no elapsed time:\n${written}`); + logTo(null); +}); + +test("a level below the one asked for is not written", () => { + const path = join(mkdtempSync(join(tmpdir(), "mesh-lab-log-")), "run.log"); + logTo(path, "info"); + log.info("kept"); + log.debug("dropped"); + log.trace("dropped"); + const written = read(path); + assert.match(written, /kept/); + assert.doesNotMatch(written, /dropped/); + logTo(null); +}); + +// **The assertion this file exists for** (novox/hq 04-ISSUES/024). A line before and a line after +// tells you nothing until the after arrives, which is precisely the case that matters. +test("something still running says so while it runs", async () => { + const path = intoAFile(); + await around("a slow thing", async () => { + await new Promise((r) => setTimeout(r, 250)); + }, { heartbeatMs: 60 }); + const written = read(path); + assert.match(written, /still running after/, + `nothing was said while it ran:\n${written}`); + assert.match(written, /→ a slow thing/); + assert.match(written, /← a slow thing \(0\.\ds\)/, `it did not say how long it took:\n${written}`); + logTo(null); +}); + +// A quick command should not litter the log with heartbeats it never needed. +test("something quick says only that it happened", async () => { + const path = intoAFile(); + await around("a quick thing", async () => "done", { heartbeatMs: 10_000 }); + const written = read(path); + assert.doesNotMatch(written, /still running/); + assert.match(written, /← a quick thing/); + logTo(null); +}); + +// A failure is worth more than a success, so it survives a level that would have dropped it. +test("a failure is recorded even when the step was not", async () => { + const path = join(mkdtempSync(join(tmpdir(), "mesh-lab-log-")), "run.log"); + logTo(path, "info"); + await assert.rejects( + around("a failing thing", async () => { + throw new Error("it did not work"); + }, { at: "info" }), + ); + const written = read(path); + assert.match(written, /✗ a failing thing/, `the failure was not recorded:\n${written}`); + assert.match(written, /it did not work/); + logTo(null); +}); + +test("what it returns is what the step returned", async () => { + logTo(null); + assert.equal(await around("a thing", async () => 42), 42); +}); + +// A command line is identifiable in the log even when it is very long. +test("a long command keeps its beginning", () => { + const long = shorten(["incus", "exec", "machine", "--", "sh", "-c", "x".repeat(500)]); + assert.ok(long.length <= 200, `it is ${long.length} characters`); + assert.match(long, /^incus exec machine/); + assert.match(long, /…$/); +}); + +// A question that answers "no" is not a fault, and must not be logged as one. +// +// Plenty of the lab's commands are questions — does this network exist, is the agent up yet — and +// they fail constantly while a scenario comes up. Recording those as faults fills a healthy run +// with ✗ and teaches whoever reads it that ✗ means nothing. +test("a failure the caller expects is recorded quietly", async () => { + const path = join(mkdtempSync(join(tmpdir(), "mesh-lab-log-")), "run.log"); + logTo(path, "info"); + await assert.rejects( + around("asking whether a thing exists", async () => { + throw new Error("not found"); + }, { expectedToFail: true }), + ); + assert.equal(read(path), "", `an expected answer was reported as a fault:\n${read(path)}`); + + // And it is still there for anyone who turns the log up. + logTo(path, "debug"); + await assert.rejects( + around("asking again", async () => { + throw new Error("not found"); + }, { expectedToFail: true }), + ); + assert.match(read(path), /· asking again/, `it was dropped entirely:\n${read(path)}`); + logTo(null); +}); + +// And a failure nobody expected is still loud at the same level. +test("a failure the caller does not expect is recorded loudly", async () => { + const path = join(mkdtempSync(join(tmpdir(), "mesh-lab-log-")), "run.log"); + logTo(path, "info"); + await assert.rejects( + around("doing a thing", async () => { + throw new Error("it broke"); + }), + ); + assert.match(read(path), /✗ doing a thing/, `a real failure was quiet:\n${read(path)}`); + logTo(null); +}); + +// A turned-down log still shows a failure and still beats while something runs. +// +// **Both were lost, briefly, to an optimisation.** `around` skipped its own wrapper whenever the +// step's level was below the configured one — which is the ordinary case at `info` — taking the +// failure line and the heartbeat with it. The two things worth having at a low level were the two +// that disappeared. +test("at info, a debug step still reports failing and still beats", async () => { + const path = join(mkdtempSync(join(tmpdir(), "mesh-lab-log-")), "run.log"); + logTo(path, "info"); + + await around("a slow debug step", async () => { + await new Promise((r) => setTimeout(r, 200)); + }, { heartbeatMs: 50, at: "debug" }); + assert.match(read(path), /still running after/, `no heartbeat at info:\n${read(path)}`); + + await assert.rejects( + around("a failing debug step", async () => { + throw new Error("it broke"); + }, { at: "debug" }), + ); + assert.match(read(path), /✗ a failing debug step/, `no failure at info:\n${read(path)}`); + + // And the ordinary begin/end pair is still held back, which is what the level asked for. + assert.doesNotMatch(read(path), /→ a slow debug step/); + logTo(null); +}); diff --git a/test/pinning.test.ts b/test/pinning.test.ts new file mode 100644 index 0000000..d31fb2a --- /dev/null +++ b/test/pinning.test.ts @@ -0,0 +1,73 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; + +import { pinnedInto, repositoryOf, stillUnpinned } from "../src/pinning.ts"; + +const SERVED = [ + "192.0.2.250:5000/postgres@sha256:" + "a".repeat(64), + "192.0.2.250:5000/mesh-provision-postgres@sha256:" + "b".repeat(64), + "192.0.2.250:5000/gitea/gitea@sha256:" + "c".repeat(64), + "192.0.2.250:5000/ghcr.io/mailu/admin@sha256:" + "d".repeat(64), +]; + +test("the repository is what survives being served somewhere else", () => { + assert.equal(repositoryOf(SERVED[0]!), "postgres"); + assert.equal(repositoryOf(SERVED[2]!), "gitea/gitea"); + assert.equal(repositoryOf(SERVED[3]!), "ghcr.io/mailu/admin"); + assert.equal(repositoryOf("alpine"), "alpine"); +}); + +// The case this exists for: an image the mesh builds has no digest until it is built, so a +// manifest ships sixty-four zeros and would stop on the machine (novox/hq 04-ISSUES/025). +test("a placeholder for one of our own images becomes the one this scenario serves", () => { + const before = `"image": "mesh-provision-postgres@sha256:${"0".repeat(64)}"`; + const after = pinnedInto(before, SERVED); + assert.match(after, /192\.0\.2\.250:5000\/mesh-provision-postgres@sha256:b{64}/); + assert.deepEqual(stillUnpinned(after), []); +}); + +// And a real third-party digest is replaced too — the text says which image, the scenario says +// which copy of it. +test("a real digest is redirected to this scenario's copy", () => { + const before = `"image": "gitea/gitea@sha256:${"f".repeat(64)}"`; + assert.match(pinnedInto(before, SERVED), /192\.0\.2\.250:5000\/gitea\/gitea@sha256:c{64}/); +}); + +test("a reference that already carries a registry is still redirected", () => { + const before = `"image": "docker.io/postgres@sha256:${"e".repeat(64)}"`; + assert.match(pinnedInto(before, SERVED), /192\.0\.2\.250:5000\/postgres@sha256:a{64}/); +}); + +// **Left alone, not blanked.** A repository this scenario did not stock may be reachable some +// other way, and emptying the reference would produce the exact failure this prevents. +test("something the scenario does not serve is untouched", () => { + const before = `"image": "redis@sha256:${"9".repeat(64)}"`; + assert.equal(pinnedInto(before, SERVED), before); +}); + +// A longer repository ending in a shorter one must not be half-replaced. +test("a repository that ends in another one is not partly rewritten", () => { + const before = `"image": "my-postgres@sha256:${"7".repeat(64)}"`; + assert.equal(pinnedInto(before, SERVED), before, + "'my-postgres' was rewritten because it ends in 'postgres'"); +}); + +test("every image in a whole manifest is redirected at once", () => { + const manifest = JSON.stringify({ + resources: [ + { id: "db", image: `postgres@sha256:${"1".repeat(64)}` }, + { id: "prov", image: `mesh-provision-postgres@sha256:${"0".repeat(64)}` }, + { id: "app", image: `gitea/gitea@sha256:${"2".repeat(64)}` }, + ], + }); + const after = pinnedInto(manifest, SERVED); + assert.deepEqual(stillUnpinned(after), []); + for (const want of ["a".repeat(64), "b".repeat(64), "c".repeat(64)]) { + assert.ok(after.includes(want), `missing ${want.slice(0, 6)}… in ${after}`); + } +}); + +test("what is still a placeholder can be named", () => { + const text = `"image": "something-of-ours@sha256:${"0".repeat(64)}"`; + assert.deepEqual(stillUnpinned(text), ["something-of-ours"]); +}); diff --git a/test/place.test.ts b/test/place.test.ts index 29cb3d0..009a225 100644 --- a/test/place.test.ts +++ b/test/place.test.ts @@ -1,12 +1,12 @@ import { test } from "node:test"; import assert from "node:assert/strict"; import { parseScenario } from "../src/declaration/parse.ts"; -import { planPlacements, PLACEABLE } from "../src/lifecycle/place.ts"; +import { planPlacements, PLACEABLE, isPlaceable } from "../src/lifecycle/place.ts"; import { assertSupported, UnsupportedError } from "../src/lifecycle/supported.ts"; /** * `place:` is the seam where the lab stops being infrastructure with no consumer. Each test - * names what it defends, per novox/hq ADR 0034. + * names what it defends, per novox/hq ADR 0017. */ function scenario(place: string): ReturnType { @@ -87,3 +87,45 @@ test("the refusal says what CAN be placed", () => { } } }); + +// --- runtime and image placement (novox/hq ADR 0006: the lab places what a sealed scenario +// cannot fetch) --- + +test("an image reference is placeable, and a bare 'image:' is not", () => { + assert.ok(isPlaceable("image:alpine@sha256:abc"), "a reference should be placeable"); + assert.ok(isPlaceable("image:postgres:17"), "a tag is the scenario's business, not the lab's"); + assert.ok(!isPlaceable("image:"), "there is nothing to place"); + assert.ok(!isPlaceable("image: "), "whitespace is not a reference"); +}); + +test("the placeables are host, runtime and an image", () => { + assert.ok(isPlaceable("host")); + assert.ok(isPlaceable("runtime")); + assert.ok(!isPlaceable("substrate"), "the tiers above tier 0 do not exist yet"); + assert.ok(!isPlaceable("control-plane")); +}); + +test("an unplaceable artifact is named, not refused as a whole", () => { + // A scenario placing a host and a substrate is told which half the lab cannot do — refusing + // wholesale would send somebody looking for the wrong problem. + const scenario = { + name: "s", + segments: {}, + machines: { a: {} as never }, + place: { a: ["host", "runtime", "image:alpine@sha256:x", "substrate"] }, + } as unknown as Parameters[0]; + + assert.throws( + () => assertSupported(scenario), + (err: Error) => { + // Checked as `place: —`, which is how an artifact is REPORTED as + // unplaceable. Searching for the bare word matched the message's own list of what CAN + // be placed, which mentions runtime — so the test failed on a correct message. + assert.ok(err.message.includes("place: substrate —"), "the unplaceable one is named"); + assert.ok(!err.message.includes("place: image:alpine"), "a placeable one is not"); + assert.ok(!err.message.includes("place: runtime —"), "nor is runtime"); + assert.ok(!err.message.includes("place: host —"), "nor is host"); + return true; + }, + ); +}); diff --git a/test/rebuild.test.ts b/test/rebuild.test.ts new file mode 100644 index 0000000..cffe81b --- /dev/null +++ b/test/rebuild.test.ts @@ -0,0 +1,68 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { planned } from "../src/rebuild.ts"; +import { repositories } from "../src/repos.ts"; + +// The control plane's image and the builder are one step, not two. +// +// Both parse manifests. On 2026-08-30 a rename was built into the image and not the binary, and +// the run that found out was a full lab raise. novox/hq 04-ISSUES/005. +test("the control plane's image and builder are always built together", () => { + const builds = planned({ + MESH_LAB_MODULES: "/repo/control/examples/modules", + MESH_LAB_BUILDER: "/repo/control/build/mesh-builder", + }); + const what = builds.map((b) => b.what); + assert.ok(what.includes("images"), "the images were not built"); + assert.ok(what.includes("builder"), "the builder was not built"); + for (const build of builds) assert.equal(build.in, "/repo/control"); +}); + +// Every image the lab runs, not only the control plane's. +// +// On 2026-09-01 a run had a control-plane image built that minute and a provisioner image built +// the day before. A test against a real database failed, and it looked exactly like the change +// under test being wrong: the provisioner was creating logins by a naming rule that had been +// replaced hours earlier. novox/hq 04-ISSUES/005 again, one target along. +// +// Named individually rather than by counting, because the failure this guards is a target that +// exists and is not run — which a count would not notice. +test("every image the lab runs is rebuilt, not only the control plane's", () => { + const builds = planned({ MESH_LAB_MODULES: "/repo/control/examples/modules" }); + const images = builds.find((b) => b.what === "images"); + assert.ok(images, "no image build at all"); + for (const target of [ + "image", "builder-image", "provisioner-image", "objectstore-image", + "redis-provisioner-image", "proxy-image", + ]) { + assert.ok(images.argv.includes(target), `${target} is never built, so the lab runs a stale one`); + } +}); + +// A repository this run was not pointed at is not built, and not claimed. +test("only what this run was pointed at is built", () => { + assert.deepEqual(planned({}), []); + const hostOnly = planned({ MESH_LAB_HOST_BINARY: "/repo/host/mesh-host" }); + assert.deepEqual(hostOnly.map((b) => b.what), ["host"]); + assert.equal(hostOnly[0]!.in, "/repo/host"); +}); + +// What the receipt claims and what the run built come from one derivation. +// +// They are separate concerns that must agree: a receipt naming a repository the run did not build +// is false coverage arriving by nobody's decision — just two derivations drifting apart. +// novox/hq 04-ISSUES/005. +test("every repository the receipt claims was built by the run", () => { + const env = { + MESH_LAB_HOST_BINARY: "/repo/host/mesh-host", + MESH_LAB_MODULES: "/repo/control/examples/modules", + MESH_LAB_BUILDER: "/repo/control/build/mesh-builder", + }; + const built = new Set(planned(env).map((b) => b.in)); + for (const [name, directory] of Object.entries(repositories(env))) { + // mesh-lab is the exception, and it is not an omission: it is TypeScript run from source, so + // the code under test *is* the code running. There is nothing to build and nothing to go stale. + if (name === "mesh-lab") continue; + assert.ok(built.has(directory), `${name} (${directory}) is claimed but never built`); + } +}); diff --git a/test/registry.test.ts b/test/registry.test.ts new file mode 100644 index 0000000..36229d5 --- /dev/null +++ b/test/registry.test.ts @@ -0,0 +1,66 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { digestFrom, pinnedReference, registryAddress, repositoryFor } from "../src/lifecycle/registry.ts"; + +/** + * The registry inside a scenario (novox/hq 04-ISSUES/009). + * + * These test the pure parts. The parts that need a registry are exercised by raising a + * scenario, because a fake registry would assert that the fake behaves as expected + * (novox/hq ADR 0017). + */ + +test("a digest is read from what the registry actually said", () => { + // The real shape of `docker push` output. The digest here is the REGISTRY's, not Docker + // Hub's, and that is the point: a declaration pins what this registry serves. + const output = + "The push refers to repository [localhost:5000/alpine]\n" + + "63f227048c13: Pushed\n" + + "3.20: digest: sha256:6c2a9711b0a9f32b0239d9222eb1072309cf46c6431d319ae249186d811a987c size: 528\n"; + assert.equal( + digestFrom(output), + "sha256:6c2a9711b0a9f32b0239d9222eb1072309cf46c6431d319ae249186d811a987c", + ); +}); + +test("no digest is not an empty digest", () => { + // A push that reported no digest leaves nothing for a declaration to pin, and inventing one + // would be worse than failing — the host would refuse it later, further from the cause. + assert.equal(digestFrom("The push refers to repository [localhost:5000/alpine]\n"), null); + assert.equal(digestFrom(""), null); + // Hex, but the wrong LENGTH. An earlier version used "tooshort", whose letters fall outside + // a-f — so it failed the character class and proved nothing about the length check. + assert.equal(digestFrom("digest: sha256:abc123"), null); + assert.equal(digestFrom("digest: sha256:" + "a".repeat(63)), null, "63 is not 64"); +}); + +test("the repository is the reference without its tag", () => { + assert.equal(repositoryFor("alpine:3.20"), "alpine"); + assert.equal(repositoryFor("alpine"), "alpine"); + assert.equal(repositoryFor("library/postgres:17"), "library/postgres"); + // A port in a hostname is a colon that is NOT a tag, and treating it as one would serve the + // image from a truncated path. + assert.equal(repositoryFor("localhost:5000/alpine:3.20"), "localhost:5000/alpine"); + assert.equal(repositoryFor("localhost:5000/alpine"), "localhost:5000/alpine"); +}); + +test("the registry's address is derived from its segment", () => { + assert.equal(registryAddress("192.0.2.0/24"), "192.0.2.250"); + assert.equal(registryAddress("198.51.100.0/24"), "198.51.100.250"); + // An IPv6-only segment cannot host it, and saying so beats producing an address nothing + // can be pointed at. + assert.throws(() => registryAddress("2001:db8:a::/48"), /not an IPv4 network/); +}); + +test("what a declaration pins is the registry's own digest", () => { + // Not Docker Hub's. ADR 0006 requires a reference that is exact and cannot move, and a + // digest this registry assigned is both. + const pinned = pinnedReference("192.0.2.250", { + requested: "alpine:3.20", + repository: "alpine", + digest: "sha256:" + "6".repeat(64), + }); + assert.equal(pinned, `192.0.2.250:5000/alpine@sha256:${"6".repeat(64)}`); + assert.ok(pinned.includes("@sha256:"), "the host refuses anything not pinned by digest"); + assert.ok(!pinned.includes(":3.20"), "a tag would move; the digest is what is pinned"); +}); diff --git a/test/suite.test.ts b/test/suite.test.ts new file mode 100644 index 0000000..a168555 --- /dev/null +++ b/test/suite.test.ts @@ -0,0 +1,68 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { counted, reportOn } from "../src/suite.ts"; + +// The totals come from the runner's own summary, and from nothing else. +test("the runner's totals are read from its summary", () => { + const said = counted("✔ something (1ms)\nℹ tests 22\nℹ pass 22\nℹ fail 0\n"); + assert.deepEqual(said, { passed: 22, failed: 0 }); +}); + +// A test *named* like a total must not be mistaken for one. The summary is a line of its own, and +// the pattern says so — otherwise a test called "pass 3" would rewrite the record. +test("a test named like a total is not a total", () => { + const said = counted("✔ a machine reports pass 3 things (1ms)\nℹ pass 22\nℹ fail 0\n"); + assert.equal(said.passed, 22, "a test name was read as the total"); +}); + +// A failing run is read as a failing run. +test("failures are read", () => { + const said = counted("ℹ pass 21\nℹ fail 1\n"); + assert.deepEqual(said, { passed: 21, failed: 1 }); +}); + +// **No totals is not zero failures.** A run whose result could not be read is a run nobody can say +// anything about, and writing "0 failed" because nothing said otherwise is how a green record +// comes to mean nothing — which is the whole of 04-ISSUES/005. +test("output with no summary yields no totals rather than a clean bill", () => { + const said = counted("the runner crashed before it said anything\n"); + assert.equal(said.passed, null); + assert.equal(said.failed, null); +}); + +// No receipt rather than a guessed one. +// +// The rule that keeps the record meaning something, and it was first written where nothing could +// check it — 04-ISSUES/005 in miniature, inside the fix for it. +test("a run whose result could not be read writes nothing", () => { + let wrote = false; + const said = reportOn({ passed: null, failed: null }, () => { + wrote = true; + return { against: {}, ran: [] }; + }); + assert.equal(wrote, false, "a receipt was written for a run nobody could read"); + assert.match(said, /no receipt written/); +}); + +test("a run that was read is recorded, with what it was read against", () => { + let got: [number, number] | null = null; + const said = reportOn({ passed: 22, failed: 0 }, (p, f) => { + got = [p, f]; + return { against: { "mesh-lab": "abc1234" }, ran: ["test/integration/mesh.test.ts"] }; + }); + assert.deepEqual(got, [22, 0]); + assert.match(said, /mesh\.test\.ts — 22 passed, 0 failed, against mesh-lab abc1234/); +}); + +// The runner's real output, colours and all. +// +// Captured from `node --test` writing into a pipe rather than written by hand: the first version of +// counted() passed every test and read nothing, because the fixtures were clean text and the runner +// emits escape codes. A fixture that agrees with the mistake proves the mistake. +test("the runner's totals are read from output as it actually arrives", () => { + const real = "\u001b[34m\u2139 suites 0\u001b[39m\n" + + "\u001b[34m\u2139 pass 22\u001b[39m\n" + + "\u001b[34m\u2139 fail 0\u001b[39m\n" + + "\u001b[34m\u2139 duration_ms 98.9\u001b[39m\n"; + assert.deepEqual(counted(real), { passed: 22, failed: 0 }); +}); diff --git a/test/supported.test.ts b/test/supported.test.ts index eae6fd6..bda3058 100644 --- a/test/supported.test.ts +++ b/test/supported.test.ts @@ -51,7 +51,7 @@ machines: { a: { at: { segment: net, address: [192.0.2.1] }, inbound: deny } }`) test("the host is placeable — it used to be refused, and tier 0 now exists", () => { // These two tests failed the moment placement worked, which is what they were for. They // defended "there is nothing to place yet" while that was true; the decision changed, so - // they change with it rather than being deleted (novox/hq ADR 0034). + // they change with it rather than being deleted (novox/hq ADR 0017). const scenario = parseScenario(`scenario: x segments: { net: { kind: public, cidr: [192.0.2.0/24] } } machines: { a: { at: { segment: net, address: [192.0.2.1] } } } @@ -84,3 +84,13 @@ segments: { net: { kind: public, cidr: [192.0.2.0/24] } } machines: { a: { at: { segment: net, address: [192.0.2.1] }, inbound: allow } }`); assert.doesNotThrow(() => assertSupported(scenario)); }); + +test("egress is parsed, and off unless asked for", () => { + const scenario = parseScenario(`scenario: x +segments: { net: { kind: public, cidr: [192.0.2.0/24] } } +machines: + a: { at: { segment: net, address: [192.0.2.1] }, egress: true } + b: { at: { segment: net, address: [192.0.2.2] } }`); + assert.equal(scenario.machines["a"]?.egress, true); + assert.equal(scenario.machines["b"]?.egress, undefined); +}); diff --git a/test/validate.test.ts b/test/validate.test.ts index 1256c1a..e07a309 100644 --- a/test/validate.test.ts +++ b/test/validate.test.ts @@ -243,3 +243,17 @@ machines: /one gateway.*disagree.*forwardable/s, ); }); + +test("a detached machine cannot declare egress", () => { + refuses(`scenario: x +segments: { net: { kind: public, cidr: [192.0.2.0/24] } } +machines: { a: { at: detached, egress: true } }`, + /detached but declares egress/); +}); + +test("a segment may not be named 'uplink' — the lab claims that name for egress", () => { + refuses(`scenario: x +segments: { uplink: { kind: public, cidr: [192.0.2.0/24] } } +machines: { a: { at: { segment: uplink, address: [192.0.2.1] }, egress: true } }`, + /reserved for the lab's own NAT bridge/); +}); diff --git a/test/warm.test.ts b/test/warm.test.ts new file mode 100644 index 0000000..adf2207 --- /dev/null +++ b/test/warm.test.ts @@ -0,0 +1,61 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { judge, type Warm } from "../src/warm.ts"; + +const at = "2026-08-31T20:00:00Z"; +const built = { "mesh-lab": "aaa", "mesh-host": "bbb", "mesh-control": "ccc" }; +const warm = (over: Partial = {}): Warm => + ({ scenario: "two-nodes", instanceId: "mlab-two-nodes-1", images: [], against: built, at, ...over }); + +// The check this exists for: a mesh warmed against code that has since moved would pass today's +// tests against yesterday's binaries, and the result would say nothing about it. +// +// Same fault as novox/hq 04-ISSUES/005, one level down — a green result standing for a run +// against something other than what is in front of you. +test("a warm mesh built from code that has moved is refused", () => { + const said = judge(warm(), "two-nodes", ["mlab-two-nodes-1"], true, + { ...built, "mesh-host": "moved" }); + assert.equal(said.use, "raise"); + assert.match(said.use === "raise" ? said.why : "", /mesh-host was at bbb.*now at moved/); +}); + +test("a warm mesh built from the same code is used", () => { + const said = judge(warm(), "two-nodes", ["mlab-two-nodes-1"], true, built); + assert.equal(said.use, "restore"); +}); + +// Each refusal is named, because each is a different thing being wrong. +test("every reason to raise instead says which reason it was", () => { + const cases: [string, ReturnType][] = [ + ["nothing kept", judge(null, "two-nodes", [], true, built)], + ["another scenario", judge(warm({ scenario: "first-node" }), "two-nodes", + ["mlab-two-nodes-1"], true, built)], + ["not standing", judge(warm(), "two-nodes", [], true, built)], + ["no snapshot", judge(warm(), "two-nodes", ["mlab-two-nodes-1"], false, built)], + ]; + for (const [what, said] of cases) { + assert.equal(said.use, "raise", what); + assert.ok(said.use === "raise" && said.why.length > 10, + `${what} was refused without saying why: ${JSON.stringify(said)}`); + } +}); + +// A repository the warm record never accounted for is a difference, not a match. +test("a repository that was not recorded when it was warmed is refused", () => { + const said = judge(warm({ against: { "mesh-lab": "aaa" } }), "two-nodes", + ["mlab-two-nodes-1"], true, built); + assert.equal(said.use, "raise"); +}); + +// A repository the current run cannot see is not a repository that agrees. +// +// **Found by testing the guard rather than trusting it.** The first version walked only the +// repositories the current environment names, so running without that environment compared +// nothing and called a stale mesh usable. The mesh had genuinely moved; the check had looked at +// neither side. +test("a repository this run cannot locate is refused, not passed over", () => { + const said = judge(warm(), "two-nodes", ["mlab-two-nodes-1"], true, { "mesh-lab": "aaa" }); + assert.equal(said.use, "raise"); + assert.match(said.use === "raise" ? said.why : "", + /nothing says where it is now|so nothing can say whether it moved/); +});