Both image scripts copied the runtime's installed tree and relied on the sdk
inside it being a link into the sibling repository. That is true only where
somebody linked them by hand, and false as soon as the dependencies are
installed the ordinary way — which fetches the sdk as sources with nothing
compiled. The image still built, and every entry point in it pointed at
nothing.
The four-node bed proves genesis entangled with three machines joining across a
gateway, so the cheapest check of the install path costs a four-machine raise.
This is genesis alone: one machine, the installer, and the question asked of the
machine rather than inferred from an exit code.
Genesis moves into a shared routine both beds call, rather than being described a
second time here — a second description kept in step with the first is what put
the whole procedure inside a fixture to begin with.
The scenario needs two things the first draft missed, and both cost a full raise
to discover: a way out to the internet, because the installer's first act is to
pull the substrate; and a container runtime, because the installer's first refusal
is a machine that has none.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
postgres and lavinmq carried 127.0.0.1: in their remap, and it broke a consumer
on a node that has no substrate to collide with. A module is told to reach its
provider at <node>.internal, that name is the node's overlay address, and a
provider listening only on loopback refuses it — letta on ace failed with 'is the
server running on that host and accepting TCP/IP connections?' while postgres sat
healthy beside it.
The collision needed a different port, which is what every other entry here does.
The address was never part of it, and it made the provider unreachable by the one
name the mesh hands its consumers.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
ADR 0067's own acceptance check said the lab must raise its anchor by running the
program a bare machine runs. It did not: whole-mesh-full applied the substrate bundle
by hand and then looped enrolment over all four machines as one continuous operation.
That gets the order right by accident and models the wrong shape — and an install
procedure that exists only as a test fixture is exercised by whoever writes tests and
never by whoever installs, which is why every bootstrap fault this year was found late.
Two acts now, and the first gates the second.
GENESIS is novox running mesh-bootstrap: the installer is built from source before
the raise (make bootstrap, carrying the control-plane image built in the same run),
placed beside the host binary, given the two manifests it reads, and run. The bed
then asserts a WORKING MESH OF ONE — the control plane answers, the registry replies
on /v2/, the container called mesh-control is running from a registry-pinned digest
rather than an image id, the registry agrees it serves it, temp-mesh-control is gone,
and the mesh has heard from its node. The image-id check is ADR 0067's "the pivot
completed" verbatim: if it is still an id, nothing was published and this mesh can
never roll out its own upgrades.
JOINING is ace, shanks and g14: host binary, token, enrol, run. novox is NOT enrolled
again — the installer already did it, and a second identity is one the mesh does not
know.
If genesis stops, the bed prints which of the installer's ten steps it stopped at and
goes no further. A second machine joining a mesh that is not ready is a different
failure, and running it would bury this one underneath it.
The anchor is no longer handed mesh-control:development. Its absence is the point: the
installer carries that image inside itself, and handing it over as well would make the
load say "already held" and leave the carrying untested — the same class of fiction the
lab's own registry used to hide. A unit test asserts the scenario keeps it out.
The registry is reached at 127.0.0.1:5000, which is a finding rather than a shortcut: a
runtime refuses a plain-HTTP registry at any address but a loopback one, so the digest
the control-plane module is pinned to is one only the anchor can pull. Enough here,
because only the anchor runs a control plane. Written down in the bed.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Deleting the lab's registry left the operator's own images to be pulled like
anything else, and they cannot be: their registry wants an account and a
scenario machine has none. The pull fails with 'no basic auth credentials',
which is not something more patience fixes.
So the test is no longer 'did the mesh build it' but 'can the machine get it at
all'. Two ways to fail that — published nowhere, or published somewhere the
machine cannot authenticate to — and one consequence: the workstation, which
does hold the credential, exports it and loads it.
Worth saying what this stands in for. In a finished mesh these are built by the
builder and published to the mesh's own store, and every machine pulls them from
there with a credential the mesh granted. Until that store exists there is
nowhere for them to come from, and handing them over is the closest honest thing
— not a registry the lab invents, which is what was just removed.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The id is the digest of an image's configuration, and a runtime rewrites that
configuration as it loads: this workstation saves in one format, the machines
store it in another, and the same bytes arrive under a different name. Measured,
not assumed — b86bb81c here, 2dc21904 there.
So it is read back from the machine instead of predicted from here. Predicting
it failed at the only moment it mattered: every manifest would have been
rewritten to a reference no machine holds, and these images exist in no registry,
so each apply would have stopped at a pull that cannot succeed. A manifest
carries one reference, so machines that disagree stop the raise rather than
having one of them silently win.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
It is the commonest home LAN range there is, so on an ordinary workstation the
lab's private segment and the machine's own network are the same addresses. The
scenario routes an egress machine explicitly and marks the rest unreachable, so
nothing leaked — but that guard was carrying the whole weight of a collision
nobody chose, and a guard is a bad place for that.
10.99.1.0/24 is still RFC 1918, so the bed still models a home LAN behind an
access point. It is simply far from what this kind of machine already has:
192.168.1 is the LAN, 172.16-31 and 192.168.16-95 are container bridges, and
10.10/10.42/10.208 are a tunnel, the mesh overlay and the virtualisation daemon.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
0056 was already 'the authority is the control plane, not a database'. The
routing record was renumbered where it lives; these citations pointed at the
wrong decision, which is worse than pointing at none.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Seven catalogue modules — photos, photos-eef, photos-filip, invoicing, novox.be,
de-spiegel, amqp-email-forwarder — name `registry-api.…/novox/…:latest`. That is a TAG,
which ADR 0006 forbids and mesh-host refuses. It has never shown, because the lab's
registry rewrote every reference to a digest it had assigned, tag or not: the fiction
was not only serving the images, it was silently pinning them.
There is nothing to pin them with now. Asserting here would take whole-mesh-full down in
`before()`, before the overlay it exists to prove; the useful outcome is that each of
those modules fails to apply on the node that carries it, saying exactly why, while the
rest of the bed runs. So the reference passes through and the harness says so out loud.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Twenty-eight integration tests each carried their own copy of the same two helpers,
which pointed a manifest and the substrate bundle at whatever the lab's registry had
assigned. They now share two in the harness, and the difference is the point: ours is
rewritten to the ID the machine holds it under, and everything else is left exactly as
written so the machine pulls it.
**The substrate bundle is where the fiction was most load-bearing.** mesh-host's
`examples/substrate-first-node.lock` pins all three of its images at
`192.0.2.250:5000/…`, which is the address the lab's registry served from — it was
written for a target, and the target was the lab. Two of those are ordinary third-party
images and become the digests mesh-catalog's own postgres and lavinmq modules pin, so
the substrate's store and broker are literally the images the mesh runs. mesh-control
exists in no registry at all and becomes the ID the machine was handed. **The bundle
itself should be fixed in mesh-host and this substitution deleted with it.**
Beds that wrote a manifest by hand named an image by repository and let the rewrite
supply a digest. There is nothing to supply one now, so `onTheMachine` refuses an
unpinned reference and hands back the digest the catalogue pins — a bed runs the image
the mesh ships, and a bed that drifts from the catalogue is testing a different
postgres.
Three beds took a third-party image out of the raised list, which no longer contains
one: certificates (pebble), objectstore (minio and its client) and provisioner
(postgres) now name theirs and pull it. builds and mesh publish into the MESH's own
artifact store — the `registry` module's image, on the node, on 5000 — rather than into
scenery the lab raised. That is a different claim, and only one of them exists in
production.
New unit tests cover what a full raise would otherwise be the only way to check: the
routes an egress machine gets (that its gateway is still the path to the rest of the
scenario, that a range with no path is unreachable rather than leaked to the uplink,
that each family gets its own next hop), which machine is handed which of our images,
and the `images:` rule that refuses a third-party entry. The "shipped scenarios are
valid" test now loads every scenario rather than two of them.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Every scenario that places a container runtime gives each of its machines
`egress: true` — a node that runs modules pulls images from the internet, which is what
a node does. The underlay-only scenarios (bootstrap-single, behind-nat,
segmented-and-unforwardable, the-ordinary-shape, two-on-a-segment) stay sealed on
purpose: an extra NIC would change the very reachability they are asserting about.
`images:` keeps only the mesh's own — 55 third-party entries leave whole-mesh-full
alone, and the machine fetches them itself by the digest its module.json already pins.
The whole-mesh beds also say per machine which of ours they get: novox the substrate
control plane and its own fifteen runtimes, ace its twenty-four, the two workstations
one each. That is not a lab economy. An operator's workstation holds the images its own
modules need, and giving these two the union would put some thirty gigabytes onto a
thirty-gigabyte disk.
bootstrap-with-registry.yml is deleted. It existed only to demonstrate the lab's
registry, nothing referenced it, and there is nothing left for it to demonstrate.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The lab raised a `registry` VM, pushed ~73 images into it from the workstation, and
rewrote every manifest reference — third-party ones included — to point at it. No
production mesh has such a thing. So every bed proved that a machine could fetch an
image from a registry that exists nowhere else, and the bootstrap problems that only
appear when a machine has to fetch for itself went unfound.
What replaces it is the two things that are true in the world:
**Public images come from the public internet.** mesh-lab already created a NAT'd
uplink for exactly this and attached it to any machine declaring `egress`; no scenario
ever declared it. They do now, and third-party references are left exactly as the
catalogue writes them.
**The mesh's own images have no registry and never will.** mesh-control, mesh-builder,
mesh-route-proxy and the per-module runtimes are built from source and exist in no
registry. A machine gets them the way an operator's machine does — they are built here
and loaded onto it — and is then named by the digest of its own image configuration,
which mesh-host now accepts as "an image this machine already holds".
`images:` therefore means only *ours*, and a third-party entry is refused rather than
quietly loaded: otherwise the fiction returns one convenient line at a time. It is
per-machine as well, because "everything, everywhere" was never a description of
anything real — handing whole-mesh-full's union to its two 30GiB workstations would
fill the disk with runtimes nothing on them will start.
**The uplink and the declared gateway would have fought, silently.** A gateway container
and the transit router reach the scenario and nothing else; a default route through
either is a black hole for anything outside, and it beats the uplink's DHCP route on
metric. So a machine with egress states the scenario's ranges explicitly — through the
same gateway or transit it would have defaulted to, so the overlay-across-NAT path is
unchanged — and leaves the default to the uplink. A range with no path inside the
scenario becomes `unreachable` rather than falling through: 192.168.1.0/24 is an
ordinary private range in fact, and letting it escape would put scenario traffic on
whatever network the workstation is sitting on. `scenarioRoutesFor` is pure and tested,
because a decision only a full raise could check is one nobody checks.
The registry-reachability check the raise gained earlier is kept, pointed at the real
thing: every machine with egress must resolve a name and reach the internet before the
raise says it finished. Same failure it was written for — a raise that returns, an apply
that dies on its first pull, an instance left a bare shell — now guarding the path that
actually carries.
The base image's trust of the documentation ranges as plain-HTTP registries STAYS. It
was never only for the lab's registry: the mesh has one of its own, the `registry`
module, serving artifacts to the whole mesh over plain HTTP from whatever node runs it.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The control plane's image is FROM scratch and runs as 65534, and docker cp keeps
the mode a file had outside — openssl writes a private key 0600 root-owned, so
the copy landed unreadable, secret accept failed with permission denied, and the
CA crash-looped on a root it never got. Chowning it inside the container is not
available: there is no shell in there to do it with.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The bed set no node a `public-domain` and assigned no `acme-ca` provider, so it
was testing a mesh the design no longer describes — and going green while doing
it, which is the worse half.
**No public domain means no route.** A module now contributes a `label` and
nothing else; the mesh joins it to the node's public domain, and a label with no
domain to join composes to nothing at all. Every routed module on this bed was
therefore unreachable by name, silently, and no assertion noticed. novox now
carries `novox.incus` and ace `zurag.incus` — `.incus`, because this repository's
beds name nothing routable. The workstations carry none, which is also the design
being exercised: a node that does not face outward has no public domain.
**No acme-ca provider means no proxy.** route-proxy requires one, so without a
provider it is unresolvable and takes every routed module with it. step-ca is
assigned on the anchor, at mesh scope, and given an operator root — made with
openssl on the anchor and handed over through the real `secret accept` path,
because the mesh cannot invent a PEM and the random bytes it makes for an
own-secret nobody supplied would leave the CA crash-looping on a root key that is
not a key.
**What is asserted is the half that is decided and cheap**: that each routed
module's name composes to `<label>.<public-domain>` — read from the proxy's own
received-routes file, the mesh's answer on the machine rather than this test's
arithmetic checked against itself — with `@` composing to the bare domain, and
that the proxy answers for one of them over HTTP.
**What is NOT asserted is issuance.** Whether route-proxy obtains a certificate
from step-ca over ACME depends on mesh-control fixes landing as this is written,
and a bed that gated on them would report somebody else's in-flight work as its
own failure. step-ca is listed as a reported gap for the same reason.
The substrate apply also retries up to three times. `raise` now refuses to return
until every machine can fetch a manifest from the scenario registry, so the first
attempt should be the only one; a pull is simply the one step here that can fail
for a reason that goes away by itself, and the cost of not retrying was a whole
raise left as a bare shell.
Typechecks; not run end-to-end — see the ADR 0056 section for what is expected to
fail until the issuance path is fixed.
The first whole-mesh raise of the ADR 0056 code died on the anchor's substrate
apply: the image pulls failed, the anchor never came up, no node could enrol,
and the instance was left a bare shell — VMs and a registry, no substrate. The
identical apply, run by hand once the registry was warm, succeeded immediately.
`raiseRegistry` proves the wrong thing. It curls `localhost:5000` from inside
the registry's OWN machine, which says the registry process is up and holds the
blobs, and says nothing about the path anybody else uses: across a segment, and
for the home nodes through a NAT gateway whose default route and firewall are
applied two steps LATER. So "serving" was reported on evidence that excluded the
network, and the caller — which pins every image in the substrate bundle to that
registry — was handed a fact it could not rely on.
So the check moves to where it means something. After the routes and the
firewalls, before the minutes spent placing, each machine is asked for `/v2/` and
for one stocked manifest BY DIGEST, at the address it will pin, over the network
it will use. That is the pair of requests a pull begins with, from the same
place. Layers are not fetched: every digest was already read back inside the
registry machine, so what is in question here is the path, not the content.
Verified by typecheck and the unit suite (136 pass), and by confirming against a
standing four-node instance that `curl` exists in the machines and that both
segments — including a home node through the gateway — answer 200 for the
registry's `/v2/`. The ordering itself is unverified in a live raise from cold,
which takes hours.
ADR 0056 made `acme-ca` a requirement of route-proxy, and step-ca is what
answers it. A scenario that does not stock the image cannot run the CA, the
proxy does not resolve, and every routed module on the mesh goes with it.
This was carried as an uncommitted edit through the first ADR 0056 raise. Kept,
because it is right, and committed, because a fix that lives in somebody's
working tree is a fix the next raise does not have.
Rewrite the flat three-node whole-mesh-full (separate anchor, one public segment)
into production's real shape: two segments and one access point. novox sits on
the routable `hosting` segment and IS the anchor — it runs the substrate, its own
service set, the overlay hub and public ingress; there is no separate anchor node.
ace, shanks and g14 sit on the household `home` segment behind a NAT gateway,
reachable from outside only through what they dial out to.
The bed drives, and verifies, the thing the flat beds never could: the WireGuard
overlay forming ACROSS the access point — a home node dialling novox's public hub
endpoint out through the gateway's masquerade, the handshake completing through the
NAT, the keepalive holding the hole open. Phase A proves it (handshake state + a
ping over the overlay) before any heavy module lands; Phase B converges both server
sets. With MESH_LAB_KEEP the instance is raised under a fixed id and left standing.
Collapsing the substrate onto novox exposed real facts the separate-anchor beds
never hit, fixed here:
- the substrate bundle advertises the broker at 192.0.2.10 (the old anchor); a
token carries that verbatim as the endpoint a node dials, so with the substrate
on novox it must be novox's own public address. Rewritten at apply (the cert is
fingerprint-pinned, not hostname-checked, so only the address needs correcting).
- the two provider host-port collisions with the co-located substrate: postgres
5432 vs the store's 127.0.0.1:5432, lavinmq 5672 vs the broker's 127.0.0.1:5672.
Both provider host publishes are remapped off the substrate's ports.
And a lab limitation this first large-union bed exposed: the image registry VM took
the profile's default `dir` pool and a ~10GiB root, which the ~28GiB union of both
server sets overflows ("no space left on device"). raiseRegistry now places the
registry on the scenario's copy-on-write pool with a sized (default 80GiB, thin)
root disk, MESH_LAB_REGISTRY_DISK overridable.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Re-runs the capstone from main after the dry-run fixes merged.
fail2ban: added to the novox set. The capability fix (intrusion-prevention ->
firewall) makes it HOSTABLE — it is now assigned, not refused — which is the
gate. Its service reaching active is a host concern the offline lab cannot meet
(the VM ships nftables but not fail2ban, and the isolated segment has no route to
the package mirror, so pacman cannot fetch it), so fail2ban joins GAPS_NOVOX: its
failed package resource is tolerated like firewall's oneshot nftables.service.
7 credential sidecars: before the push, a FAKE app credential is delivered for
each (plex/bazarr/ombi/home-assistant/nzbget/qbittorrent on ace, umami on novox)
through the real operator path — `secret accept <node> <module> <name> --from`.
The bed asserts each sidecar advances PAST its old "no credential" crash (it reads
the delivered value); app-auth failure against the real app with a bogus value is
expected and not gated.
Result: SUITE_EXIT=0. Both node-plans converge on one substrate (novox 13/13
core, ace 17/17 core), fail2ban hostable, all 7 sidecars past their crash.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Combine the novox (17-module) and ace (24-module) sets on ONE substrate and
prove both node-plans converge together. anchor runs the substrate only;
novox and ace each run their own self-contained set (own postgres/redis), so
nothing crosses a node boundary except enrolment and the shared broker/store.
The four modules both nodes run (postgres, redis, mssql, portainer) are added
once and assigned to each node, each getting its own per-node broker account.
An overlay is placed across all three nodes.
Proven green: both nodes converge together on the one substrate. ace reaches
applied+current with all 17 of its CORE up (and letta too this run); novox
reaches all 13 CORE up with its only failed resource the known firewall.load
oneshot gap. The two node-plans share one broker without collision — distinct
novox-<mod> and ace-<mod> accounts for the modules both run. No new cross-node
bug (overlay/DNS/identity/port) surfaced; ports are per-VM and the sets are
node-self-contained. Tolerates the same nine credential-sidecar gaps and
firewall's nftables.service oneshot documented in the per-server beds.
Resource envelope: 3 VMs (anchor 4GiB, novox 16GiB, ace 18GiB) + registry
scenery, ~79 union images (~35GB) stocked to one registry VM and pulled
concurrently by both nodes; fit within 125GiB host RAM and the 180GiB lab pool.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Install the real ace server's converted service set (24 modules) together on
one node behind the substrate — sibling of the whole-mesh-novox bed, the
media/home-automation half. Loads each committed module.json from
mesh-catalog, rewrites image refs to the scenario registry's digests, remaps
the co-located host-port collisions (qbittorrent/searxng/unifi :8080,
nzbget/unifi :6789), and pre-creates the ADR-0051 operator-owned media
library dirs under /services/media so the media stack's `accesses` resolve.
Proven green: the whole 24-module set RESOLVES and applies (214 resources,
node applied+current) — the ADR-0051 shared-dir `accesses` mechanism works
cleanly across eight co-accessing media modules. The CORE 17 converge whole:
postgres/redis/mssql, sonarr/radarr/lidarr/jackett/tautulli/bookshelf,
mosquitto/influxdb/grafana/baserow/nodered/searxng/unifi/portainer.
Reported as escalated gaps (do not gate green): six tool-runtime sidecars
crash-loop because the committed manifest does not wire the app credential
they need (plex MESH_PLEX_TOKEN, bazarr MESH_BAZARR_API_KEY, nzbget
MESH_NZBGET_URL/PASSWORD, qbittorrent MESH_QBITTORRENT_URL/PASSWORD, ombi
MESH_OMBI_API_KEY, home-assistant MESH_HOMEASSISTANT_TOKEN) — the umami/photos
class from novox; each server is up, only the sidecar is down. sonarr/radarr/
lidarr/jackett/tautulli self-configure from the app's config file and their
runtimes come up. letta's app has a first-boot postgres migration race
(pgvector the deeper blocker, per two-node-db).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Install the real novox server's converted service set together on one node
behind the substrate — the whole-catalogue install this rebuild never ran.
The bed loads each committed module.json from mesh-catalog (no hand-written
manifests), rewrites image refs to the scenario registry's digests, and
remaps the co-located host-port collisions (nextcloud/invoicing/route-proxy
:80, minio/invoicing :9000, gitea/umami :3000).
Proven green: the whole set of 17 modules RESOLVES and applies (191
resources); the CORE 13 converge whole — all five providers (postgres,
redis, minio, mongodb, mssql) plus keycloak, gitea, nextcloud and invoicing
reaching their providers and staying up, plus portainer, verdaccio, registry
and route-proxy.
Reported as escalated gaps (do not gate green): fail2ban (declares
capability intrusion-prevention that no host detector provides, and an
unappliable assignment blocks whole-node resolution), umami/photos/mailu
(catalog manifests do not wire the runtime/app env the images need; photos'
server image is an alpine placeholder), and firewall (nftables.service is a
oneshot that exits, but the module declares state running so mesh-host marks
it failed).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A single-node VM bed: an ollama provider and a local-model consumer are
assigned; the resolver answers the consumer's model-access with the local node
(no licence demanded), the consumer's openai.env is templated with the served
endpoint, and a request to it reaches the running model server. Proves the
node-answer of model-access end to end (ollama on host network, keyless).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A single-node VM bed: the operator sets an API key on an openai licence, the
mesh seals it to the consumer, the host unseals and mounts it, and the consumer
writes it as OPENAI_API_KEY (env + Codex auth.json). Asserts the written key
equals the one set — the other shape ADR 0050 defines, and the ADR 0054 branch
where a static-key vendor records no usage. Green on the first run.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A VM bed: a postgres provider and the model-usage consumer on one node, the
substrate on the other. A usage event injected into the mesh is upserted into
model-usage's provisioned store, asserted at both grains, latest-per-key, and
in the clear.
Also, in build-module-runtime.sh, add migrate/index.ts and pg.d.ts to the
compiled entrypoint set so a module may carry a run-once entry and an ambient
type declaration (model-usage uses the latter for the pg driver).
The bed surfaced and drove several fixes elsewhere: a short module slug for the
S3-key identity bound (ADR 0049), host-network containers getting the mesh's
names (mesh-control), and injecting the event from a publisher rather than the
pure-consumer store (its account has no publish right by design).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Two harness fixes the green end-to-end run needed:
- build-module-runtime.sh installs a module's non-@novox runtime deps under
/app/modules/<module>/node_modules, so a module can carry a private dependency
(the anthropic-manager seals with tweetnacl-sealedbox-js). The shared tree still
answers @novox/* and common packages. A no-op for modules that declare none.
- stageIntoControl chmods the manager's 0600 adopt/refresh outputs to 0644 on the
anchor host before docker cp, so the distroless mesh-control (non-root, no chmod)
can read the staged file. What is staged is a sealed box or the access token,
never a cleartext refresh token.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The bed follows the reworked flow: the manager module seals the refresh token to the node's
PUBLIC key, the HOST unseals it and mounts the cleartext at the manager's bound path, and the
refresh reads that cleartext -- no fake node key pair is mounted any more, the host uses its
own real sealing key.
- the manager is a model-access holder deployed first, so its bound facts (carrying the node
public key) are delivered; the consumer is added only once an access token exists to seal.
- adopt reads the node public key from the bound facts; the test asserts the host mounts the
cleartext refresh token for the manager, and that it reaches nowhere on the consuming node.
- the refresh_grant assertion reads { sealed, manager_key }.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A lab bed for Phase C of model-access (ADR 0050), OAuth endpoint stubbed.
It drives the real runtime images through the whole flow: the manager
seals a refresh token at rest and opens it on the manager node alone,
mesh-control is handed only the access token and an opaque re-sealed
envelope via licence submit-refresh, and the consumer writes an
access-token-only credential. Asserts the refresh token -- original and
rotated -- is nowhere on the consuming node and only ciphertext in the
control plane's database.
build-module-runtime.sh also compiles adopt/refresh/apply/usage
entrypoints. Stubbed and flagged: the vendor endpoint, the manager node's
private key (mounted; a host capability to deliver it does not exist
today), and the submit transport (the test invokes the CLI on the
manager's output).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Two-node: substrate broker on anchor, lavinmq provider + amqp-ping consumer on laptop. The
consumer gets its scoped vhost+user, connects, and round-trips a message. Requires the
mesh-control require-only-mint fix. Diagnostic removed now it's green. SUITE_EXIT=0.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The lavinmq AMQP provider comes up and serves, but its receives file has given:[] — the
consumer amqp-ping (requires amqp, contributes nothing, as a parameterless provision like
redis-cache takes no per-consumer payload) is never minted a credential, so the provisioner
creates no vhost. Diagnostic in the test dumps the empty grants file + the provisioner log.
This is the resolve/plan mint path (grantsFor -> SecretsFrom), ADR 0048 territory, and it
likely affects redis-cache consumers the same way. Preserved for a focused fix; not merged.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A lavinmq provider and an amqp-ping consumer ride laptop while the
substrate's own broker owns 5672 on anchor — the twin of two-node-db.
lavinmq is the mesh's control broker AND a user-facing capability, so a
provider must publish 5672 for its consumers and cannot share a node
with the control broker that already owns it; the split unblocks the
chain single-node.
The bed proves, layered: the run-once bootstrap computed the admin hash
and wrote the broker config before the broker started (ADR 0052); the
service and both runtimes are up and stable; each module got its scoped
broker account on the substrate broker; the provisioner created the
consumer's vhost AND user, both named for the derived login; and the
consumer connected to that vhost with the mesh-minted password and
round-tripped a message. The consumer uses ${bound:amqp:as} for user
and vhost both, and the provider's serves carries the port.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A scenario and integration test assign route-proxy (provider) and hello-web
(consumer) on one node, then assert a request to the consumer's name -- sent to
the proxy -- is forwarded to the workload and returns its answer, and that
unassigning the consumer withdraws the route so the same request stops working
(the proxy replaces its table rather than merging). Modeled on
mesh-grant-end-to-end and schedule-tick: module add, assign, one push, settled,
with no module issue (route-proxy needs no scoped account).
build-route-proxy-image.sh compiles the Go proxy from
mesh-control/examples/route-proxy into mesh-route-proxy:development for the
scenario to stock. This bed proves route-forwarding over plain HTTP;
public-ACME TLS is proven separately by certificates.test.ts against a real
ACME server.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The bed that proves the scheduled-container primitive end to end. schedtest
is the thinnest carrier of ADR 0053: one container marked
schedule: "* * * * *" that appends a timestamp to a mounted data dir each
time the host fires it -- no service, no listener, no provisioner, no
runtime, no tools, no events.
The three claims it proves, from the ADR's "How each claim is checked":
installing the schedule leaves the node current WITHOUT a run (baseline
captured right after settled, the deliberate inversion of run-once); the
container fires on its cadence (a line beyond the baseline within ~150s);
and it recurs (a second line on the next minute -- cadence, not a one-shot).
schedtest serves and consumes nothing and carries no runtime, so it is not
issued a broker account: module add -> assign -> one push is the whole
sequence, no module issue. The tick image is a bare alpine served by the
scenario's registry by digest.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The GREEN multi-node regression bed that proves the DB-consumer gate: substrate/control on
one node, postgres+redis providers and baserow+letta consumers on another, each consumer
getting its own credential and its own mesh-named database across the overlay. Requires the
mesh-control provider-seal-key fix and the mesh-catalog db-name fix.
Includes a general lab capability: a machine 'disk' field sizing the VM root disk (a broad
install exhausts the pool default and the host fails mid-apply with 'no space left on
device'). The bed sets 60GiB.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Sixth green regression bed. Same shape as tools-gitlab: a runtime-only module comes
up under the mesh, serves its full tool surface with no valid credentials (the Servarr
lesson), stays up, binds its serve queues, and gets its scoped broker account.
SUITE_EXIT=0.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Prove gitlab — the exemplar tools-only, outbound-only external-SaaS
integration — installs: a scenario assigning gitlab to one node, and a test
asserting the mesh-runtime-gitlab container comes up and stays up, logs
[mesh-tools] serving 23 tool(s), binds its serve queues on the broker, and
gets its own scoped account — all with NO valid GitLab token, the case the
Servarr lesson is about.
No gitlab arm is needed in build-module-runtime.sh: gitlab speaks HTTP and
needs no extra CLI in the image.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A bed that assigns mosquitto and asserts the run-once step seeded dynsec
before the broker: the bootstrap ran to completion (not left running), the
seed is on disk owned by the broker's uid, the broker is up and stable
(it crash-loops against an unseeded store, so a stable broker is the proof),
and the node reached current. On top, the seeded admin authenticates over
MQTT and the provisioner grants a scoped client a consumer connects with.
build-module-runtime.sh gains a mosquitto arm (install mosquitto_ctrl from
the mosquitto package — it is not in mosquitto-clients on bookworm, and a
musl binary from eclipse-mosquitto would not load) and compiles the module's
bootstrap/index.ts entrypoint.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
sonarr and radarr both access /services/media/downloads — the exact duplicate path
the resolver refused before novox/hq ADR 0051 (04-ISSUES/036, 012). Each now declares
it as an `access`, not a `directory` resource, so the pair co-resolves and one push
configures both. The operator provides the shared media dirs before apply (the host
refuses an absent access); the bed creates them after enrol and before the push.
Proves: the push is not refused, the node converges once, both modules' server and
runtime containers are up, and both server containers mount the same operator-owned
spool.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Adds the mongodb runtime CLI (mongosh) to build-module-runtime.sh, a
catalogue-apps scenario, and its install test. Proven so far: the ADR-0054 slug
applies and the mesh accepts the push (mongodb consumer identity mesh_anchor_mongo
fits). NOT green: the node applies but never reaches 'current' within 1200s — a
persistent reconcile divergence (applied-but-never-current, no crash), likely a
module declaring a resource its container mutates (issue-011 class). Needs live
VM inspection to name the module. Not merged.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Raises the first-node substrate and assigns postgres, minio, redis and
plex to one anchor in a single push, proving they resolve and come up
together on one node. postgres and minio each get a consumer that
connects with a real granted credential.
redis follows the corrected provider contract (ADR 0048, issue 032): its
runtime reconciles the contributions the mesh delivers at MESH_RECEIVES
and creates each consumer's ACL user with the mesh-minted password,
sealing nothing — no MESH_SEAL_KEY, no *.grant.json/*.credential path.
The provisioning proof authenticates as the consumer with the mesh's
password (PONG), matching the green provider-uses-mesh-credential bed.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Issue 010 fixed: bucketuser declares slug `bkt`, so its identity mesh_anchor_bkt (15)
fits an S3 access key where mesh_anchor_bucketuser (22) did not. Unskips the test.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Mirrors the postgres bed for minio: a provider (runtime carries mc) + a consumer
requiring s3-bucket, proving the consumer reaches its bucket with the access key and
secret the mesh delivered. It surfaced a real limit: the mesh derives `as` =
mesh_<node>_<module> (22 chars), and an S3 access key is capped at 20, so minio refuses
the service account. The test is correct and skipped pending 04-ISSUES/010, not worked
around.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Assigns a postgres provider (its runtime carries psql) and a module that requires
postgres-database; the mesh mints one password, postgres's provisioner creates a role
and database under the mesh's login with it, and the consumer connects to its database
with the delivered credential (a password-checked connection) — select 1. Nothing placed
by the test. The postgres half of the per-backend provider proof (ADR 0052/0053).
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Assigns a redis provider and a module that requires redis-cache; the mesh mints one
password, seals a copy to each end, writes redis its contributions and the consumer
its bound file, and the host unseals each side. redis's provisioner creates the ACL
user under the mesh's login with the mesh's password, and the consumer's delivered
credential authenticates (PONG). Nothing is placed by the test — the provider/consumer
contract (ADR 0053) working as one thing, no shared key anywhere.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
build-module-runtime.sh adds psql to the postgres image and mc to the minio image
(their clients shell out to those). provider-on-backend-network asserts redis's
runtime, on the backend's private network, binds the broker via NAT and provisions
a consumer with the mesh's credential — the shape the committed provider manifests use.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Assigns redis as a provider, puts the contributions and unsealed password the mesh
would deliver in its receives path, and authenticates as the consumer with the mesh's
password — PONG proves the login was created with exactly that password (a
self-generated one answers WRONGPASS), with MESH_SEAL_KEY set nowhere.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Assigns grafana configured by settings, changes the token, pushes again, and
asserts the container was replaced (new id) and the rendered config carries the new
value. Builds the host from source, since the behaviour under test is the host's.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
assigned-sonarr proves the Servarr detection path: the runtime discovers its API
key from the app's config.xml and serves its tools. assigned-grafana proves the
settings path: the operator states URL and token as settings, the mesh merges them
into the module's config file, and the runtime serves from that with nothing in the
manifest. Both green.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
build-module-runtime.sh generalises the audit-logger runtime image to any module
(mesh-tools + sdk + the module's dist, entrypoints for tools/events/provisioner).
Two scenarios and two tests: assigned-plex proves a tools+events module serves its
tools over a mesh-issued scoped account; assigned-redis proves a provider's runtime
serves tools AND runs its provisioner in the same broker-bound process, provisioning
a grant and emitting its lifecycle event. Both green.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
test/integration/assigned-audit.test.ts raises a node into a mesh, assigns it
the audit-logger through the control plane, and asserts the mesh delivered a
scoped amqps account (not the broker's own), the host ran the container, and an
emitted event reached the trail — the delivered credential authenticating is
the proof. scenarios/audit-node.yml is the lean single-node bed that stocks the
runtime image. Passes 1/1 against the real lab.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Adds test/integration/events.test.ts: raises first-node (which raises the
broker as tier-1 substrate), runs the runtime+audit-logger against that
broker, emits a module and a node event, and asserts they reach the trail
with their metadata read back from ADR 0047 headers (a pure body), plus that
the durable per-consumer queue and mesh.events.dead exchange exist on the
raised broker — asked of the broker, not assumed.
scripts/build-runtime-image.sh builds the self-contained runtime image
(mesh-tools + vendored sdk + audit-logger) it runs, saved to a tar for
MESH_LAB_RUNTIME. The events path itself is verified; the incus raise is the
part a lab run exercises.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
With caught-up finally an equality, the last red test turned out to be
telling the truth about something real: declarations queue, the machine
applies them one at a time at half a minute each, and by the twenty-
fifth test it is minutes behind the latest push. 240 seconds was not a
generous bound on one apply — it was an accidental bound on the whole
backlog.
Doubled rather than tuned, and the real remedy filed instead: a machine
asked to be five successive things should become the last one, which
is a decision about the link rather than about this timeout
(novox/hq 04-ISSUES/031).
The timestamp comparison lost the race between one test's closing push
and the next test's opening one: the old apply's report landed newer
than the new send and settled() passed for a declaration the machine
had not read. The report names its declaration now, the mesh says
whether it is the current one, and this reads the answer instead of
inferring it.
The provisioner makes the role and then the database; a poll that
waited for the first and checked the second once was racing the gap
between two statements, and lost it once, eighteen seconds into a run.
The cache test's provisioner could not resolve the store's name, which
usually means the store's container never registered it — and the
diagnostics showed only the provisioner's side of that conversation. A
grant that never arrives now prints the container states, the store's
log and the provisioner's, so the next failure names the half that
actually fell over.
Eleven modules planned as one set on one machine, which is what a real
node looks like. Two are left out by name rather than silently: the
mesh under test already runs a module called registry and an adopted
workload called umami, and adding the catalogue's manifests would
replace the records of things that are live and assigned — the adopted
umami would suddenly require a database it never asked for.
settled() returned the moment a declaration was current, because the
sent digest is recorded at send — so both new tests asserted on a
machine still applying, and found containers not created yet and
bindings not written. The eternal-waiting fault had been standing in
front of this gap the whole time; fixing it is what let the tests get
far enough to fall in.
Caught up now means the machine's own last report is newer than what
was sent to it — two timestamps the mesh recorded itself, read from the
`reported` section status now carries. The residual latency between
"reported" and "every container answers" stays with the tests' own
polls, where it always was.
A consumer contributes a key prefix and gets an ACL user; the test is
that the grant means exactly what the manifest said, in both
directions: its own keys usable, anyone else's refused by the store
itself, and the flush a tenant must never have refused with them.
Waited for through the store rather than through logs: the user list,
asked with the password the host wrote into the server's own conf file
on the machine — nothing invented, both ends reading what the mesh
delivered.
And the forge is asked on the port the mesh assigned, not the one the
module declared. The old curl aimed at 3000, which was right until
ADR 0038 moved the machine side — a latent break that would have fired
on the first run to get past the settling that used to fail first.
The scenario stocks redis and its provisioner, and the rebuild builds
the provisioner image with the others.
whatWasTested read the repositories when the run ended, so a commit
landing during the twenty minutes a suite takes was recorded as tested
without ever being in the binaries. It happened: one receipt named a
commit made mid-run, and the verdict it carried belonged to an older
tree.
The heads are read once, right after the build, and carried to the
receipt. A verdict is only worth something attributed to one exact
state, which is the receipt's whole reason to exist.
A segment named "uplink" is refused. The lab claims that name for the
NAT bridge behind `egress: true`, and a scenario wearing it first would
have its egress machines silently attached to an isolated bridge — a
declared key doing nothing, which is the fault this repo exists to
refuse, in the repo that refuses it.
settled() parses inside the try. A truncated status from a struggling
machine was the one shape of bad answer that still threw out of the
wait, and the likeliest moment for one is exactly the machine the poll
is watching. Malformed now counts as "could not ask", like the exec
that times out.
And a sentence on the uplink's UseDNS saying its inertness is
load-bearing: it matters only where systemd-resolved runs, and on a
machine whose modules own resolv.conf the uplink must not outvote the
resolver a scenario is testing.
Three changes, found by one failing test.
The forge failed three runs in a row as "status hangs", and it was
diagnosed twice as contention — real defects, fixed, and not the cause.
The heartbeats told the truth in the end: every exec on anchor crawled
from 15s to 105s, because eleven containers plus a database pull were
running in a 1GiB machine. Starvation presents as whatever you were
doing when the page-outs start, which is why it wore two other bugs'
clothes first.
So machine size is now the scenario's to declare — memory and cpus per
machine, default unchanged. The anchor that carries the whole substrate
is bigger than the laptop that joins it, and the comment on the
scenario says why in terms of what lands there.
`egress: true` gives a machine one extra interface on a lab-supplied
NAT network, addressed by DHCP because the one address a scenario has
no business choosing is on the host's side of the fence. Declared
per machine and off by default: a closed scenario stays the rule
(novox/hq ADR 0016), and the exception exists because a first node
fetches its images before any mesh can serve them — which is now the
tested path (04-ISSUES/029), and a lab that can never reach upstream
cannot prove the bootstrap it exists to prove. The uplink route is
metric-4096, so it never shadows a route the scenario declared. A
detached machine declaring egress is refused, not ignored.
And settled() treats a poll that threw as a poll that missed. An exec
timeout at minute four of a wait is "could not ask", not a verdict on
the machine.
A mesh that has just bootstrapped cannot build the module that gives it
an artifact store: building publishes to the store, and the builder will
not start without one (novox/hq 04-ISSUES/029).
This test built it and passed, because the scenario's registry was
already standing to receive the push — which is precisely why a real
first mesh would have hit this and the lab never did. A stand-in for
Docker Hub was quietly also standing in for the thing under test.
So the module now names its image by digest, the way the bundle names
the three a first node starts from, and is added as a manifest rather
than built. That is the only path open to a real first mesh, so it is
the path this walks.
Its skip on MESH_LAB_BUILDER goes with it. Nothing in the test needs a
builder any more, and a skip that names a thing the test does not use
sends the next person to look in the wrong place.
The canary walked one path on one machine — a mesh comes up, a module
lands, a consumer gets a credential — and stopped the run if it broke.
That path is exactly what the first three tests of the long run walk,
and the long run finishes them about 160 seconds in.
So the gate cost a whole scenario on every passing run to save roughly
45 seconds on a failing one. A scenario is three machines, one of them a
registry that boots a kernel in order to serve files, which is where the
two minutes went.
The test file stays and still runs when it is named. What is gone is
raising it on the way to everything else.
Measured rather than argued: the canary's scenario took 116s of which
60s was standing up a registry, and the run reached the same assertions
without it.
`settled` used `must`, so a failed exec ended the wait as though the
machine had reported a failure. It had reported nothing: the control
plane is a container on the node being polled, and while that node
applies a declaration an exec into it can lose its stdout fifo to
containerd. The run then blamed the mesh for a question that missed.
Could not ask and asked, and the answer was bad are different facts, and
only the second is the machine's. A failed poll now keeps the reason and
tries again; the timeout reports whichever came last, so a control plane
that is genuinely unreachable still fails the test — with the reason
rather than with a stack trace.
Every five seconds rather than every two. Each poll is an exec into a
container on a machine that is busy applying, and thirty times a minute
was competing with the apply rather than observing it.
The forge test read `status` the instant `push` returned and concluded
the machine was fine. It was describing the apply before this one.
`push` sends and returns — it prints "sent N resource(s)" and the
machine applies afterwards. So every assertion made immediately after
one is racing it, and this race lost quietly: no failure reported, and
a container that did not exist yet read as a container that would never
exist.
`settled` asks the mesh, in its own terms: a node is caught up when it
is neither waiting for what it was sent nor wrong about what it applied
— the two questions `status` already answers, read as JSON so a test is
not parsing a report written for a person. A machine reporting a failure
ends the wait immediately rather than at the timeout, because it will
not become right by being waited for.
The container assertion now also prints the plan. A container missing
because the mesh never asked for it and one missing because the machine
could not make it are one sentence and two entirely different faults,
and the plan is what separates them.
The forge test pushed and then waited for a database login. When the
containers were never created at all, it reported "no login was created"
— true, and silent about why. Two hundred and thirty seconds spent
proving something downstream of the actual failure.
A push being accepted and an apply having worked are different facts,
and this test depends on the second. It now reads what the machine says
about itself, and whether a container exists, before it starts waiting —
and carries the host's own log into the failure either way.
The suite otherwise passed 24 of 25 on this run, which is the first time
the forge reached a clean attempt with nothing upstream blocking it.
A scenario is a closed address space: two raised from the same
declaration hold the same addresses and never meet, which is what lets
two run at once and why the lab talks to machines through the
hypervisor rather than over IP. Reaching in from outside breaks that, so
it is opt-in, one scenario at a time, and reversible.
`connect` takes an address on the scenario's public link and writes a
resolver rule answering everything under each machine's name.
`disconnect` gives both back. `connected` says what is true right now,
for somebody who cannot remember.
It refuses rather than guessing when more than one scenario is standing
— the failure being avoided is not an error but one scenario's traffic
arriving in another. It also refuses when a machine's name is already
answered here for something real, because connecting would point that
name at the lab, and the damage would land on the real thing.
Names answer with the segment address rather than the overlay one.
Inside the mesh a name gives a machine's private address; from here that
would need this workstation on the overlay, which is a much larger door.
The segment address reaches the same machine and the same ports, which
is what opening a board in a browser actually needs.
Proven against a live two-node scenario: registry.internal:5000/v2/
answered 200 from this workstation, and so did a wildcard name under the
same machine. Disconnect put the address back, stopped answering, and
left the real mesh's own names alone.
One thing measured rather than assumed: it restarts dnsmasq instead of
reloading it. A reload is SIGHUP, which re-reads the hosts file and
clears the cache but not the configuration — the rule was written, the
reload reported success, and nothing resolved. The daemon's start time
was nine days old afterwards.
Suggested by Jochen, and it paid for itself on its first run.
A suite that takes forty minutes is a suite you hear from once a day.
Every fault found today would have shown up in the first three minutes
of it — a module pinned to an image that does not exist, a consumer
given a password and no name to present with it, a credential file
nothing could read, a search for a password that read the password as an
option. The other thirty-seven minutes proved things that were already
working.
So this runs first, on one machine, with the three images the mesh needs
for itself. It walks one path: a mesh comes up, a module lands, and a
consumer gets a credential it can actually use — the name to present,
the address, the port, and a password only the host could put there.
Deliberately not a smaller copy of the full suite: that path is where
everything went wrong, and a canary checking many things shallowly is a
canary whose failure nobody can read.
`suite` runs it and stops if it dies, saying why rather than leaving
somebody to wonder what the missing thirty-seven minutes would have
said. Skipped when the caller named its own files.
It measured 164 seconds against forty-odd minutes, and failed three
times on its first run for one reason: applying the bundle raises a
control plane but does not tell it a machine exists. I had left out
enrolment, and the long suite would have taken forty minutes to say so.
**A password beginning with a dash broke the search for it.** The
credential test greps the machine's own files for the delivered
password; this run's password started `-S`, so grep read it as an option
and refused the whole invocation. The test compared the usage message
against "0" and reported the password as leaked. That is the worst way
for a search to fail — it says it found something. Fixed with `-e` and
`--`, which is what those exist for.
**The planning test could not redirect what the scenario does not
serve.** Rewriting an image reference only works for repositories the
scenario's registry actually holds, and the object store's provisioner
was not stocked — so that module kept its placeholder and the refusal
fired, correctly. It is stocked now, so all five are planned again. The
skip path stays for anything genuinely unserved, and says which module
and why: a planning test quietly covering four instead of five is the
false coverage this suite exists to prevent.
**The forge failed because of the one above it.** The planning test
threw before its cleanup could run, leaving a module assigned that
refused the next push, so no database container was ever created.
Yesterday's fix moved that cleanup where a failure cannot skip it — but
`after` still only unassigns what was assigned, so it now tracks what
actually got added rather than what was intended.
Two faults, both found by the guard that now refuses a placeholder
digest on its way to a machine.
The planning test added the manifests exactly as they sit on disk, which
includes an image the mesh builds — and that image has no digest until
it is built, so the file legitimately carries a placeholder. The test
was therefore planning something that could never run, which is the
whole complaint. It now points the references at this scenario's
registry first, exactly as the forge test does.
The second is worse and more ordinary. Its cleanup was the last
statement in the test body, so the first failure skipped it and left
five modules assigned. The next test's push was then refused by a module
this one had abandoned — a failure that reads as a fault in the test
that was working. Cleanup that only runs on success is not cleanup, so
it moved to `after`, where a failure cannot skip it.
Everything up to now stopped at composing a declaration. That proves the
control plane and the host agree, and proves nothing about whether the
thing described works — which is how five modules sat pinned to images
that did not exist while parsing and resolving perfectly.
The forge is the right one to run first. It needs a database from
another module, a password it did not choose, and a connection string it
could not have written itself: the address and port come from what the
database serves, the user name from what the mesh decided both ends
would call it. If any of that is wrong it cannot start, and nothing else
in this suite would notice.
The test checks the chain in the order it has to happen — the login
exists, the database it owns exists, the forge answers, and its log does
not say authentication failed. That last one matters: a forge that
started and could not reach its database would still answer on its port.
Rewriting image references is now shared rather than copied from the
bundle, which had the same problem first. A digest is not knowable until
something is built, and when it is, it belongs to whichever registry
served it — so the text says which image and the scenario says which
copy. Matching is on the repository, with a test that a repository
ending in another one is not half-replaced.
Also makes the planning test put the machine back. Tests here share one
mesh, so the five modules it assigned were inherited by whatever ran
next; harmless while nothing pushed, and not harmless now.
novox/hq 04-ISSUES/024. A run stalled for thirty-five minutes and said
nothing. The cause was a link systemd was still configuring, three
layers down inside a `docker load` blocked on a socket — and every one
of those layers knew what it was waiting for. None of them said so.
Three decisions, each doing work.
**Every external command is logged, at the three places that run one.**
Ninety-seven call sites reach a hypervisor or a container runtime
through three wrappers, so instrumenting the wrappers covers all of them
and nothing has to remember to log.
**A command still running says so while it runs.** A line before and a
line after tells you nothing until the after arrives, which is exactly
the case that matters. Anything outstanding past fifteen seconds reports
itself with how long it has been going. It is reported as still running,
not as stuck — which it is is not knowable from there, and a log that
calls a slow step a hang teaches people to ignore it.
**It goes to a file, written synchronously.** Node block-buffers stdout
when redirected and a test runner buffers it again, so a console log can
sit minutes behind. `appendFileSync` cannot lag.
Two things this found in itself while being written, both the same shape
as what it exists to catch:
A question that answers no is not a fault. Half the lab's commands are
questions — does this network exist, is the agent up yet — and they fail
constantly while a scenario comes up. Logging those as faults filled a
healthy run with ✗, which is how you end up ignoring ✗ when one is real.
They are recorded quietly now, and still recorded.
And `around` skipped its own wrapper when a step's level was below the
configured one — taking the failure line and the heartbeat with it. The
two things worth having at a low level were the two that vanished at
exactly the level somebody would use. The gate belongs in `write`.
Also unsilences the four call sites that passed a callback throwing
everything away, including the one the stall sat in, and tees `raise`'s
progress into the file whether or not a caller asked to see it — the
end-to-end test passed no callback, so the one run that mattered
reported not a single step.
The last failure of the run: `MESH_BROKER_FILE=/var/lib/mesh/builder/broker`
reported as "something secret-shaped, which the broker would see".
`/` is in the base64 alphabet, so any absolute path of 24 characters or
more matched the pattern meant to catch a sealed value. An absolute path
is a *reference* to a secret and naming one is the whole design — the
mesh delivers a credential as a file and a module says where.
Excluded explicitly rather than by loosening the pattern, and checked
both ways: a real sealed value and a base64 blob are still flagged, a
relative path still is, only an absolute path is passed over.
Worth the words in the comment. A check that fires on the right shape
for the wrong reason is worse than none — it is the one that gets
suppressed, and then it is not there when it is right.
It surfaced now because tests in this file share one mesh: the builder
was assigned by an earlier test and appears in this one's declaration.
Node block-buffers stdout to a file, so a run log can sit unchanged for
minutes while the run is fine. Read that way twice today — the second
time straight after fixing a real stall, which is the worst version of
it, because a buffering artifact then reads as the fix having failed.
The machines are the source of truth and answer immediately. Written
down with the two commands that settle it.
novox/hq 04-ISSUES/024. The registry machine had its address set with
`ip addr add`; every other machine gets a systemd-networkd unit. That
one difference stalled the lab indefinitely.
An address set by hand leaves networkd waiting to configure a link it
was never told about, so the link sits at `configuring` for ever.
`systemd-networkd-wait-online` has TimeoutStartUSec=infinity, so
`network-online.target` is never reached — and Docker is ordered after
it. `docker load` then blocked on a socket whose daemon was queued
behind a target that would never come.
Measured before and after on the same scenario: stuck with five pending
systemd jobs and `docker` inactive; now `enp5s0 configured`, `docker`
active, no jobs, and the whole raise completes in 87.5s.
The guess in the issue was wrong, and it was wrong in the usual way —
stocking had just been changed, so stocking looked guilty. Stocking
takes 34s and always did.
Two things that made this cost hours rather than minutes are fixed with
it. Placing an image now waits for the container runtime to answer and
refuses after 120s naming what systemd is waiting on, so a stall becomes
a failure that says why instead of three stacked timeouts totalling 35
minutes. And the end-to-end test passes `onProgress`, so a raise says
what step it is on — it printed nothing at all until it finished, which
is why 35 minutes of nothing read as a slow test.
Pointed at the repository root, the binary it writes is 12 MB of tracked
artifact. Said in the example rather than left to be discovered by a
On branch initialization
Your branch is ahead of 'origin/initialization' by 43 commits.
(use "git push" to publish your local commits)
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: README.md that looks wrong.
Nothing did. The sealed-placeholder substitution and the bound-value
substitution were each covered by unit tests in the repository that
performs them, and the two expressions that find the holes live in
different repositories — so both sides could agree with themselves and
disagree with each other, and the first thing to notice would be a
program connecting to a host called "${bound:postgres-database:at}".
So the consumer in the credential test now ships a configuration file
with four holes in it: the address and port from what the provider
serves, the name to present from what the mesh decided, and the password
sealed. The mesh fills the first three before sending, the host opens
the credential and fills the last on the machine, and the test reads the
file off the machine and checks that the password in it is the same one
the credential file holds — and that no ${ survived.
This is the only place those two mechanisms meet a real host.
The credential moved: the sealed password is a password alone, at
`.secret`, and `database.env` is now the connection keycloak could not
have written — address and port from what the provider serves, user name
from what the mesh decided both ends would call this consumer.
So the test asks for both, and for the seam between them: the password
is still a hole, the sealed value travels beside the file that needs it,
and no ${bound:...} survives as a value. That last one matters most —
a placeholder written through would be read as a hostname, and the
failure would name neither the module nor the mesh.
A run today had a control-plane image built that minute and a
provisioner image built the day before. The rotation test failed against
a real database and it looked exactly like the change under test being
wrong — the provisioner was creating logins by a naming rule that had
been replaced hours earlier.
It was the rebuild. It covered `make image` and the builder binary and
none of the three other image targets, all of which the suite runs.
This is the same fault the builder line was added for, one target along,
and the comment there already names the precedent: building one and not
the other is the eleven-hour-old binary. A rebuild that covers most of
what a run uses is worse than one that covers none, because the run that
follows it is believed.
The test names each target rather than counting them, because what goes
wrong is a target that exists and is not run, and a count would not
notice.
Two assertions in the full-mesh test encoded the old naming: the grant
file read back from the provider, and the PostgreSQL role the real
application logs in as. Both are named after the consumer now, and a
consumer is a module on a machine.
These are the two that matter most in this file — it is the only place
where a real application authenticates against a real database with a
password the mesh delivered and cannot read, so they are what would have
caught the naming going wrong end to end.
jschoubben