**A password beginning with a dash broke the search for it.** The
credential test greps the machine's own files for the delivered
password; this run's password started `-S`, so grep read it as an option
and refused the whole invocation. The test compared the usage message
against "0" and reported the password as leaked. That is the worst way
for a search to fail — it says it found something. Fixed with `-e` and
`--`, which is what those exist for.
**The planning test could not redirect what the scenario does not
serve.** Rewriting an image reference only works for repositories the
scenario's registry actually holds, and the object store's provisioner
was not stocked — so that module kept its placeholder and the refusal
fired, correctly. It is stocked now, so all five are planned again. The
skip path stays for anything genuinely unserved, and says which module
and why: a planning test quietly covering four instead of five is the
false coverage this suite exists to prevent.
**The forge failed because of the one above it.** The planning test
threw before its cleanup could run, leaving a module assigned that
refused the next push, so no database container was ever created.
Yesterday's fix moved that cleanup where a failure cannot skip it — but
`after` still only unassigns what was assigned, so it now tracks what
actually got added rather than what was intended.
Everything up to now stopped at composing a declaration. That proves the
control plane and the host agree, and proves nothing about whether the
thing described works — which is how five modules sat pinned to images
that did not exist while parsing and resolving perfectly.
The forge is the right one to run first. It needs a database from
another module, a password it did not choose, and a connection string it
could not have written itself: the address and port come from what the
database serves, the user name from what the mesh decided both ends
would call it. If any of that is wrong it cannot start, and nothing else
in this suite would notice.
The test checks the chain in the order it has to happen — the login
exists, the database it owns exists, the forge answers, and its log does
not say authentication failed. That last one matters: a forge that
started and could not reach its database would still answer on its port.
Rewriting image references is now shared rather than copied from the
bundle, which had the same problem first. A digest is not knowable until
something is built, and when it is, it belongs to whichever registry
served it — so the text says which image and the scenario says which
copy. Matching is on the repository, with a test that a repository
ending in another one is not half-replaced.
Also makes the planning test put the machine back. Tests here share one
mesh, so the five modules it assigned were inherited by whatever ran
next; harmless while nothing pushed, and not harmless now.
**The adoption.** Software nobody here wrote, taking its credentials the
way such software does — from its environment — and needing two
containers that reach each other by name. The first module that could
not have been declared this morning: it needs the network shape and it
needs a sealed value to reach a container's environment.
Its password is accepted rather than generated, which is the whole shape
of an adoption: a service that already exists keeps the credential it
already has. Asserted properly — a wrong password is refused by the same
database, so the passing case means something.
**The warm scenario.** A mesh kept between runs and returned to, which
turned twelve minutes of bootstrap into thirty seconds of restore. Off
unless asked for: a run that is meant to mean something raises from
nothing.
Its guard fired for real during this work, unprompted — a mesh-host
commit landed and it refused the stale base, naming both commits, rather
than passing tests against yesterday's binary. That is 04-ISSUES/005's
rule one level down.
Three things the guard learned the hard way and now handles: a snapshot
captures disk and not memory, so the host is restarted after a restore
and asserted to have come back; the stocked image digests are worked out
while raising and a restored instance never raises, so they are kept;
and comparing only the repositories this run can see clears the ones it
cannot, so both directions are compared.
The one real bug behind five failed attempts was in mesh-host and it
reported itself precisely: a network shape the language had and no host
implemented. Everything else was scaffolding of mine.
The other half of the certificate split: the mesh's own authority
certifies internal names, and a name reachable from outside needs one
the world already trusts. Against a real server rather than a stub,
because what is under test is whether an order, a challenge and a
handshake agree, and a stub would be told to agree.
One assertion passes and one fails, and the failure is filed as
novox/hq 04-ISSUES/020: the authority issues a certificate and the
client never collects it. Kept as a failing test rather than deleted or
skipped — it is the reproduction, and it proves everything up to the
last hop.
The passing one is the guard that matters day to day: no certificate is
ordered for a name the mesh does not route, so a scan cannot spend an
account's rate limit.
The failure output gathers both sides before asserting. The first
version reported only what the proxy said, which made a server-side
question unanswerable — "the client never spoke to it" and "it refused
what the client said" are different faults with nothing in common.
Seven assertions against a real store, the important one being that a
consumer cannot reach another consumer's bucket — isolation here is a
policy somebody wrote rather than a boundary the product has.
The revocation test stages its own precondition. The first version
asserted a key left by an earlier test, and the rotation test had
already revoked it two tests early: the behaviour was correct and the
test was measuring residue. Its precondition assertion is what caught
that, rather than it passing green having verified nothing.
The request goes to the name, across the private network, and returns the
workload's own answer. Then the module is unassigned and the same request must
stop working — a stale public name pointing at nothing fails more visibly than
a stale grant.
The workload declares its port as well as its route, because they are different
questions and the earlier test leaves this machine filtering: a module that
asked for a route and not for the port would be unreachable by the proxy it
just asked for.
Two ends holding a matching string proves they agree, not that either is right.
So the check is three logins over the private network from the consumer's own
machine: the delivered credential works, the rotated one works, and the one
that was rotated away does not. Without the last, the test passes against a
provider that added a password without replacing one.
Not over loopback: pg_hba trusts anything there, and a deliberately wrong
password returned a row for a whole afternoon once.
Two setup faults, each of which looked like the thing being tested failing.
The builder module was assigned without its artifact ever being built, so
nothing could start — and the build has to happen while the hand-started
builder is still alive. Same chicken-and-egg as the registry, resolved the same
way: the builder that exists builds the one that replaces it.
The firewall test's listeners were squeezed through three levels of shell
quoting and never started, so the test failed on its own setup — which reads
exactly like the firewall working.
Written and loaded are different things, and loaded and enforcing are different
again. The test opens two ports on a machine, declares one of them, and checks
from the other machine that the declared one answers and the undeclared one
does not — then removes the module and checks the port closes with nobody
editing a rule.
The base image gains nftables, read back through `nft --version` like the other
three: a machine that cannot load a rule set applies the mesh's filtering,
reports success and filters nothing, which is the exact fault the derivation
exists to remove.
Two earlier tests were asking for things that are not there. The lab's registry
drops tags when it stocks, so `registry:2` is not served and the mirror test
failed with "not found" — it now uses the pinned digest, which is what a
declaration carries anyway.
The mesh generates a password, seals it to the machine that must accept
it, and discards the plaintext — so it cannot tell PostgreSQL to start
accepting it. Something on that machine reads what the host wrote and
makes it true. Everything up to that step is proven elsewhere; this is
where a password either becomes a login or does not.
A scenario with one machine and a database, and six assertions: the
password works, running again reaches the same state and says nothing,
rotation makes the new one work and the old one stop, a departed consumer
loses its login, a role nobody here made is left alone, and a manifest
naming a credential that was never written is refused rather than
creating a login with no password.
Each was confirmed to fail — and only it to fail — with the behaviour
removed from the provisioner: only-creates breaks rotation, no-revoke
breaks revocation, revoking everything breaks the bystander role, and
ignoring a missing credential breaks the refusal.
Two faults in the test itself, both worth recording:
- it checked logins from inside the database's own container over
127.0.0.1, which PostgreSQL's default pg_hba trusts. No password was
ever verified. Demonstrated directly: over loopback a deliberately
wrong password still returns a row. Only the rotation assertion
noticed, because it is the one that requires a password to STOP
working — which is an argument for writing that assertion every time.
- the fix then read .NetworkSettings.IPAddress, which docker 29 no
longer populates. It templates to empty, psql falls back to a unix
socket that is not there, and every login looks impossible rather than
misconfigured.
A sealed scenario cannot install wireguard-tools any more than it can install a
container runtime, so a lab without them cannot test connectivity at all --
which is most of what the mesh does between machines. Installed and not
started: what a node runs is the mesh's decision, and a lab that brought the
interface up itself would be testing its own setup.
growing-mesh exists to be grown. The point is not the third machine, it is that
adding one changes every other node's peer list -- so each has to be told
again, or the newcomer is on a network nobody else can see.
The first raises everything from the bundle its host carries and joins the mesh
it made; the second has a host and nothing else, and a person carries it a
token. This is the first scenario where the mesh is a mesh -- everything before
it proved a machine could talk to a control plane on its own loopback, which
proves less than it looks.
'incus list' failed because this shell had no permission to reach the daemon,
incusOk returned null, and the caller wrote ?? "[]". So 'mesh-lab list'
printed 'no scenario instances standing' -- confidently, about a question it
had never managed to ask.
The comment on incusOk warns about exactly this, in those words: absence and
success made indistinguishable. Three of its own callers then did it. Two
listings and the live diagram, which would have drawn an empty scenario rather
than fail -- a picture that is confidently wrong, which is worse than none.
Anything enumerating what exists now goes through enumerate() and throws.
incusOk stays right where failure genuinely means no, like instanceExists,
and there is a test holding that line so this does not get over-corrected
until nothing can be asked at all.
Worth noting 'mesh-lab check' already diagnoses this precise cause, down to
'a session that predates it cannot see it'. The diagnosis existed; the
listing just never asked for it.
One machine with PostgreSQL in its registry, for developing the bootstrap.
Used to verify that a sealed machine can raise a store and a database from the
bundle its host carries.
Closes 04-ISSUES/009. A scenario declares `images:` by tag; the lab stocks a
registry on this workstation where there is a network, raises it inside the
scenario as scenery, and reports the references a declaration pins -- which are
the digests THIS registry assigned, and are not knowable until it is raised.
Verified in a sealed machine, confirmed by ping to have no route out: package,
service including boot state, a container pinned by digest, and an action
inside that container. Applied, idempotent on re-apply, and read back from the
machine rather than from the apply's own report. That is the first time the
container shape has worked in the lab at all, and it was the shape blocking the
substrate bootstrap.
Four faults found by running it, three of them mine and one worth keeping:
The read-back checked that the catalog endpoint answered, by looking for the
substring "repositories" -- which `{"repositories":[]}` also contains. So it
passed on a registry holding nothing, and the failure surfaced much later as a
container that could not be pulled. It now asks for each image's manifest BY
DIGEST, which is what a machine does.
A recursive push needs its destination to exist, or incus copies the source's
contents rather than the source. The data landed one directory too shallow and
the registry found nothing where it looks.
The registry writes its blobs as root through a bind mount, so the workstation
could not remove its own scratch directory afterwards. Whoever made the files
removes them -- the cleanup now runs in a container too. And a cleanup failure
no longer fails a raise that succeeded: the scenario is standing and usable,
and saying otherwise would be a false report.
The base image build did not verify that the runtime trusts the documentation
ranges as plain-HTTP registries. Writing the file is not the daemon honouring
it, and a base image that looks right fails much later, in a sealed scenario,
a long way from its cause. It is now read back from `docker info`.
The lab raised an underlay and put nothing on it: correct, and useless, because
the thing it exists to test did not exist. Tier 0 now does, so `place: [host]`
works and a raised scenario finally contains something.
The refusal narrows rather than disappearing. A scenario placing a host and a
substrate is told which half is missing, by name — not that `place:` is
unsupported when half of it now works.
Placement reads back rather than assuming. A file arriving is not a host
working, so the binary is run before it is trusted to answer questions, and what
it reports is read from the machine (ADR 0035). The binary comes from an
explicit path, because the declaration design leaves where artifacts come from
open and a search would harden into the answer by accident.
The integration test that matters is the one asserting the host reports the
MACHINE and not the workstation that placed it. A raised VM and this workstation
differ in every capability — root versus uid 1000, a clean init versus a
degraded one, no docker versus docker, no wireguard versus wg0 — so a host
reporting the wrong machine is obvious here and invisible anywhere else.
And the placed host independently confirms ADR 0031: overlay absent on a freshly
raised machine. The underlay suite already asserted that by looking for
wireguard interfaces; this is a second witness rather than the same check twice.
Two tests failed the moment placement worked, which is what they were for. They
defended "there is nothing to place yet" while that was true; the decision
changed, so they change with it rather than being deleted.
Gate: 75 unit, 20 integration.
The full topology now raises: four machines, three routers, a transit
router, six segments, in 35 seconds. Everything the declaration model can
express except `place`, which is refused because the node host it would
place does not exist yet.
Transit was a real gap, not a bug. The design says public networks are
unrelated and routed to each other, never bridged — and I built the
segments and never built the thing that routes between them, so three
public networks were islands and nothing crossed. A transit router now
holds an interface on every public segment, forwarding and no translation:
the closest thing the lab has to the internet, deliberately dumb.
Proven rather than asserted, by ping TTL across the raised topology:
within one segment ttl=64 no hops
across two unrelated public networks ttl=62 gateway + transit
multicast between public networks 0 replies
A flat internet would have shown ttl=64 and answered multicast — which
would let a node discover a peer it could never reach in production, and
report success. That is the fault the as-is layer records the mesh already
hitting with multicast name resolution.
inbound: deny is implemented as a host firewall on the machine, read back
after applying. A declared refusal that silently did not load leaves the
machine wide open, which looks exactly like a machine that is working.
Established and related traffic is accepted, so a defended machine can
still dial out rather than being a disconnected one.
Verified by running, all of it:
home -> devices (policy allow) reachable
devices -> home (policy deny) blocked
behind unforwardable NAT -> out reachable
in -> behind unforwardable NAT unreachable
inbound: deny, dialling out reachable
reaching a machine that denies inbound refused
The two routers differ exactly as declared: the forwardable one carries the
policy rule and no inbound drop, the unforwardable one carries `ct state
new drop` and no DNAT.
A gateway is the one implicit machine in a declaration — a scenario says a
segment sits behind one and never names the thing that serves it. This
materialises it.
A router is a container, not a virtual machine, because it is scenery
rather than something under test (hq ADR 0033). Verified before building
that a plain unprivileged container can do all of it: ip_forward and ipv6
forwarding settable, nftables masquerade accepted, and the conntrack
timeouts mapping_ttl depends on both writable. No privileged mode.
Verified by running, on a machine behind a household gateway reached from
one on a routable address:
home-server -> anchor 0% loss, through masquerade
anchor -> 192.168.1.135 (private, direct) unreachable
anchor -> 192.0.2.50:8080 (the GATEWAY) HTTP 200
The last line is the published-but-behind-NAT case research 004 says only
exists in production. It is now a 32-second scenario on a workstation.
Segments sharing a gateway declaration share ONE router — that is what a
VLAN-capable router is, and two routers sharing an external address would
not work anyway.
mapping_ttl is read back after setting rather than assumed. Those sysctls
are not on every kernel, and a scenario that declared an expiring mapping
and silently got a permanent one would be exactly the fault being built
against.
Four bugs found by running it, three of them the same fault — a failure
made invisible.
The router had no route to a package repository, by design, so installing
nftables at raise time could not work. The image is now built once with
temporary connectivity and cached; every scenario after that needs no
network. That failure was hidden behind `|| true`, which is why it took a
raise to find.
The builder then failed on DNS: exec works before a container has an
address, and I had treated usable as ready. It now waits for the thing
actually needed.
The stock Alpine image ships `auto eth0 / inet dhcp` and its boot-time
networking service flushed the static address the scenario set — on eth0
only, so the outside interface came up bare while inside ones were fine.
The image build now neutralises it: a router reconfiguring itself from an
image default is the lab overriding the declaration. `ip addr add … || true`
had hidden this too, and is now `ip addr replace` with no swallow.
And routers were orphaned by destroy, holding their networks open so
destroy reported removing zero segments. They now carry the same machine
tag as everything else, so one query finds an instance's resources.
A declaration goes in and a disposable mesh comes out. Verified on a
workstation, not asserted: two machines raised and addressed in 14.6s,
snapshot 0.28s, restore-to-usable 11.6s, both families pinging with no
loss, and the workstation with no route into any of it.
The declaration layer implements the model in full — three positions a
machine can be in, keyed on forwardability; gateways carrying the address
the world sees them as; both address families; multi-homing; MTU;
inter-segment policy. It is validated hard because the failures it prevents
are silent: a private range on a public segment produces no error, the mesh
simply never forms. Public segments are refused unless they use RFC 5737 or
RFC 3849 space, and a range wider than the reserved block is refused too.
33 tests, all offline.
The runtime implements less than the model, and refuses the difference.
A scenario declaring gateways, published ports, policy, inbound deny or
place is rejected at raise with every gap named. Raising it would produce a
mesh that silently lacks what it declared, which is the fault this lab
exists to catch — 04-ISSUES/003, where a firewall key is declared in five
manifests and read by no code.
Three bugs found by review and by running it, all of one family:
The readiness check truthiness-tested incusOk's return. `exec … true`
succeeds with EMPTY output, so every machine reported unreachable while
incus exec on it worked perfectly. succeeds() now exists so the mistake is
not available, and network delete had the same bug — it counted zero
segments removed while removing them.
list() split instance from machine on the last dash, so a machine called
home-server absorbed half the instance id and destroy found nothing.
Resources are now found by the metadata they carry, never by name.
restore reported success in 0.79s while the machine's agent was still
starting, so the next command failed. Both raise and restore now wait for
usable and say how long that took — reporting the earlier number is
transport reported as effect, which is the fault the lab is being built to
find.
Two incus behaviours worth recording. Its CLI reads a YAML definition from
stdin when stdin is not a terminal, so a spawned command hangs until the
timeout kills it and arrives with empty stderr — a failure with no
explanation, on a command that works when typed. And it assigns a MAC at
runtime without recording it in device config, so MACs are derived and set
explicitly, which the guest needs anyway: it names interfaces by bus
position, and matching by name configures the wrong one on a multi-homed
machine.
No build step; Node strips the types. The lifecycle has no unit tests
because a fake hypervisor would assert that the fake behaves as expected,
which is the shape of test this project exists to stop shipping.
jschoubben