A per-module bed stocks mesh-runtime-<module>:development from the workstation's
image store, built by hand by a script the suite never called; six were two weeks
older than the manifests they served. For the beds named, every runtime their
scenarios stock is compared against the module's source and the tool runtime and
SDK it is built on, and rebuilt where older, missing or uncommitted; a failed
build stops the suite (novox/hq 04-ISSUES/075).
A bed installs a catalogue module by reading its manifest from MESH_LAB_CATALOG at
run time, so a receipt naming no catalogue commit cannot say whether a catalogue
change was ever proven. Claimed under either spelling of the variable; not built,
because a manifest is read, not compiled (novox/hq 04-ISSUES/073).
ready() asked a gone instance for its snapshots before judge could say 'no longer
standing'; a stale warm.json from any earlier scenario made every warm run fail in
milliseconds. The question is now asked only of an instance that still exists.
https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
One name per thing, per the HQ glossary: the module/container/image/binary/repo
becomes mesh-controller, the seat the-controller, and the store+broker pair the
foundation (embedded base bundles, default template and example lock renamed with
their go:embed directives). No behaviour change — a pure vocabulary rename.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
Installs the shipped nox-mesh-host launcher and unit in the machine and lets the
installer's --host-service start and enable it, instead of --host-in-background
which cannot survive a reboot. E2 now proves the mesh comes back on its own.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
Every machine in a bed is a clone of one base image, so all of them booted with
the same /etc/machine-id. systemd's DHCP client derives its client identifier
from that file and dnsmasq keys leases on the identifier rather than the MAC, so
four machines with four distinct MACs were handed one address and the host kept
one ARP entry for it. Whichever machine last answered an ARP request received
everybody's replies.
This is the fault behind every run lost to "flaky lab DNS": resolution that works
two times in three, pulls that succeed on a retry, and one machine out of four
being fine while the rest have no path at all. It survived an earlier diagnosis
that blamed resolver ordering, because reordering resolvers on a machine that has
just won the ARP race looks exactly like a fix.
Each machine is now given its own machine-id before the uplink lease is asked
for, and a check after addresses are applied refuses to go on if two machines
took the same one — the positive control this never had, since the fault is
invisible where it happens and unrecognisable where it surfaces.
The egress check also now demands five consecutive lookups rather than one. A
single answer is what let a machine resolving one query in three pass and then
die twenty minutes later inside a pull.
And fresh-mesh: whole-mesh-full's topology with genesis-single's honesty. The
four-machine bed loads thirty-four of the mesh's own images onto its machines
from the workstation because it does not build them, which is a shape no real
installation has and the same fiction the lab removed when it deleted its own
registry. This scenario names no images at all. The machines pull what is public,
the installer builds the control plane, and the mesh builds the rest — including,
last and deliberately, a module on a machine that did not build it.
Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
The images a bed stocks are almost entirely the same bytes: the base they share
is 227 MB and a module's own code is a few. Exported one at a time that base is
written, pushed and loaded once per image — for this bed, the same 227 MB crossed
thirty-odd times, and the whole set measured 9.7 GB.
Measured on eight of them: 2.24 GB as separate archives, 0.29 GB as one. 87 per
cent less, and it improves with the count.
This is the slowest thing a raise does, and the four-machine bed has been
exceeding its own ninety-minute limit while still copying — so it was failing on
the clock rather than on anything it was testing.
Also stops shipping a compiler in every module image. The image runs compiled
code and never compiles any; tsc runs on the workstation. Worth 26 MB an image,
which is small beside the above but was pure waste.
The first attempt wrote /etc/resolv.conf. On these images that is a symlink
owned by systemd-resolved, so the file is either reverted or the link is broken
— found by reading a running machine instead of assuming the change had worked.
The real shape shows in resolvectl: the machine has sensible global fallbacks,
and the link carrying the default route has exactly one server, the uplink
gateway. resolved will not reach a global fallback while the link has a server
of its own, so one unanswered packet is one failed lookup. Three runs have died
that way, each long after the egress check passed.
The uplink stays first, so the modelled path is still what is used and still
what the check proves. Verified on a live machine: three servers on the link,
uplink first, resolution intact.
The egress check proves the uplink resolves and reaches the internet, and that
is the right thing to check. What it does not cover is the hours afterwards: a
bed pulls images on four machines at once, the uplink's resolver is one server
under exactly that load, and three runs have now died at a lookup timeout long
after the check passed — the path was never broken, a query just went
unanswered.
The uplink stays first and keeps proving the path. Public resolvers sit behind
it and answer only when it does not, and the retry is tightened so a silent
server costs seconds. A genuinely broken uplink still fails the check, before
any of this applies.
ADR 0067's own acceptance check said the lab must raise its anchor by running the
program a bare machine runs. It did not: whole-mesh-full applied the substrate bundle
by hand and then looped enrolment over all four machines as one continuous operation.
That gets the order right by accident and models the wrong shape — and an install
procedure that exists only as a test fixture is exercised by whoever writes tests and
never by whoever installs, which is why every bootstrap fault this year was found late.
Two acts now, and the first gates the second.
GENESIS is novox running mesh-bootstrap: the installer is built from source before
the raise (make bootstrap, carrying the control-plane image built in the same run),
placed beside the host binary, given the two manifests it reads, and run. The bed
then asserts a WORKING MESH OF ONE — the control plane answers, the registry replies
on /v2/, the container called mesh-control is running from a registry-pinned digest
rather than an image id, the registry agrees it serves it, temp-mesh-control is gone,
and the mesh has heard from its node. The image-id check is ADR 0067's "the pivot
completed" verbatim: if it is still an id, nothing was published and this mesh can
never roll out its own upgrades.
JOINING is ace, shanks and g14: host binary, token, enrol, run. novox is NOT enrolled
again — the installer already did it, and a second identity is one the mesh does not
know.
If genesis stops, the bed prints which of the installer's ten steps it stopped at and
goes no further. A second machine joining a mesh that is not ready is a different
failure, and running it would bury this one underneath it.
The anchor is no longer handed mesh-control:development. Its absence is the point: the
installer carries that image inside itself, and handing it over as well would make the
load say "already held" and leave the carrying untested — the same class of fiction the
lab's own registry used to hide. A unit test asserts the scenario keeps it out.
The registry is reached at 127.0.0.1:5000, which is a finding rather than a shortcut: a
runtime refuses a plain-HTTP registry at any address but a loopback one, so the digest
the control-plane module is pinned to is one only the anchor can pull. Enough here,
because only the anchor runs a control plane. Written down in the bed.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
Deleting the lab's registry left the operator's own images to be pulled like
anything else, and they cannot be: their registry wants an account and a
scenario machine has none. The pull fails with 'no basic auth credentials',
which is not something more patience fixes.
So the test is no longer 'did the mesh build it' but 'can the machine get it at
all'. Two ways to fail that — published nowhere, or published somewhere the
machine cannot authenticate to — and one consequence: the workstation, which
does hold the credential, exports it and loads it.
Worth saying what this stands in for. In a finished mesh these are built by the
builder and published to the mesh's own store, and every machine pulls them from
there with a credential the mesh granted. Until that store exists there is
nowhere for them to come from, and handing them over is the closest honest thing
— not a registry the lab invents, which is what was just removed.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The id is the digest of an image's configuration, and a runtime rewrites that
configuration as it loads: this workstation saves in one format, the machines
store it in another, and the same bytes arrive under a different name. Measured,
not assumed — b86bb81c here, 2dc21904 there.
So it is read back from the machine instead of predicted from here. Predicting
it failed at the only moment it mattered: every manifest would have been
rewritten to a reference no machine holds, and these images exist in no registry,
so each apply would have stopped at a pull that cannot succeed. A manifest
carries one reference, so machines that disagree stop the raise rather than
having one of them silently win.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The lab raised a `registry` VM, pushed ~73 images into it from the workstation, and
rewrote every manifest reference — third-party ones included — to point at it. No
production mesh has such a thing. So every bed proved that a machine could fetch an
image from a registry that exists nowhere else, and the bootstrap problems that only
appear when a machine has to fetch for itself went unfound.
What replaces it is the two things that are true in the world:
**Public images come from the public internet.** mesh-lab already created a NAT'd
uplink for exactly this and attached it to any machine declaring `egress`; no scenario
ever declared it. They do now, and third-party references are left exactly as the
catalogue writes them.
**The mesh's own images have no registry and never will.** mesh-control, mesh-builder,
mesh-route-proxy and the per-module runtimes are built from source and exist in no
registry. A machine gets them the way an operator's machine does — they are built here
and loaded onto it — and is then named by the digest of its own image configuration,
which mesh-host now accepts as "an image this machine already holds".
`images:` therefore means only *ours*, and a third-party entry is refused rather than
quietly loaded: otherwise the fiction returns one convenient line at a time. It is
per-machine as well, because "everything, everywhere" was never a description of
anything real — handing whole-mesh-full's union to its two 30GiB workstations would
fill the disk with runtimes nothing on them will start.
**The uplink and the declared gateway would have fought, silently.** A gateway container
and the transit router reach the scenario and nothing else; a default route through
either is a black hole for anything outside, and it beats the uplink's DHCP route on
metric. So a machine with egress states the scenario's ranges explicitly — through the
same gateway or transit it would have defaulted to, so the overlay-across-NAT path is
unchanged — and leaves the default to the uplink. A range with no path inside the
scenario becomes `unreachable` rather than falling through: 192.168.1.0/24 is an
ordinary private range in fact, and letting it escape would put scenario traffic on
whatever network the workstation is sitting on. `scenarioRoutesFor` is pure and tested,
because a decision only a full raise could check is one nobody checks.
The registry-reachability check the raise gained earlier is kept, pointed at the real
thing: every machine with egress must resolve a name and reach the internet before the
raise says it finished. Same failure it was written for — a raise that returns, an apply
that dies on its first pull, an instance left a bare shell — now guarding the path that
actually carries.
The base image's trust of the documentation ranges as plain-HTTP registries STAYS. It
was never only for the lab's registry: the mesh has one of its own, the `registry`
module, serving artifacts to the whole mesh over plain HTTP from whatever node runs it.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The first whole-mesh raise of the ADR 0056 code died on the anchor's substrate
apply: the image pulls failed, the anchor never came up, no node could enrol,
and the instance was left a bare shell — VMs and a registry, no substrate. The
identical apply, run by hand once the registry was warm, succeeded immediately.
`raiseRegistry` proves the wrong thing. It curls `localhost:5000` from inside
the registry's OWN machine, which says the registry process is up and holds the
blobs, and says nothing about the path anybody else uses: across a segment, and
for the home nodes through a NAT gateway whose default route and firewall are
applied two steps LATER. So "serving" was reported on evidence that excluded the
network, and the caller — which pins every image in the substrate bundle to that
registry — was handed a fact it could not rely on.
So the check moves to where it means something. After the routes and the
firewalls, before the minutes spent placing, each machine is asked for `/v2/` and
for one stocked manifest BY DIGEST, at the address it will pin, over the network
it will use. That is the pair of requests a pull begins with, from the same
place. Layers are not fetched: every digest was already read back inside the
registry machine, so what is in question here is the path, not the content.
Verified by typecheck and the unit suite (136 pass), and by confirming against a
standing four-node instance that `curl` exists in the machines and that both
segments — including a home node through the gateway — answer 200 for the
registry's `/v2/`. The ordering itself is unverified in a live raise from cold,
which takes hours.
Rewrite the flat three-node whole-mesh-full (separate anchor, one public segment)
into production's real shape: two segments and one access point. novox sits on
the routable `hosting` segment and IS the anchor — it runs the substrate, its own
service set, the overlay hub and public ingress; there is no separate anchor node.
ace, shanks and g14 sit on the household `home` segment behind a NAT gateway,
reachable from outside only through what they dial out to.
The bed drives, and verifies, the thing the flat beds never could: the WireGuard
overlay forming ACROSS the access point — a home node dialling novox's public hub
endpoint out through the gateway's masquerade, the handshake completing through the
NAT, the keepalive holding the hole open. Phase A proves it (handshake state + a
ping over the overlay) before any heavy module lands; Phase B converges both server
sets. With MESH_LAB_KEEP the instance is raised under a fixed id and left standing.
Collapsing the substrate onto novox exposed real facts the separate-anchor beds
never hit, fixed here:
- the substrate bundle advertises the broker at 192.0.2.10 (the old anchor); a
token carries that verbatim as the endpoint a node dials, so with the substrate
on novox it must be novox's own public address. Rewritten at apply (the cert is
fingerprint-pinned, not hostname-checked, so only the address needs correcting).
- the two provider host-port collisions with the co-located substrate: postgres
5432 vs the store's 127.0.0.1:5432, lavinmq 5672 vs the broker's 127.0.0.1:5672.
Both provider host publishes are remapped off the substrate's ports.
And a lab limitation this first large-union bed exposed: the image registry VM took
the profile's default `dir` pool and a ~10GiB root, which the ~28GiB union of both
server sets overflows ("no space left on device"). raiseRegistry now places the
registry on the scenario's copy-on-write pool with a sized (default 80GiB, thin)
root disk, MESH_LAB_REGISTRY_DISK overridable.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
The GREEN multi-node regression bed that proves the DB-consumer gate: substrate/control on
one node, postgres+redis providers and baserow+letta consumers on another, each consumer
getting its own credential and its own mesh-named database across the overlay. Requires the
mesh-control provider-seal-key fix and the mesh-catalog db-name fix.
Includes a general lab capability: a machine 'disk' field sizing the VM root disk (a broad
install exhausts the pool default and the host fails mid-apply with 'no space left on
device'). The bed sets 60GiB.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
A consumer contributes a key prefix and gets an ACL user; the test is
that the grant means exactly what the manifest said, in both
directions: its own keys usable, anyone else's refused by the store
itself, and the flush a tenant must never have refused with them.
Waited for through the store rather than through logs: the user list,
asked with the password the host wrote into the server's own conf file
on the machine — nothing invented, both ends reading what the mesh
delivered.
And the forge is asked on the port the mesh assigned, not the one the
module declared. The old curl aimed at 3000, which was right until
ADR 0038 moved the machine side — a latent break that would have fired
on the first run to get past the settling that used to fail first.
The scenario stocks redis and its provisioner, and the rebuild builds
the provisioner image with the others.
whatWasTested read the repositories when the run ended, so a commit
landing during the twenty minutes a suite takes was recorded as tested
without ever being in the binaries. It happened: one receipt named a
commit made mid-run, and the verdict it carried belonged to an older
tree.
The heads are read once, right after the build, and carried to the
receipt. A verdict is only worth something attributed to one exact
state, which is the receipt's whole reason to exist.
A segment named "uplink" is refused. The lab claims that name for the
NAT bridge behind `egress: true`, and a scenario wearing it first would
have its egress machines silently attached to an isolated bridge — a
declared key doing nothing, which is the fault this repo exists to
refuse, in the repo that refuses it.
settled() parses inside the try. A truncated status from a struggling
machine was the one shape of bad answer that still threw out of the
wait, and the likeliest moment for one is exactly the machine the poll
is watching. Malformed now counts as "could not ask", like the exec
that times out.
And a sentence on the uplink's UseDNS saying its inertness is
load-bearing: it matters only where systemd-resolved runs, and on a
machine whose modules own resolv.conf the uplink must not outvote the
resolver a scenario is testing.
Three changes, found by one failing test.
The forge failed three runs in a row as "status hangs", and it was
diagnosed twice as contention — real defects, fixed, and not the cause.
The heartbeats told the truth in the end: every exec on anchor crawled
from 15s to 105s, because eleven containers plus a database pull were
running in a 1GiB machine. Starvation presents as whatever you were
doing when the page-outs start, which is why it wore two other bugs'
clothes first.
So machine size is now the scenario's to declare — memory and cpus per
machine, default unchanged. The anchor that carries the whole substrate
is bigger than the laptop that joins it, and the comment on the
scenario says why in terms of what lands there.
`egress: true` gives a machine one extra interface on a lab-supplied
NAT network, addressed by DHCP because the one address a scenario has
no business choosing is on the host's side of the fence. Declared
per machine and off by default: a closed scenario stays the rule
(novox/hq ADR 0016), and the exception exists because a first node
fetches its images before any mesh can serve them — which is now the
tested path (04-ISSUES/029), and a lab that can never reach upstream
cannot prove the bootstrap it exists to prove. The uplink route is
metric-4096, so it never shadows a route the scenario declared. A
detached machine declaring egress is refused, not ignored.
And settled() treats a poll that threw as a poll that missed. An exec
timeout at minute four of a wait is "could not ask", not a verdict on
the machine.
The canary walked one path on one machine — a mesh comes up, a module
lands, a consumer gets a credential — and stopped the run if it broke.
That path is exactly what the first three tests of the long run walk,
and the long run finishes them about 160 seconds in.
So the gate cost a whole scenario on every passing run to save roughly
45 seconds on a failing one. A scenario is three machines, one of them a
registry that boots a kernel in order to serve files, which is where the
two minutes went.
The test file stays and still runs when it is named. What is gone is
raising it on the way to everything else.
Measured rather than argued: the canary's scenario took 116s of which
60s was standing up a registry, and the run reached the same assertions
without it.
A scenario is a closed address space: two raised from the same
declaration hold the same addresses and never meet, which is what lets
two run at once and why the lab talks to machines through the
hypervisor rather than over IP. Reaching in from outside breaks that, so
it is opt-in, one scenario at a time, and reversible.
`connect` takes an address on the scenario's public link and writes a
resolver rule answering everything under each machine's name.
`disconnect` gives both back. `connected` says what is true right now,
for somebody who cannot remember.
It refuses rather than guessing when more than one scenario is standing
— the failure being avoided is not an error but one scenario's traffic
arriving in another. It also refuses when a machine's name is already
answered here for something real, because connecting would point that
name at the lab, and the damage would land on the real thing.
Names answer with the segment address rather than the overlay one.
Inside the mesh a name gives a machine's private address; from here that
would need this workstation on the overlay, which is a much larger door.
The segment address reaches the same machine and the same ports, which
is what opening a board in a browser actually needs.
Proven against a live two-node scenario: registry.internal:5000/v2/
answered 200 from this workstation, and so did a wildcard name under the
same machine. Disconnect put the address back, stopped answering, and
left the real mesh's own names alone.
One thing measured rather than assumed: it restarts dnsmasq instead of
reloading it. A reload is SIGHUP, which re-reads the hosts file and
clears the cache but not the configuration — the rule was written, the
reload reported success, and nothing resolved. The daemon's start time
was nine days old afterwards.
Suggested by Jochen, and it paid for itself on its first run.
A suite that takes forty minutes is a suite you hear from once a day.
Every fault found today would have shown up in the first three minutes
of it — a module pinned to an image that does not exist, a consumer
given a password and no name to present with it, a credential file
nothing could read, a search for a password that read the password as an
option. The other thirty-seven minutes proved things that were already
working.
So this runs first, on one machine, with the three images the mesh needs
for itself. It walks one path: a mesh comes up, a module lands, and a
consumer gets a credential it can actually use — the name to present,
the address, the port, and a password only the host could put there.
Deliberately not a smaller copy of the full suite: that path is where
everything went wrong, and a canary checking many things shallowly is a
canary whose failure nobody can read.
`suite` runs it and stops if it dies, saying why rather than leaving
somebody to wonder what the missing thirty-seven minutes would have
said. Skipped when the caller named its own files.
It measured 164 seconds against forty-odd minutes, and failed three
times on its first run for one reason: applying the bundle raises a
control plane but does not tell it a machine exists. I had left out
enrolment, and the long suite would have taken forty minutes to say so.
Everything up to now stopped at composing a declaration. That proves the
control plane and the host agree, and proves nothing about whether the
thing described works — which is how five modules sat pinned to images
that did not exist while parsing and resolving perfectly.
The forge is the right one to run first. It needs a database from
another module, a password it did not choose, and a connection string it
could not have written itself: the address and port come from what the
database serves, the user name from what the mesh decided both ends
would call it. If any of that is wrong it cannot start, and nothing else
in this suite would notice.
The test checks the chain in the order it has to happen — the login
exists, the database it owns exists, the forge answers, and its log does
not say authentication failed. That last one matters: a forge that
started and could not reach its database would still answer on its port.
Rewriting image references is now shared rather than copied from the
bundle, which had the same problem first. A digest is not knowable until
something is built, and when it is, it belongs to whichever registry
served it — so the text says which image and the scenario says which
copy. Matching is on the repository, with a test that a repository
ending in another one is not half-replaced.
Also makes the planning test put the machine back. Tests here share one
mesh, so the five modules it assigned were inherited by whatever ran
next; harmless while nothing pushed, and not harmless now.
novox/hq 04-ISSUES/024. A run stalled for thirty-five minutes and said
nothing. The cause was a link systemd was still configuring, three
layers down inside a `docker load` blocked on a socket — and every one
of those layers knew what it was waiting for. None of them said so.
Three decisions, each doing work.
**Every external command is logged, at the three places that run one.**
Ninety-seven call sites reach a hypervisor or a container runtime
through three wrappers, so instrumenting the wrappers covers all of them
and nothing has to remember to log.
**A command still running says so while it runs.** A line before and a
line after tells you nothing until the after arrives, which is exactly
the case that matters. Anything outstanding past fifteen seconds reports
itself with how long it has been going. It is reported as still running,
not as stuck — which it is is not knowable from there, and a log that
calls a slow step a hang teaches people to ignore it.
**It goes to a file, written synchronously.** Node block-buffers stdout
when redirected and a test runner buffers it again, so a console log can
sit minutes behind. `appendFileSync` cannot lag.
Two things this found in itself while being written, both the same shape
as what it exists to catch:
A question that answers no is not a fault. Half the lab's commands are
questions — does this network exist, is the agent up yet — and they fail
constantly while a scenario comes up. Logging those as faults filled a
healthy run with ✗, which is how you end up ignoring ✗ when one is real.
They are recorded quietly now, and still recorded.
And `around` skipped its own wrapper when a step's level was below the
configured one — taking the failure line and the heartbeat with it. The
two things worth having at a low level were the two that vanished at
exactly the level somebody would use. The gate belongs in `write`.
Also unsilences the four call sites that passed a callback throwing
everything away, including the one the stall sat in, and tees `raise`'s
progress into the file whether or not a caller asked to see it — the
end-to-end test passed no callback, so the one run that mattered
reported not a single step.
novox/hq 04-ISSUES/024. The registry machine had its address set with
`ip addr add`; every other machine gets a systemd-networkd unit. That
one difference stalled the lab indefinitely.
An address set by hand leaves networkd waiting to configure a link it
was never told about, so the link sits at `configuring` for ever.
`systemd-networkd-wait-online` has TimeoutStartUSec=infinity, so
`network-online.target` is never reached — and Docker is ordered after
it. `docker load` then blocked on a socket whose daemon was queued
behind a target that would never come.
Measured before and after on the same scenario: stuck with five pending
systemd jobs and `docker` inactive; now `enp5s0 configured`, `docker`
active, no jobs, and the whole raise completes in 87.5s.
The guess in the issue was wrong, and it was wrong in the usual way —
stocking had just been changed, so stocking looked guilty. Stocking
takes 34s and always did.
Two things that made this cost hours rather than minutes are fixed with
it. Placing an image now waits for the container runtime to answer and
refuses after 120s naming what systemd is waiting on, so a stall becomes
a failure that says why instead of three stacked timeouts totalling 35
minutes. And the end-to-end test passes `onProgress`, so a raise says
what step it is on — it printed nothing at all until it finished, which
is why 35 minutes of nothing read as a slow test.
A run today had a control-plane image built that minute and a
provisioner image built the day before. The rotation test failed against
a real database and it looked exactly like the change under test being
wrong — the provisioner was creating logins by a naming rule that had
been replaced hours earlier.
It was the rebuild. It covered `make image` and the builder binary and
none of the three other image targets, all of which the suite runs.
This is the same fault the builder line was added for, one target along,
and the comment there already names the precedent: building one and not
the other is the eleven-hour-old binary. A rebuild that covers most of
what a run uses is worse than one that covers none, because the run that
follows it is believed.
The test names each target rather than counting them, because what goes
wrong is a target that exists and is not run, and a count would not
notice.
**The adoption.** Software nobody here wrote, taking its credentials the
way such software does — from its environment — and needing two
containers that reach each other by name. The first module that could
not have been declared this morning: it needs the network shape and it
needs a sealed value to reach a container's environment.
Its password is accepted rather than generated, which is the whole shape
of an adoption: a service that already exists keeps the credential it
already has. Asserted properly — a wrong password is refused by the same
database, so the passing case means something.
**The warm scenario.** A mesh kept between runs and returned to, which
turned twelve minutes of bootstrap into thirty seconds of restore. Off
unless asked for: a run that is meant to mean something raises from
nothing.
Its guard fired for real during this work, unprompted — a mesh-host
commit landed and it refused the stale base, naming both commits, rather
than passing tests against yesterday's binary. That is 04-ISSUES/005's
rule one level down.
Three things the guard learned the hard way and now handles: a snapshot
captures disk and not memory, so the host is restarted after a restore
and asserted to have come back; the stocked image digests are worked out
while raising and a restored instance never raises, so they are kept;
and comparing only the repositories this run can see clears the ones it
cannot, so both directions are compared.
The one real bug behind five failed attempts was in mesh-host and it
reported itself precisely: a network shape the language had and no host
implemented. Everything else was scaffolding of mine.
The danger is not that the suite breaks. It is that nobody notices it
stopped running (novox/hq 04-ISSUES/005). The harness this replaces had
not built for two and a half months and nothing said so — and this suite
needs a hypervisor, so it inherits exactly that: it runs when somebody
remembers, and remembering is not a mechanism.
So running, recording, and rebuilding are one act:
- the host binary, control-plane image and builder are rebuilt from
source first. The last two both parse manifests; building one and not
the other left a binary eleven hours old refusing a field the mesh had
just renamed, found by a full run.
- a receipt lands in XDG state — outside git, because the question is
whether *this machine* has run it, and a receipt in git would be a
claim about everybody's machine made by whoever committed last.
- `last-run` judges it and exits non-zero when it no longer counts.
Three faults found by running the thing rather than reading it, each now
held by a test confirmed to fail without it:
- counted() passed every test while parsing nothing. The runner colours
its summary even into a pipe; the fixtures were clean text that had
been imagined rather than captured. A fixture that agrees with the
mistake proves the mistake.
- a receipt for `suite test/lastrun.test.ts` was indistinguishable from
one for the real thing — 005's own symptom, rebuilt inside its remedy.
The receipt now records what ran.
- a tree with uncommitted work reported the bare commit, claiming
coverage of code nobody can check out. Nothing else could tell: the
hash is identical either way.
Proven on real machines: 22/22, against all three repositories.
Through the path an application actually takes — nsswitch, files, then DNS —
because the resolv.conf module is half of what is being tested and only that
path goes through it. Asking a server directly would prove less.
Both machines resolve, from their own copy: a mesh where one machine answers
for all of them stops resolving when that machine does, which is the
arrangement this design refuses everywhere else.
The manifests are read from mesh-control's examples rather than written here,
so what is proven is what ships. And dnsmasq joins the base image, read back
through --version like the others: a machine that cannot answer names applies
the resolver data, reports success, and resolves nothing.
Written and loaded are different things, and loaded and enforcing are different
again. The test opens two ports on a machine, declares one of them, and checks
from the other machine that the declared one answers and the undeclared one
does not — then removes the module and checks the port closes with nobody
editing a rule.
The base image gains nftables, read back through `nft --version` like the other
three: a machine that cannot load a rule set applies the mesh's filtering,
reports success and filters nothing, which is the exact fault the derivation
exists to remove.
Two earlier tests were asking for things that are not there. The lab's registry
drops tags when it stocks, so `registry:2` is not served and the mirror test
failed with "not found" — it now uses the pinned digest, which is what a
declaration carries anyway.
went
`exec` waited two minutes always. A build, or anything that waits on
another machine, needs longer — and a caller that cannot say so has to
split the work to fit, which is a test shaped by its harness rather than
by what it is testing.
The scenario also places a build machine when one is given, so anything
in it can ask the mesh to build something. Nothing else here would start
one.
And the mesh-runs-its-own-artifact-store test is not here. It needs a
fourth image so the module has a registry to mirror, and that is caught
behind 04-ISSUES/012 — left as a note saying where it went and why,
rather than silently deleted, because what it asserted is worth
asserting.
The chain this closes: a repository exists, the mesh asks for it, a build
machine takes the work, publishes what it made, and the catalogue then
says what the module is, which commit it came from, and — after the
source moves — that it is behind.
Three assertions, against a real broker and registry, because what is
under test is four processes agreeing over a wire:
- the mesh asks, a machine builds, and the artifact is really in the
registry at the digest the manifest names
- a build that cannot succeed says why and records nothing. A failure
that is silent is indistinguishable from a builder that is not running
- the source moving makes the catalogue say "behind", and rebuilding
catches it up
git is now in the base image, with the same reasoning as docker and
wireguard-tools: a machine that builds modules clones them, and a sealed
scenario cannot install anything. Read back from `git --version` rather
than from the package manager — an installed package is not a
capability, and a build machine whose clone fails does so three minutes
into a scenario with the failure reported as a build problem rather than
a lab one.
A sealed scenario cannot install wireguard-tools any more than it can install a
container runtime, so a lab without them cannot test connectivity at all --
which is most of what the mesh does between machines. Installed and not
started: what a node runs is the mesh's decision, and a lab that brought the
interface up itself would be testing its own setup.
growing-mesh exists to be grown. The point is not the third machine, it is that
adding one changes every other node's peer list -- so each has to be told
again, or the newcomer is on a network nobody else can see.
'incus list' failed because this shell had no permission to reach the daemon,
incusOk returned null, and the caller wrote ?? "[]". So 'mesh-lab list'
printed 'no scenario instances standing' -- confidently, about a question it
had never managed to ask.
The comment on incusOk warns about exactly this, in those words: absence and
success made indistinguishable. Three of its own callers then did it. Two
listings and the live diagram, which would have drawn an empty scenario rather
than fail -- a picture that is confidently wrong, which is worse than none.
Anything enumerating what exists now goes through enumerate() and throws.
incusOk stays right where failure genuinely means no, like instanceExists,
and there is a test holding that line so this does not get over-corrected
until nothing can be asked at all.
Worth noting 'mesh-lab check' already diagnoses this precise cause, down to
'a session that predates it cannot see it'. The diagnosis existed; the
listing just never asked for it.
Closes 04-ISSUES/009. A scenario declares `images:` by tag; the lab stocks a
registry on this workstation where there is a network, raises it inside the
scenario as scenery, and reports the references a declaration pins -- which are
the digests THIS registry assigned, and are not knowable until it is raised.
Verified in a sealed machine, confirmed by ping to have no route out: package,
service including boot state, a container pinned by digest, and an action
inside that container. Applied, idempotent on re-apply, and read back from the
machine rather than from the apply's own report. That is the first time the
container shape has worked in the lab at all, and it was the shape blocking the
substrate bootstrap.
Four faults found by running it, three of them mine and one worth keeping:
The read-back checked that the catalog endpoint answered, by looking for the
substring "repositories" -- which `{"repositories":[]}` also contains. So it
passed on a registry holding nothing, and the failure surfaced much later as a
container that could not be pulled. It now asks for each image's manifest BY
DIGEST, which is what a machine does.
A recursive push needs its destination to exist, or incus copies the source's
contents rather than the source. The data landed one directory too shallow and
the registry found nothing where it looks.
The registry writes its blobs as root through a bind mount, so the workstation
could not remove its own scratch directory afterwards. Whoever made the files
removes them -- the cleanup now runs in a container too. And a cleanup failure
no longer fails a raise that succeeded: the scenario is standing and usable,
and saying otherwise would be a false report.
The base image build did not verify that the runtime trusts the documentation
ranges as plain-HTTP registries. Writing the file is not the daemon honouring
it, and a base image that looks right fails much later, in a sealed scenario,
a long way from its cause. It is now read back from `docker info`.
Comments naming records that no longer exist now point at the consolidated
record holding their reasoning -- the four lab records are 0016, a test defends
a decision is 0017.
Issue 009's resolution, proven manually end to end before any of it was
written.
A sealed machine pulled an image BY DIGEST from a registry on its own segment
and ran it; then the host applied all four shapes -- package, service with
boot, container from that digest, and an action inside it -- idempotently. That
is the first time the container shape has worked anywhere but a workstation,
and it was the shape blocking the whole substrate bootstrap.
The registry's digests are its own, not Docker Hub's, and that is correct
rather than a compromise: ADR 0046 requires a reference that is exact and
cannot move, and a digest this registry assigned is both. It is also not a
lab workaround -- 0048 names an OCI registry as substrate and 0046 says a first
node fetches "upstream, wherever the image ordinarily lives". This IS that
upstream, scenery in the same sense the transit router is the internet.
The base image now trusts the RFC 5737 and RFC 3849 documentation ranges as
plain-HTTP registries. Scoped to those rather than an address because they
never route on the real internet, so it cannot make a real machine trust a real
registry whatever it is copied onto.
Three faults found while verifying, two of them mine:
My probe script picked an interface with `ls /sys/class/net | head -1`, which
returns docker0 once a runtime exists -- so it addressed the wrong interface and
then, because that address overlapped the segment, broke routing on the machine
entirely. The lab itself is immune: it matches by MAC, for a related reason it
already recorded (bus-position naming on multi-homed machines).
And a test that proved nothing: I asserted `sha256:tooshort` is rejected, but
its letters fall outside a-f, so it failed the character class rather than the
length check. Replaced with hex of the wrong length, after which removing the
length check bites.
ADR 0046's open consequence: "the lab needs a way to place images, and the
machine it places them into needs a container runtime, which a sealed scenario
cannot install either."
The runtime half is done, and it is research 012's reframing applied literally
-- fetch at build time on a machine with a network, apply on a target that
needs nothing. `mesh-lab base build` launches a machine WITH a network,
installs a runtime, verifies it by asking the runtime rather than the package
manager, and publishes the result. Measured: ~30s to install, ~60s to publish,
~700MiB, paid once per lab rather than per scenario.
A scenario that places `runtime` or an image is then raised from that base
image, chosen rather than declared -- a scenario says what it needs, not which
image provides it. If the base does not exist it says so and how to build it.
Verified in a genuinely sealed machine (no route out, confirmed by ping):
package, service including the new `boot: enabled`, and action all applied,
were idempotent on a second run, and read back correctly. Those three had never
run anywhere but a workstation.
The image half is NOT done, and testing found why: a digest-pinned image cannot
be placed from an archive. `docker save alpine@sha256:...` produces an archive
with no repo tag, because a repo digest only exists for an image a registry
served -- so it loads dangling and a container declaring that digest reaches
for a registry the machine cannot see.
That collides with ADR 0046, which has the host REFUSE an unpinned image. Tag
refused by the host, digest unusable in the lab: there is currently no
declaration the lab can raise that exercises the container shape at all. Filed
as 04-ISSUES/009, whose resolution is a registry inside the scenario -- which is
what the real mesh does rather than a workaround for the lab.
Also fixed a weak check of my own, which is the same fault in miniature: the
load was tested with `includes("Loaded image")`, a prefix of both `Loaded
image:` and `Loaded image ID:`. So an unusable dangling load reported success
and the failure surfaced later as a container that would not start.
The lab raised an underlay and put nothing on it: correct, and useless, because
the thing it exists to test did not exist. Tier 0 now does, so `place: [host]`
works and a raised scenario finally contains something.
The refusal narrows rather than disappearing. A scenario placing a host and a
substrate is told which half is missing, by name — not that `place:` is
unsupported when half of it now works.
Placement reads back rather than assuming. A file arriving is not a host
working, so the binary is run before it is trusted to answer questions, and what
it reports is read from the machine (ADR 0035). The binary comes from an
explicit path, because the declaration design leaves where artifacts come from
open and a search would harden into the answer by accident.
The integration test that matters is the one asserting the host reports the
MACHINE and not the workstation that placed it. A raised VM and this workstation
differ in every capability — root versus uid 1000, a clean init versus a
degraded one, no docker versus docker, no wireguard versus wg0 — so a host
reporting the wrong machine is obvious here and invisible anywhere else.
And the placed host independently confirms ADR 0031: overlay absent on a freshly
raised machine. The underlay suite already asserted that by looking for
wireguard interfaces; this is a second witness rather than the same check twice.
Two tests failed the moment placement worked, which is what they were for. They
defended "there is nothing to place yet" while that was true; the decision
changed, so they change with it rather than being deleted.
Gate: 75 unit, 20 integration.
The address collision was found by eye. This is the mechanical form of it: no
two machines hold one address on one segment.
Pure over already-collected facts, so the logic is tested without a hypervisor
— including the cases that would make it useless if got wrong: the same address
on DIFFERENT segments is normal and must not be reported, and one machine
holding an address twice is not two machines.
Asserted against whatever the integration suite has standing, read from the
hypervisor rather than from the declaration. The declaration is what was
accepted, and it was accepted.
Found by asking what gw-devices and gw-home actually were, in a picture that
finally made them easy to see side by side.
planRouters grouped on the exact address list, so `home` declaring a v4 and a v6
address and `devices` declaring only the v4 became two router containers — both
holding 198.51.100.7 on the same segment. The lab raised it without complaint.
Not theoretical. On the raised instance the transit router resolved that one
address to two different MACs across a cache flush:
198.51.100.7 -> 02:c9:16:70:23:29 (gw0, which HAS the :443 dnat)
198.51.100.7 -> 02:bd:75:0b:b0:75 (gw1, which has none)
So home-server's published port worked or did not depending on which container
answered ARP last — intermittent, and it would have presented as a flaky test
rather than as a broken scenario.
One public address is one box. Checked against the thing this models rather than
argued from the model: a bridged modem, a single gateway holding the public
address, one network behind it, and every port forward landing on one host at
that address. Two routers on one address is not a topology, it is a collision.
Gateways to the same segment sharing any address are now one router and their
address lists union, so a v6 address declared on only one of the segments it
serves is still carried. Where such declarations disagree on nat, forwardable or
mapping_ttl, validate refuses — one box cannot behave two ways.
the-ordinary-shape now raises 7 machines instead of 8, and gw0 holds the public
address on eth0 while serving home on eth1 and devices on eth2.
The drawings were confusing, and looking at them showed why: a single stack
ordered by depth put a private network far from the public one it sits behind,
so a gateway's link to the outside ran the full height of the picture through
three networks it had nothing to do with — and two such links overlapped, so
they read as one wire.
Now each public network is followed by everything behind it, depth first. Every
gateway is adjacent to the network it serves, every link is a short stub, and
"behind" is shown by INDENTATION rather than by a line to follow. Gaps are sized
to what they hold, so a gap with no gateway in it takes no room. Transit is not
on a boundary — it reaches every public network at once — so it is stated once
at the top instead of drawing a line to each.
Both sources now order by name rather than by the order the source yielded. The
hypervisor cannot know declaration order, and two pictures laid out differently
cannot be compared, which is the whole point of having both.
Fixed while testing: the gap size and the box placement each decided separately
which network a gateway sat above, and disagreed — reserving the gap above one
sibling while drawing the box above the other, which landed a gateway on top of
a machine in an unrelated network. Both now read one map.
Five new tests, run across every scenario: no link crosses a network it does not
touch, no box is drawn inside a network it is not on, a network behind another
is indented inside it, a public network is not split apart by another group, and
both sources lay the same topology out identically.
Found by rendering the pictures and looking at them, which is the only way
a layout fault shows up.
A gateway was placed below its OUTWARD lane, so one serving `home` and
`devices` was drawn straddling `hosting` and an unrelated `cafe`, with its
connection crossing a network it has nothing to do with. Its first attachment
is the segment it faces; the rest are the ones it serves, and it belongs above
the topmost of those. Transit faces every lane and serves none, so it keeps the
old rule.
Also: the live picture kept its attachments sorted alphabetically, which threw
away the outside-first order the placement now depends on. A segment holding
only gateways-in-the-gaps was counted as occupied and drawn full height with
nothing in it. Badges read left to right, in the order the facts are stated.
The gap between lanes is wide enough that a straddling node no longer covers
the lane's own name and ranges.
`mesh-lab diagram` renders a scenario as draw.io, from either source, through
one layout — so a difference between what was asked for and what exists is a
difference you can see.
The shape says what a resource is and is fixed per kind. The badges say what is
true about that particular one and come entirely from metadata: translation,
forwardability, mapping expiry, refuses-inbound, container-or-VM, running. The
interesting properties of a network are exactly the ones with no visual
consequence — a translated address looks identical to an untranslated one.
For the live picture to be a record rather than a restatement, raise now writes
down what it applied: a segment's kind, ranges and MTU on the link; a gateway's
translation, forwardability and expiry on the gateway; inbound: deny on the
machine. Every behavioural tag is written AFTER the thing works, never at
creation — a failed raise leaves wreckage standing on purpose, and a picture of
that wreckage must not badge translation the router never got.
The pairing earned itself immediately: drawn side by side, every virtual machine
held no addresses. A container's interface carries the device's name and a VM
names its own, so joining them by name silently dropped one whole class of
machine. Fixed by joining on MAC.
Also brings tests under the typecheck gate, which caught integration timeouts
being passed as a 4th argument and therefore ignored entirely.
Reviewed and the criticism was right: 1,072 of 2,128 lines untested, all of
it the half that touches the hypervisor, and no gate. The verification I had
done was real — pings across NAT, TTL counts, ruleset comparisons — and none
of it survived the terminal it ran in, which is 04-ISSUES/005 in miniature.
Ten integration tests against a real hypervisor, each named for what it
defends. ADR 0031: a raised machine carries no overlay, no wireguard, no
mesh config — a scenario that pre-built peering would certify its own work.
ADR 0032: exec is the only way in. ADR 0033: routers are containers while
machines are virtual machines. And the design's claims: raise waits for
usable, snapshots are whole-scenario, NAT hides a private address,
published reaches the machine at the gateway's address.
Mocking the hypervisor is forbidden, so they skip with a reason on a
machine that cannot raise scenarios rather than passing green having
checked nothing.
The suite earned itself on its first run. It found that a snapshot of a
running machine could miss a file written seconds earlier — not stale,
absent — because the write was still in the guest's page cache. That is
exactly the question the lifecycle design listed as open: does a scenario
snapshot need the machines stopped? It does not, but it does need them
flushed. snapshot now syncs every machine before capturing, and the design
records the answer.
The fix buys write-durability, not application-consistency: anything
mid-transaction is still captured mid-transaction, and that is now stated
rather than assumed.
npm run check is the gate — typecheck, 40 unit tests, 10 integration tests.
The full topology now raises: four machines, three routers, a transit
router, six segments, in 35 seconds. Everything the declaration model can
express except `place`, which is refused because the node host it would
place does not exist yet.
Transit was a real gap, not a bug. The design says public networks are
unrelated and routed to each other, never bridged — and I built the
segments and never built the thing that routes between them, so three
public networks were islands and nothing crossed. A transit router now
holds an interface on every public segment, forwarding and no translation:
the closest thing the lab has to the internet, deliberately dumb.
Proven rather than asserted, by ping TTL across the raised topology:
within one segment ttl=64 no hops
across two unrelated public networks ttl=62 gateway + transit
multicast between public networks 0 replies
A flat internet would have shown ttl=64 and answered multicast — which
would let a node discover a peer it could never reach in production, and
report success. That is the fault the as-is layer records the mesh already
hitting with multicast name resolution.
inbound: deny is implemented as a host firewall on the machine, read back
after applying. A declared refusal that silently did not load leaves the
machine wide open, which looks exactly like a machine that is working.
Established and related traffic is accepted, so a defended machine can
still dial out rather than being a disconnected one.
Verified by running, all of it:
home -> devices (policy allow) reachable
devices -> home (policy deny) blocked
behind unforwardable NAT -> out reachable
in -> behind unforwardable NAT unreachable
inbound: deny, dialling out reachable
reaching a machine that denies inbound refused
The two routers differ exactly as declared: the forwardable one carries the
policy rule and no inbound drop, the unforwardable one carries `ct state
new drop` and no DNAT.
A gateway is the one implicit machine in a declaration — a scenario says a
segment sits behind one and never names the thing that serves it. This
materialises it.
A router is a container, not a virtual machine, because it is scenery
rather than something under test (hq ADR 0033). Verified before building
that a plain unprivileged container can do all of it: ip_forward and ipv6
forwarding settable, nftables masquerade accepted, and the conntrack
timeouts mapping_ttl depends on both writable. No privileged mode.
Verified by running, on a machine behind a household gateway reached from
one on a routable address:
home-server -> anchor 0% loss, through masquerade
anchor -> 192.168.1.135 (private, direct) unreachable
anchor -> 192.0.2.50:8080 (the GATEWAY) HTTP 200
The last line is the published-but-behind-NAT case research 004 says only
exists in production. It is now a 32-second scenario on a workstation.
Segments sharing a gateway declaration share ONE router — that is what a
VLAN-capable router is, and two routers sharing an external address would
not work anyway.
mapping_ttl is read back after setting rather than assumed. Those sysctls
are not on every kernel, and a scenario that declared an expiring mapping
and silently got a permanent one would be exactly the fault being built
against.
Four bugs found by running it, three of them the same fault — a failure
made invisible.
The router had no route to a package repository, by design, so installing
nftables at raise time could not work. The image is now built once with
temporary connectivity and cached; every scenario after that needs no
network. That failure was hidden behind `|| true`, which is why it took a
raise to find.
The builder then failed on DNS: exec works before a container has an
address, and I had treated usable as ready. It now waits for the thing
actually needed.
The stock Alpine image ships `auto eth0 / inet dhcp` and its boot-time
networking service flushed the static address the scenario set — on eth0
only, so the outside interface came up bare while inside ones were fine.
The image build now neutralises it: a router reconfiguring itself from an
image default is the lab overriding the declaration. `ip addr add … || true`
had hidden this too, and is now `ip addr replace` with no swallow.
And routers were orphaned by destroy, holding their networks open so
destroy reported removing zero segments. They now carry the same machine
tag as everything else, so one query finds an instance's resources.
jschoubben