5d1f7762c2bc4daa122025adcb5c9657beec5e2b
32
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jschoubben
|
5d1f7762c2 |
One archive per machine, not one per image
The images a bed stocks are almost entirely the same bytes: the base they share is 227 MB and a module's own code is a few. Exported one at a time that base is written, pushed and loaded once per image — for this bed, the same 227 MB crossed thirty-odd times, and the whole set measured 9.7 GB. Measured on eight of them: 2.24 GB as separate archives, 0.29 GB as one. 87 per cent less, and it improves with the count. This is the slowest thing a raise does, and the four-machine bed has been exceeding its own ninety-minute limit while still copying — so it was failing on the clock rather than on anything it was testing. Also stops shipping a compiler in every module image. The image runs compiled code and never compiles any; tsc runs on the workstation. Worth 26 MB an image, which is small beside the above but was pure waste. |
||
|
jschoubben
|
fa5e5cb912 |
Configure the resolver rather than fight it
The first attempt wrote /etc/resolv.conf. On these images that is a symlink owned by systemd-resolved, so the file is either reverted or the link is broken — found by reading a running machine instead of assuming the change had worked. The real shape shows in resolvectl: the machine has sensible global fallbacks, and the link carrying the default route has exactly one server, the uplink gateway. resolved will not reach a global fallback while the link has a server of its own, so one unanswered packet is one failed lookup. Three runs have died that way, each long after the egress check passed. The uplink stays first, so the modelled path is still what is used and still what the check proves. Verified on a live machine: three servers on the link, uplink first, resolution intact. |
||
|
jschoubben
|
ea0a7f7e64 |
One dropped lookup should not cost a two-hour run
The egress check proves the uplink resolves and reaches the internet, and that is the right thing to check. What it does not cover is the hours afterwards: a bed pulls images on four machines at once, the uplink's resolver is one server under exactly that load, and three runs have now died at a lookup timeout long after the check passed — the path was never broken, a query just went unanswered. The uplink stays first and keeps proving the path. Public resolvers sit behind it and answer only when it does not, and the retry is tightened so a silent server costs seconds. A genuinely broken uplink still fails the check, before any of this applies. |
||
|
jschoubben
|
555401a787 |
The bed bootstraps through the installer, not around it
ADR 0067's own acceptance check said the lab must raise its anchor by running the program a bare machine runs. It did not: whole-mesh-full applied the substrate bundle by hand and then looped enrolment over all four machines as one continuous operation. That gets the order right by accident and models the wrong shape — and an install procedure that exists only as a test fixture is exercised by whoever writes tests and never by whoever installs, which is why every bootstrap fault this year was found late. Two acts now, and the first gates the second. GENESIS is novox running mesh-bootstrap: the installer is built from source before the raise (make bootstrap, carrying the control-plane image built in the same run), placed beside the host binary, given the two manifests it reads, and run. The bed then asserts a WORKING MESH OF ONE — the control plane answers, the registry replies on /v2/, the container called mesh-control is running from a registry-pinned digest rather than an image id, the registry agrees it serves it, temp-mesh-control is gone, and the mesh has heard from its node. The image-id check is ADR 0067's "the pivot completed" verbatim: if it is still an id, nothing was published and this mesh can never roll out its own upgrades. JOINING is ace, shanks and g14: host binary, token, enrol, run. novox is NOT enrolled again — the installer already did it, and a second identity is one the mesh does not know. If genesis stops, the bed prints which of the installer's ten steps it stopped at and goes no further. A second machine joining a mesh that is not ready is a different failure, and running it would bury this one underneath it. The anchor is no longer handed mesh-control:development. Its absence is the point: the installer carries that image inside itself, and handing it over as well would make the load say "already held" and leave the carrying untested — the same class of fiction the lab's own registry used to hide. A unit test asserts the scenario keeps it out. The registry is reached at 127.0.0.1:5000, which is a finding rather than a shortcut: a runtime refuses a plain-HTTP registry at any address but a loopback one, so the digest the control-plane module is pinned to is one only the anchor can pull. Enough here, because only the anchor runs a control plane. Written down in the bed. Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF |
||
|
jschoubben
|
6c09ddb528 |
An image the machine has no account for is handed over, not fetched
Deleting the lab's registry left the operator's own images to be pulled like anything else, and they cannot be: their registry wants an account and a scenario machine has none. The pull fails with 'no basic auth credentials', which is not something more patience fixes. So the test is no longer 'did the mesh build it' but 'can the machine get it at all'. Two ways to fail that — published nowhere, or published somewhere the machine cannot authenticate to — and one consequence: the workstation, which does hold the credential, exports it and loads it. Worth saying what this stands in for. In a finished mesh these are built by the builder and published to the mesh's own store, and every machine pulls them from there with a credential the mesh granted. Until that store exists there is nowhere for them to come from, and handing them over is the closest honest thing — not a registry the lab invents, which is what was just removed. Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF |
||
|
jschoubben
|
d637c77f08 |
An image id belongs to the machine holding it, so ask the machine
The id is the digest of an image's configuration, and a runtime rewrites that configuration as it loads: this workstation saves in one format, the machines store it in another, and the same bytes arrive under a different name. Measured, not assumed — b86bb81c here, 2dc21904 there. So it is read back from the machine instead of predicted from here. Predicting it failed at the only moment it mattered: every manifest would have been rewritten to a reference no machine holds, and these images exist in no registry, so each apply would have stopped at a pull that cannot succeed. A manifest carries one reference, so machines that disagree stop the raise rather than having one of them silently win. Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF |
||
|
jschoubben
|
751948f0f9 |
The lab had a registry that production does not, so it tested a fiction
The lab raised a `registry` VM, pushed ~73 images into it from the workstation, and rewrote every manifest reference — third-party ones included — to point at it. No production mesh has such a thing. So every bed proved that a machine could fetch an image from a registry that exists nowhere else, and the bootstrap problems that only appear when a machine has to fetch for itself went unfound. What replaces it is the two things that are true in the world: **Public images come from the public internet.** mesh-lab already created a NAT'd uplink for exactly this and attached it to any machine declaring `egress`; no scenario ever declared it. They do now, and third-party references are left exactly as the catalogue writes them. **The mesh's own images have no registry and never will.** mesh-control, mesh-builder, mesh-route-proxy and the per-module runtimes are built from source and exist in no registry. A machine gets them the way an operator's machine does — they are built here and loaded onto it — and is then named by the digest of its own image configuration, which mesh-host now accepts as "an image this machine already holds". `images:` therefore means only *ours*, and a third-party entry is refused rather than quietly loaded: otherwise the fiction returns one convenient line at a time. It is per-machine as well, because "everything, everywhere" was never a description of anything real — handing whole-mesh-full's union to its two 30GiB workstations would fill the disk with runtimes nothing on them will start. **The uplink and the declared gateway would have fought, silently.** A gateway container and the transit router reach the scenario and nothing else; a default route through either is a black hole for anything outside, and it beats the uplink's DHCP route on metric. So a machine with egress states the scenario's ranges explicitly — through the same gateway or transit it would have defaulted to, so the overlay-across-NAT path is unchanged — and leaves the default to the uplink. A range with no path inside the scenario becomes `unreachable` rather than falling through: 192.168.1.0/24 is an ordinary private range in fact, and letting it escape would put scenario traffic on whatever network the workstation is sitting on. `scenarioRoutesFor` is pure and tested, because a decision only a full raise could check is one nobody checks. The registry-reachability check the raise gained earlier is kept, pointed at the real thing: every machine with egress must resolve a name and reach the internet before the raise says it finished. Same failure it was written for — a raise that returns, an apply that dies on its first pull, an instance left a bare shell — now guarding the path that actually carries. The base image's trust of the documentation ranges as plain-HTTP registries STAYS. It was never only for the lab's registry: the mesh has one of its own, the `registry` module, serving artifacts to the whole mesh over plain HTTP from whatever node runs it. Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF |
||
|
jschoubben
|
2f4cb871d9 |
A raise does not finish until the machines can pull from the registry
The first whole-mesh raise of the ADR 0056 code died on the anchor's substrate apply: the image pulls failed, the anchor never came up, no node could enrol, and the instance was left a bare shell — VMs and a registry, no substrate. The identical apply, run by hand once the registry was warm, succeeded immediately. `raiseRegistry` proves the wrong thing. It curls `localhost:5000` from inside the registry's OWN machine, which says the registry process is up and holds the blobs, and says nothing about the path anybody else uses: across a segment, and for the home nodes through a NAT gateway whose default route and firewall are applied two steps LATER. So "serving" was reported on evidence that excluded the network, and the caller — which pins every image in the substrate bundle to that registry — was handed a fact it could not rely on. So the check moves to where it means something. After the routes and the firewalls, before the minutes spent placing, each machine is asked for `/v2/` and for one stocked manifest BY DIGEST, at the address it will pin, over the network it will use. That is the pair of requests a pull begins with, from the same place. Layers are not fetched: every digest was already read back inside the registry machine, so what is in question here is the path, not the content. Verified by typecheck and the unit suite (136 pass), and by confirming against a standing four-node instance that `curl` exists in the machines and that both segments — including a home node through the gateway — answer 200 for the registry's `/v2/`. The ordering itself is unverified in a live raise from cold, which takes hours. |
||
|
jschoubben
|
80b0670ebe |
whole-mesh-full: the real segmented topology, and the overlay proven across the access point
Rewrite the flat three-node whole-mesh-full (separate anchor, one public segment)
into production's real shape: two segments and one access point. novox sits on
the routable `hosting` segment and IS the anchor — it runs the substrate, its own
service set, the overlay hub and public ingress; there is no separate anchor node.
ace, shanks and g14 sit on the household `home` segment behind a NAT gateway,
reachable from outside only through what they dial out to.
The bed drives, and verifies, the thing the flat beds never could: the WireGuard
overlay forming ACROSS the access point — a home node dialling novox's public hub
endpoint out through the gateway's masquerade, the handshake completing through the
NAT, the keepalive holding the hole open. Phase A proves it (handshake state + a
ping over the overlay) before any heavy module lands; Phase B converges both server
sets. With MESH_LAB_KEEP the instance is raised under a fixed id and left standing.
Collapsing the substrate onto novox exposed real facts the separate-anchor beds
never hit, fixed here:
- the substrate bundle advertises the broker at 192.0.2.10 (the old anchor); a
token carries that verbatim as the endpoint a node dials, so with the substrate
on novox it must be novox's own public address. Rewritten at apply (the cert is
fingerprint-pinned, not hostname-checked, so only the address needs correcting).
- the two provider host-port collisions with the co-located substrate: postgres
5432 vs the store's 127.0.0.1:5432, lavinmq 5672 vs the broker's 127.0.0.1:5672.
Both provider host publishes are remapped off the substrate's ports.
And a lab limitation this first large-union bed exposed: the image registry VM took
the profile's default `dir` pool and a ~10GiB root, which the ~28GiB union of both
server sets overflows ("no space left on device"). raiseRegistry now places the
registry on the scenario's copy-on-write pool with a sized (default 80GiB, thin)
root disk, MESH_LAB_REGISTRY_DISK overridable.
Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
|
||
|
jschoubben
|
a9ecce25cb |
Two-node DB-consumer bed + a scenario disk field
The GREEN multi-node regression bed that proves the DB-consumer gate: substrate/control on one node, postgres+redis providers and baserow+letta consumers on another, each consumer getting its own credential and its own mesh-named database across the overlay. Requires the mesh-control provider-seal-key fix and the mesh-catalog db-name fix. Includes a general lab capability: a machine 'disk' field sizing the VM root disk (a broad install exhausts the pool default and the host fails mid-apply with 'no space left on device'). The bed sets 60GiB. Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF |
||
|
jschoubben
|
0d286b7cc8 |
What review found in the lab, fixed
A segment named "uplink" is refused. The lab claims that name for the NAT bridge behind `egress: true`, and a scenario wearing it first would have its egress machines silently attached to an isolated bridge — a declared key doing nothing, which is the fault this repo exists to refuse, in the repo that refuses it. settled() parses inside the try. A truncated status from a struggling machine was the one shape of bad answer that still threw out of the wait, and the likeliest moment for one is exactly the machine the poll is watching. Malformed now counts as "could not ask", like the exec that times out. And a sentence on the uplink's UseDNS saying its inertness is load-bearing: it matters only where systemd-resolved runs, and on a machine whose modules own resolv.conf the uplink must not outvote the resolver a scenario is testing. |
||
|
jschoubben
|
4a343a2652 |
A machine is as big as the scenario says, and may reach the world
Three changes, found by one failing test. The forge failed three runs in a row as "status hangs", and it was diagnosed twice as contention — real defects, fixed, and not the cause. The heartbeats told the truth in the end: every exec on anchor crawled from 15s to 105s, because eleven containers plus a database pull were running in a 1GiB machine. Starvation presents as whatever you were doing when the page-outs start, which is why it wore two other bugs' clothes first. So machine size is now the scenario's to declare — memory and cpus per machine, default unchanged. The anchor that carries the whole substrate is bigger than the laptop that joins it, and the comment on the scenario says why in terms of what lands there. `egress: true` gives a machine one extra interface on a lab-supplied NAT network, addressed by DHCP because the one address a scenario has no business choosing is on the host's side of the fence. Declared per machine and off by default: a closed scenario stays the rule (novox/hq ADR 0016), and the exception exists because a first node fetches its images before any mesh can serve them — which is now the tested path (04-ISSUES/029), and a lab that can never reach upstream cannot prove the bootstrap it exists to prove. The uplink route is metric-4096, so it never shadows a route the scenario declared. A detached machine declaring egress is refused, not ignored. And settled() treats a poll that threw as a poll that missed. An exec timeout at minute four of a wait is "could not ask", not a verdict on the machine. |
||
|
jschoubben
|
eba436b6b7 |
Reach one scenario from the workstation, by name
A scenario is a closed address space: two raised from the same declaration hold the same addresses and never meet, which is what lets two run at once and why the lab talks to machines through the hypervisor rather than over IP. Reaching in from outside breaks that, so it is opt-in, one scenario at a time, and reversible. `connect` takes an address on the scenario's public link and writes a resolver rule answering everything under each machine's name. `disconnect` gives both back. `connected` says what is true right now, for somebody who cannot remember. It refuses rather than guessing when more than one scenario is standing — the failure being avoided is not an error but one scenario's traffic arriving in another. It also refuses when a machine's name is already answered here for something real, because connecting would point that name at the lab, and the damage would land on the real thing. Names answer with the segment address rather than the overlay one. Inside the mesh a name gives a machine's private address; from here that would need this workstation on the overlay, which is a much larger door. The segment address reaches the same machine and the same ports, which is what opening a board in a browser actually needs. Proven against a live two-node scenario: registry.internal:5000/v2/ answered 200 from this workstation, and so did a wildcard name under the same machine. Disconnect put the address back, stopped answering, and left the real mesh's own names alone. One thing measured rather than assumed: it restarts dnsmasq instead of reloading it. A reload is SIGHUP, which re-reads the hosts file and clears the cache but not the configuration — the rule was written, the reload reported success, and nothing resolved. The daemon's start time was nine days old afterwards. |
||
|
jschoubben
|
bb14ecb7e0 |
Say what the lab is doing, while it is doing it
novox/hq 04-ISSUES/024. A run stalled for thirty-five minutes and said nothing. The cause was a link systemd was still configuring, three layers down inside a `docker load` blocked on a socket — and every one of those layers knew what it was waiting for. None of them said so. Three decisions, each doing work. **Every external command is logged, at the three places that run one.** Ninety-seven call sites reach a hypervisor or a container runtime through three wrappers, so instrumenting the wrappers covers all of them and nothing has to remember to log. **A command still running says so while it runs.** A line before and a line after tells you nothing until the after arrives, which is exactly the case that matters. Anything outstanding past fifteen seconds reports itself with how long it has been going. It is reported as still running, not as stuck — which it is is not knowable from there, and a log that calls a slow step a hang teaches people to ignore it. **It goes to a file, written synchronously.** Node block-buffers stdout when redirected and a test runner buffers it again, so a console log can sit minutes behind. `appendFileSync` cannot lag. Two things this found in itself while being written, both the same shape as what it exists to catch: A question that answers no is not a fault. Half the lab's commands are questions — does this network exist, is the agent up yet — and they fail constantly while a scenario comes up. Logging those as faults filled a healthy run with ✗, which is how you end up ignoring ✗ when one is real. They are recorded quietly now, and still recorded. And `around` skipped its own wrapper when a step's level was below the configured one — taking the failure line and the heartbeat with it. The two things worth having at a low level were the two that vanished at exactly the level somebody would use. The gate belongs in `write`. Also unsilences the four call sites that passed a callback throwing everything away, including the one the stall sat in, and tees `raise`'s progress into the file whether or not a caller asked to see it — the end-to-end test passed no callback, so the one run that mattered reported not a single step. |
||
|
jschoubben
|
3503ad990b |
The registry is addressed the way every other machine is
novox/hq 04-ISSUES/024. The registry machine had its address set with `ip addr add`; every other machine gets a systemd-networkd unit. That one difference stalled the lab indefinitely. An address set by hand leaves networkd waiting to configure a link it was never told about, so the link sits at `configuring` for ever. `systemd-networkd-wait-online` has TimeoutStartUSec=infinity, so `network-online.target` is never reached — and Docker is ordered after it. `docker load` then blocked on a socket whose daemon was queued behind a target that would never come. Measured before and after on the same scenario: stuck with five pending systemd jobs and `docker` inactive; now `enp5s0 configured`, `docker` active, no jobs, and the whole raise completes in 87.5s. The guess in the issue was wrong, and it was wrong in the usual way — stocking had just been changed, so stocking looked guilty. Stocking takes 34s and always did. Two things that made this cost hours rather than minutes are fixed with it. Placing an image now waits for the container runtime to answer and refuses after 120s naming what systemd is waiting on, so a stall becomes a failure that says why instead of three stacked timeouts totalling 35 minutes. And the end-to-end test passes `onProgress`, so a raise says what step it is on — it printed nothing at all until it finished, which is why 35 minutes of nothing read as a slow test. |
||
|
jschoubben
|
20690964f1 |
Prove a service is reached by a name under the machine it runs on
Through the path an application actually takes — nsswitch, files, then DNS — because the resolv.conf module is half of what is being tested and only that path goes through it. Asking a server directly would prove less. Both machines resolve, from their own copy: a mesh where one machine answers for all of them stops resolving when that machine does, which is the arrangement this design refuses everywhere else. The manifests are read from mesh-control's examples rather than written here, so what is proven is what ships. And dnsmasq joins the base image, read back through --version like the others: a machine that cannot answer names applies the resolver data, reports success, and resolves nothing. |
||
|
jschoubben
|
29cdaa4de3 |
Prove a machine filters what it was told to and nothing else
Written and loaded are different things, and loaded and enforcing are different again. The test opens two ports on a machine, declares one of them, and checks from the other machine that the declared one answers and the undeclared one does not — then removes the module and checks the port closes with nobody editing a rule. The base image gains nftables, read back through `nft --version` like the other three: a machine that cannot load a rule set applies the mesh's filtering, reports success and filters nothing, which is the exact fault the derivation exists to remove. Two earlier tests were asking for things that are not there. The lab's registry drops tags when it stocks, so `registry:2` is not served and the mirror test failed with "not found" — it now uses the pinned digest, which is what a declaration carries anyway. |
||
|
jschoubben
|
2f81701a13 |
Let a caller say how long to wait, and note where the artifact-store test
went `exec` waited two minutes always. A build, or anything that waits on another machine, needs longer — and a caller that cannot say so has to split the work to fit, which is a test shaped by its harness rather than by what it is testing. The scenario also places a build machine when one is given, so anything in it can ask the mesh to build something. Nothing else here would start one. And the mesh-runs-its-own-artifact-store test is not here. It needs a fourth image so the module has a registry to mirror, and that is caught behind 04-ISSUES/012 — left as a note saying where it went and why, rather than silently deleted, because what it asserted is worth asserting. |
||
|
jschoubben
|
e0127df7ce |
A machine in the mesh builds a module, and the catalogue records it
The chain this closes: a repository exists, the mesh asks for it, a build machine takes the work, publishes what it made, and the catalogue then says what the module is, which commit it came from, and — after the source moves — that it is behind. Three assertions, against a real broker and registry, because what is under test is four processes agreeing over a wire: - the mesh asks, a machine builds, and the artifact is really in the registry at the digest the manifest names - a build that cannot succeed says why and records nothing. A failure that is silent is indistinguishable from a builder that is not running - the source moving makes the catalogue say "behind", and rebuilding catches it up git is now in the base image, with the same reasoning as docker and wireguard-tools: a machine that builds modules clones them, and a sealed scenario cannot install anything. Read back from `git --version` rather than from the package manager — an installed package is not a capability, and a build machine whose clone fails does so three minutes into a scenario with the failure reported as a build problem rather than a lab one. |
||
|
jschoubben
|
c79b83c1bc |
The base image carries the network tools, and a scenario that grows
A sealed scenario cannot install wireguard-tools any more than it can install a container runtime, so a lab without them cannot test connectivity at all -- which is most of what the mesh does between machines. Installed and not started: what a node runs is the mesh's decision, and a lab that brought the interface up itself would be testing its own setup. growing-mesh exists to be grown. The point is not the third machine, it is that adding one changes every other node's peer list -- so each has to be told again, or the newcomer is on a network nobody else can see. |
||
|
jschoubben
|
be176bab2e |
Automate the lab registry: a sealed machine pulls by digest
Closes 04-ISSUES/009. A scenario declares `images:` by tag; the lab stocks a
registry on this workstation where there is a network, raises it inside the
scenario as scenery, and reports the references a declaration pins -- which are
the digests THIS registry assigned, and are not knowable until it is raised.
Verified in a sealed machine, confirmed by ping to have no route out: package,
service including boot state, a container pinned by digest, and an action
inside that container. Applied, idempotent on re-apply, and read back from the
machine rather than from the apply's own report. That is the first time the
container shape has worked in the lab at all, and it was the shape blocking the
substrate bootstrap.
Four faults found by running it, three of them mine and one worth keeping:
The read-back checked that the catalog endpoint answered, by looking for the
substring "repositories" -- which `{"repositories":[]}` also contains. So it
passed on a registry holding nothing, and the failure surfaced much later as a
container that could not be pulled. It now asks for each image's manifest BY
DIGEST, which is what a machine does.
A recursive push needs its destination to exist, or incus copies the source's
contents rather than the source. The data landed one directory too shallow and
the registry found nothing where it looks.
The registry writes its blobs as root through a bind mount, so the workstation
could not remove its own scratch directory afterwards. Whoever made the files
removes them -- the cleanup now runs in a container too. And a cleanup failure
no longer fails a raise that succeeded: the scenario is standing and usable,
and saying otherwise would be a false report.
The base image build did not verify that the runtime trusts the documentation
ranges as plain-HTTP registries. Writing the file is not the daemon honouring
it, and a base image that looks right fails much later, in a sealed scenario,
a long way from its cause. It is now read back from `docker info`.
|
||
|
jschoubben
|
4097ff92c1 |
Repoint ADR references after HQ consolidated 65 records to 23
Comments naming records that no longer exist now point at the consolidated record holding their reasoning -- the four lab records are 0016, a test defends a decision is 0017. |
||
|
jschoubben
|
37c6a0ba21 |
A registry inside the scenario: the mechanism, verified
Issue 009's resolution, proven manually end to end before any of it was written. A sealed machine pulled an image BY DIGEST from a registry on its own segment and ran it; then the host applied all four shapes -- package, service with boot, container from that digest, and an action inside it -- idempotently. That is the first time the container shape has worked anywhere but a workstation, and it was the shape blocking the whole substrate bootstrap. The registry's digests are its own, not Docker Hub's, and that is correct rather than a compromise: ADR 0046 requires a reference that is exact and cannot move, and a digest this registry assigned is both. It is also not a lab workaround -- 0048 names an OCI registry as substrate and 0046 says a first node fetches "upstream, wherever the image ordinarily lives". This IS that upstream, scenery in the same sense the transit router is the internet. The base image now trusts the RFC 5737 and RFC 3849 documentation ranges as plain-HTTP registries. Scoped to those rather than an address because they never route on the real internet, so it cannot make a real machine trust a real registry whatever it is copied onto. Three faults found while verifying, two of them mine: My probe script picked an interface with `ls /sys/class/net | head -1`, which returns docker0 once a runtime exists -- so it addressed the wrong interface and then, because that address overlapped the segment, broke routing on the machine entirely. The lab itself is immune: it matches by MAC, for a related reason it already recorded (bus-position naming on multi-homed machines). And a test that proved nothing: I asserted `sha256:tooshort` is rejected, but its letters fall outside a-f, so it failed the character class rather than the length check. Replaced with hex of the wrong length, after which removing the length check bites. |
||
|
jschoubben
|
d6eef25590 |
The lab can give a sealed machine a container runtime
ADR 0046's open consequence: "the lab needs a way to place images, and the
machine it places them into needs a container runtime, which a sealed scenario
cannot install either."
The runtime half is done, and it is research 012's reframing applied literally
-- fetch at build time on a machine with a network, apply on a target that
needs nothing. `mesh-lab base build` launches a machine WITH a network,
installs a runtime, verifies it by asking the runtime rather than the package
manager, and publishes the result. Measured: ~30s to install, ~60s to publish,
~700MiB, paid once per lab rather than per scenario.
A scenario that places `runtime` or an image is then raised from that base
image, chosen rather than declared -- a scenario says what it needs, not which
image provides it. If the base does not exist it says so and how to build it.
Verified in a genuinely sealed machine (no route out, confirmed by ping):
package, service including the new `boot: enabled`, and action all applied,
were idempotent on a second run, and read back correctly. Those three had never
run anywhere but a workstation.
The image half is NOT done, and testing found why: a digest-pinned image cannot
be placed from an archive. `docker save alpine@sha256:...` produces an archive
with no repo tag, because a repo digest only exists for an image a registry
served -- so it loads dangling and a container declaring that digest reaches
for a registry the machine cannot see.
That collides with ADR 0046, which has the host REFUSE an unpinned image. Tag
refused by the host, digest unusable in the lab: there is currently no
declaration the lab can raise that exercises the container shape at all. Filed
as 04-ISSUES/009, whose resolution is a registry inside the scenario -- which is
what the real mesh does rather than a workaround for the lab.
Also fixed a weak check of my own, which is the same fault in miniature: the
load was tested with `includes("Loaded image")`, a prefix of both `Loaded
image:` and `Loaded image ID:`. So an unusable dangling load reported success
and the failure surfaced later as a container that would not start.
|
||
|
jschoubben
|
16c13807a9 |
place: the host — the lab acquires a consumer
The lab raised an underlay and put nothing on it: correct, and useless, because the thing it exists to test did not exist. Tier 0 now does, so `place: [host]` works and a raised scenario finally contains something. The refusal narrows rather than disappearing. A scenario placing a host and a substrate is told which half is missing, by name — not that `place:` is unsupported when half of it now works. Placement reads back rather than assuming. A file arriving is not a host working, so the binary is run before it is trusted to answer questions, and what it reports is read from the machine (ADR 0035). The binary comes from an explicit path, because the declaration design leaves where artifacts come from open and a search would harden into the answer by accident. The integration test that matters is the one asserting the host reports the MACHINE and not the workstation that placed it. A raised VM and this workstation differ in every capability — root versus uid 1000, a clean init versus a degraded one, no docker versus docker, no wireguard versus wg0 — so a host reporting the wrong machine is obvious here and invisible anywhere else. And the placed host independently confirms ADR 0031: overlay absent on a freshly raised machine. The underlay suite already asserted that by looking for wireguard interfaces; this is a second witness rather than the same check twice. Two tests failed the moment placement worked, which is what they were for. They defended "there is nothing to place yet" while that was true; the decision changed, so they change with it rather than being deleted. Gate: 75 unit, 20 integration. |
||
|
jschoubben
|
715f367147 |
Step 1: an invariant that holds of any raised scenario
The address collision was found by eye. This is the mechanical form of it: no two machines hold one address on one segment. Pure over already-collected facts, so the logic is tested without a hypervisor — including the cases that would make it useless if got wrong: the same address on DIFFERENT segments is normal and must not be reported, and one machine holding an address twice is not two machines. Asserted against whatever the integration suite has standing, read from the hypervisor rather than from the declaration. The declaration is what was accepted, and it was accepted. |
||
|
jschoubben
|
a6b7d67e19 |
Gateways sharing an address are one gateway
Found by asking what gw-devices and gw-home actually were, in a picture that
finally made them easy to see side by side.
planRouters grouped on the exact address list, so `home` declaring a v4 and a v6
address and `devices` declaring only the v4 became two router containers — both
holding 198.51.100.7 on the same segment. The lab raised it without complaint.
Not theoretical. On the raised instance the transit router resolved that one
address to two different MACs across a cache flush:
198.51.100.7 -> 02:c9:16:70:23:29 (gw0, which HAS the :443 dnat)
198.51.100.7 -> 02:bd:75:0b:b0:75 (gw1, which has none)
So home-server's published port worked or did not depending on which container
answered ARP last — intermittent, and it would have presented as a flaky test
rather than as a broken scenario.
One public address is one box. Checked against the thing this models rather than
argued from the model: a bridged modem, a single gateway holding the public
address, one network behind it, and every port forward landing on one host at
that address. Two routers on one address is not a topology, it is a collision.
Gateways to the same segment sharing any address are now one router and their
address lists union, so a v6 address declared on only one of the segments it
serves is still carried. Where such declarations disagree on nat, forwardable or
mapping_ttl, validate refuses — one box cannot behave two ways.
the-ordinary-shape now raises 7 machines instead of 8, and gw0 holds the public
address on eth0 while serving home on eth1 and devices on eth2.
|
||
|
jschoubben
|
2243618f01 |
Draw a scenario, from the declaration and from the hypervisor
`mesh-lab diagram` renders a scenario as draw.io, from either source, through one layout — so a difference between what was asked for and what exists is a difference you can see. The shape says what a resource is and is fixed per kind. The badges say what is true about that particular one and come entirely from metadata: translation, forwardability, mapping expiry, refuses-inbound, container-or-VM, running. The interesting properties of a network are exactly the ones with no visual consequence — a translated address looks identical to an untranslated one. For the live picture to be a record rather than a restatement, raise now writes down what it applied: a segment's kind, ranges and MTU on the link; a gateway's translation, forwardability and expiry on the gateway; inbound: deny on the machine. Every behavioural tag is written AFTER the thing works, never at creation — a failed raise leaves wreckage standing on purpose, and a picture of that wreckage must not badge translation the router never got. The pairing earned itself immediately: drawn side by side, every virtual machine held no addresses. A container's interface carries the device's name and a VM names its own, so joining them by name silently dropped one whole class of machine. Fixed by joining on MAC. Also brings tests under the typecheck gate, which caught integration timeouts being passed as a 4th argument and therefore ignored entirely. |
||
|
jschoubben
|
ca2bbab836 |
Integration tests, each named for the decision it defends
Reviewed and the criticism was right: 1,072 of 2,128 lines untested, all of it the half that touches the hypervisor, and no gate. The verification I had done was real — pings across NAT, TTL counts, ruleset comparisons — and none of it survived the terminal it ran in, which is 04-ISSUES/005 in miniature. Ten integration tests against a real hypervisor, each named for what it defends. ADR 0031: a raised machine carries no overlay, no wireguard, no mesh config — a scenario that pre-built peering would certify its own work. ADR 0032: exec is the only way in. ADR 0033: routers are containers while machines are virtual machines. And the design's claims: raise waits for usable, snapshots are whole-scenario, NAT hides a private address, published reaches the machine at the gateway's address. Mocking the hypervisor is forbidden, so they skip with a reason on a machine that cannot raise scenarios rather than passing green having checked nothing. The suite earned itself on its first run. It found that a snapshot of a running machine could miss a file written seconds earlier — not stale, absent — because the write was still in the guest's page cache. That is exactly the question the lifecycle design listed as open: does a scenario snapshot need the machines stopped? It does not, but it does need them flushed. snapshot now syncs every machine before capturing, and the design records the answer. The fix buys write-durability, not application-consistency: anything mid-transaction is still captured mid-transaction, and that is now stated rather than assumed. npm run check is the gate — typecheck, 40 unit tests, 10 integration tests. |
||
|
jschoubben
|
5d01006eab |
Transit, host firewalls, and the whole topology raising
The full topology now raises: four machines, three routers, a transit router, six segments, in 35 seconds. Everything the declaration model can express except `place`, which is refused because the node host it would place does not exist yet. Transit was a real gap, not a bug. The design says public networks are unrelated and routed to each other, never bridged — and I built the segments and never built the thing that routes between them, so three public networks were islands and nothing crossed. A transit router now holds an interface on every public segment, forwarding and no translation: the closest thing the lab has to the internet, deliberately dumb. Proven rather than asserted, by ping TTL across the raised topology: within one segment ttl=64 no hops across two unrelated public networks ttl=62 gateway + transit multicast between public networks 0 replies A flat internet would have shown ttl=64 and answered multicast — which would let a node discover a peer it could never reach in production, and report success. That is the fault the as-is layer records the mesh already hitting with multicast name resolution. inbound: deny is implemented as a host firewall on the machine, read back after applying. A declared refusal that silently did not load leaves the machine wide open, which looks exactly like a machine that is working. Established and related traffic is accepted, so a defended machine can still dial out rather than being a disconnected one. Verified by running, all of it: home -> devices (policy allow) reachable devices -> home (policy deny) blocked behind unforwardable NAT -> out reachable in -> behind unforwardable NAT unreachable inbound: deny, dialling out reachable reaching a machine that denies inbound refused The two routers differ exactly as declared: the forwardable one carries the policy rule and no inbound drop, the unforwardable one carries `ct state new drop` and no DNAT. |
||
|
jschoubben
|
a270cd5b02 |
Routers: NAT, port forwarding, policy and mapping expiry
A gateway is the one implicit machine in a declaration — a scenario says a segment sits behind one and never names the thing that serves it. This materialises it. A router is a container, not a virtual machine, because it is scenery rather than something under test (hq ADR 0033). Verified before building that a plain unprivileged container can do all of it: ip_forward and ipv6 forwarding settable, nftables masquerade accepted, and the conntrack timeouts mapping_ttl depends on both writable. No privileged mode. Verified by running, on a machine behind a household gateway reached from one on a routable address: home-server -> anchor 0% loss, through masquerade anchor -> 192.168.1.135 (private, direct) unreachable anchor -> 192.0.2.50:8080 (the GATEWAY) HTTP 200 The last line is the published-but-behind-NAT case research 004 says only exists in production. It is now a 32-second scenario on a workstation. Segments sharing a gateway declaration share ONE router — that is what a VLAN-capable router is, and two routers sharing an external address would not work anyway. mapping_ttl is read back after setting rather than assumed. Those sysctls are not on every kernel, and a scenario that declared an expiring mapping and silently got a permanent one would be exactly the fault being built against. Four bugs found by running it, three of them the same fault — a failure made invisible. The router had no route to a package repository, by design, so installing nftables at raise time could not work. The image is now built once with temporary connectivity and cached; every scenario after that needs no network. That failure was hidden behind `|| true`, which is why it took a raise to find. The builder then failed on DNS: exec works before a container has an address, and I had treated usable as ready. It now waits for the thing actually needed. The stock Alpine image ships `auto eth0 / inet dhcp` and its boot-time networking service flushed the static address the scenario set — on eth0 only, so the outside interface came up bare while inside ones were fine. The image build now neutralises it: a router reconfiguring itself from an image default is the lab overriding the declaration. `ip addr add … || true` had hidden this too, and is now `ip addr replace` with no swallow. And routers were orphaned by destroy, holding their networks open so destroy reported removing zero segments. They now carry the same machine tag as everything else, so one query finds an instance's resources. |
||
|
jschoubben
|
a27d861d3b |
Scenario lifecycle: raise, exec, snapshot, restore, destroy
A declaration goes in and a disposable mesh comes out. Verified on a workstation, not asserted: two machines raised and addressed in 14.6s, snapshot 0.28s, restore-to-usable 11.6s, both families pinging with no loss, and the workstation with no route into any of it. The declaration layer implements the model in full — three positions a machine can be in, keyed on forwardability; gateways carrying the address the world sees them as; both address families; multi-homing; MTU; inter-segment policy. It is validated hard because the failures it prevents are silent: a private range on a public segment produces no error, the mesh simply never forms. Public segments are refused unless they use RFC 5737 or RFC 3849 space, and a range wider than the reserved block is refused too. 33 tests, all offline. The runtime implements less than the model, and refuses the difference. A scenario declaring gateways, published ports, policy, inbound deny or place is rejected at raise with every gap named. Raising it would produce a mesh that silently lacks what it declared, which is the fault this lab exists to catch — 04-ISSUES/003, where a firewall key is declared in five manifests and read by no code. Three bugs found by review and by running it, all of one family: The readiness check truthiness-tested incusOk's return. `exec … true` succeeds with EMPTY output, so every machine reported unreachable while incus exec on it worked perfectly. succeeds() now exists so the mistake is not available, and network delete had the same bug — it counted zero segments removed while removing them. list() split instance from machine on the last dash, so a machine called home-server absorbed half the instance id and destroy found nothing. Resources are now found by the metadata they carry, never by name. restore reported success in 0.79s while the machine's agent was still starting, so the next command failed. Both raise and restore now wait for usable and say how long that took — reporting the earlier number is transport reported as effect, which is the fault the lab is being built to find. Two incus behaviours worth recording. Its CLI reads a YAML definition from stdin when stdin is not a terminal, so a spawned command hangs until the timeout kills it and arrives with empty stderr — a failure with no explanation, on a command that works when typed. And it assigns a MAC at runtime without recording it in device config, so MACs are derived and set explicitly, which the guest needs anyway: it names interfaces by bus position, and matching by name configures the wrong one on a multi-homed machine. No build step; Node strips the types. The lifecycle has no unit tests because a fake hypervisor would assert that the fake behaves as expected, which is the shape of test this project exists to stop shipping. |