c79b83c1bc2a4406b4c11d22d1de118d820f0c66
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jschoubben
|
c79b83c1bc |
The base image carries the network tools, and a scenario that grows
A sealed scenario cannot install wireguard-tools any more than it can install a container runtime, so a lab without them cannot test connectivity at all -- which is most of what the mesh does between machines. Installed and not started: what a node runs is the mesh's decision, and a lab that brought the interface up itself would be testing its own setup. growing-mesh exists to be grown. The point is not the third machine, it is that adding one changes every other node's peer list -- so each has to be told again, or the newcomer is on a network nobody else can see. |
||
|
jschoubben
|
88cf89194a |
The lab said nothing was running while two machines were
'incus list' failed because this shell had no permission to reach the daemon, incusOk returned null, and the caller wrote ?? "[]". So 'mesh-lab list' printed 'no scenario instances standing' -- confidently, about a question it had never managed to ask. The comment on incusOk warns about exactly this, in those words: absence and success made indistinguishable. Three of its own callers then did it. Two listings and the live diagram, which would have drawn an empty scenario rather than fail -- a picture that is confidently wrong, which is worse than none. Anything enumerating what exists now goes through enumerate() and throws. incusOk stays right where failure genuinely means no, like instanceExists, and there is a test holding that line so this does not get over-corrected until nothing can be asked at all. Worth noting 'mesh-lab check' already diagnoses this precise cause, down to 'a session that predates it cannot see it'. The diagnosis existed; the listing just never asked for it. |
||
|
jschoubben
|
be176bab2e |
Automate the lab registry: a sealed machine pulls by digest
Closes 04-ISSUES/009. A scenario declares `images:` by tag; the lab stocks a
registry on this workstation where there is a network, raises it inside the
scenario as scenery, and reports the references a declaration pins -- which are
the digests THIS registry assigned, and are not knowable until it is raised.
Verified in a sealed machine, confirmed by ping to have no route out: package,
service including boot state, a container pinned by digest, and an action
inside that container. Applied, idempotent on re-apply, and read back from the
machine rather than from the apply's own report. That is the first time the
container shape has worked in the lab at all, and it was the shape blocking the
substrate bootstrap.
Four faults found by running it, three of them mine and one worth keeping:
The read-back checked that the catalog endpoint answered, by looking for the
substring "repositories" -- which `{"repositories":[]}` also contains. So it
passed on a registry holding nothing, and the failure surfaced much later as a
container that could not be pulled. It now asks for each image's manifest BY
DIGEST, which is what a machine does.
A recursive push needs its destination to exist, or incus copies the source's
contents rather than the source. The data landed one directory too shallow and
the registry found nothing where it looks.
The registry writes its blobs as root through a bind mount, so the workstation
could not remove its own scratch directory afterwards. Whoever made the files
removes them -- the cleanup now runs in a container too. And a cleanup failure
no longer fails a raise that succeeded: the scenario is standing and usable,
and saying otherwise would be a false report.
The base image build did not verify that the runtime trusts the documentation
ranges as plain-HTTP registries. Writing the file is not the daemon honouring
it, and a base image that looks right fails much later, in a sealed scenario,
a long way from its cause. It is now read back from `docker info`.
|
||
|
jschoubben
|
4097ff92c1 |
Repoint ADR references after HQ consolidated 65 records to 23
Comments naming records that no longer exist now point at the consolidated record holding their reasoning -- the four lab records are 0016, a test defends a decision is 0017. |
||
|
jschoubben
|
37c6a0ba21 |
A registry inside the scenario: the mechanism, verified
Issue 009's resolution, proven manually end to end before any of it was written. A sealed machine pulled an image BY DIGEST from a registry on its own segment and ran it; then the host applied all four shapes -- package, service with boot, container from that digest, and an action inside it -- idempotently. That is the first time the container shape has worked anywhere but a workstation, and it was the shape blocking the whole substrate bootstrap. The registry's digests are its own, not Docker Hub's, and that is correct rather than a compromise: ADR 0046 requires a reference that is exact and cannot move, and a digest this registry assigned is both. It is also not a lab workaround -- 0048 names an OCI registry as substrate and 0046 says a first node fetches "upstream, wherever the image ordinarily lives". This IS that upstream, scenery in the same sense the transit router is the internet. The base image now trusts the RFC 5737 and RFC 3849 documentation ranges as plain-HTTP registries. Scoped to those rather than an address because they never route on the real internet, so it cannot make a real machine trust a real registry whatever it is copied onto. Three faults found while verifying, two of them mine: My probe script picked an interface with `ls /sys/class/net | head -1`, which returns docker0 once a runtime exists -- so it addressed the wrong interface and then, because that address overlapped the segment, broke routing on the machine entirely. The lab itself is immune: it matches by MAC, for a related reason it already recorded (bus-position naming on multi-homed machines). And a test that proved nothing: I asserted `sha256:tooshort` is rejected, but its letters fall outside a-f, so it failed the character class rather than the length check. Replaced with hex of the wrong length, after which removing the length check bites. |
||
|
jschoubben
|
d6eef25590 |
The lab can give a sealed machine a container runtime
ADR 0046's open consequence: "the lab needs a way to place images, and the
machine it places them into needs a container runtime, which a sealed scenario
cannot install either."
The runtime half is done, and it is research 012's reframing applied literally
-- fetch at build time on a machine with a network, apply on a target that
needs nothing. `mesh-lab base build` launches a machine WITH a network,
installs a runtime, verifies it by asking the runtime rather than the package
manager, and publishes the result. Measured: ~30s to install, ~60s to publish,
~700MiB, paid once per lab rather than per scenario.
A scenario that places `runtime` or an image is then raised from that base
image, chosen rather than declared -- a scenario says what it needs, not which
image provides it. If the base does not exist it says so and how to build it.
Verified in a genuinely sealed machine (no route out, confirmed by ping):
package, service including the new `boot: enabled`, and action all applied,
were idempotent on a second run, and read back correctly. Those three had never
run anywhere but a workstation.
The image half is NOT done, and testing found why: a digest-pinned image cannot
be placed from an archive. `docker save alpine@sha256:...` produces an archive
with no repo tag, because a repo digest only exists for an image a registry
served -- so it loads dangling and a container declaring that digest reaches
for a registry the machine cannot see.
That collides with ADR 0046, which has the host REFUSE an unpinned image. Tag
refused by the host, digest unusable in the lab: there is currently no
declaration the lab can raise that exercises the container shape at all. Filed
as 04-ISSUES/009, whose resolution is a registry inside the scenario -- which is
what the real mesh does rather than a workaround for the lab.
Also fixed a weak check of my own, which is the same fault in miniature: the
load was tested with `includes("Loaded image")`, a prefix of both `Loaded
image:` and `Loaded image ID:`. So an unusable dangling load reported success
and the failure surfaced later as a container that would not start.
|
||
|
jschoubben
|
16c13807a9 |
place: the host — the lab acquires a consumer
The lab raised an underlay and put nothing on it: correct, and useless, because the thing it exists to test did not exist. Tier 0 now does, so `place: [host]` works and a raised scenario finally contains something. The refusal narrows rather than disappearing. A scenario placing a host and a substrate is told which half is missing, by name — not that `place:` is unsupported when half of it now works. Placement reads back rather than assuming. A file arriving is not a host working, so the binary is run before it is trusted to answer questions, and what it reports is read from the machine (ADR 0035). The binary comes from an explicit path, because the declaration design leaves where artifacts come from open and a search would harden into the answer by accident. The integration test that matters is the one asserting the host reports the MACHINE and not the workstation that placed it. A raised VM and this workstation differ in every capability — root versus uid 1000, a clean init versus a degraded one, no docker versus docker, no wireguard versus wg0 — so a host reporting the wrong machine is obvious here and invisible anywhere else. And the placed host independently confirms ADR 0031: overlay absent on a freshly raised machine. The underlay suite already asserted that by looking for wireguard interfaces; this is a second witness rather than the same check twice. Two tests failed the moment placement worked, which is what they were for. They defended "there is nothing to place yet" while that was true; the decision changed, so they change with it rather than being deleted. Gate: 75 unit, 20 integration. |
||
|
jschoubben
|
715f367147 |
Step 1: an invariant that holds of any raised scenario
The address collision was found by eye. This is the mechanical form of it: no two machines hold one address on one segment. Pure over already-collected facts, so the logic is tested without a hypervisor — including the cases that would make it useless if got wrong: the same address on DIFFERENT segments is normal and must not be reported, and one machine holding an address twice is not two machines. Asserted against whatever the integration suite has standing, read from the hypervisor rather than from the declaration. The declaration is what was accepted, and it was accepted. |
||
|
jschoubben
|
a6b7d67e19 |
Gateways sharing an address are one gateway
Found by asking what gw-devices and gw-home actually were, in a picture that
finally made them easy to see side by side.
planRouters grouped on the exact address list, so `home` declaring a v4 and a v6
address and `devices` declaring only the v4 became two router containers — both
holding 198.51.100.7 on the same segment. The lab raised it without complaint.
Not theoretical. On the raised instance the transit router resolved that one
address to two different MACs across a cache flush:
198.51.100.7 -> 02:c9:16:70:23:29 (gw0, which HAS the :443 dnat)
198.51.100.7 -> 02:bd:75:0b:b0:75 (gw1, which has none)
So home-server's published port worked or did not depending on which container
answered ARP last — intermittent, and it would have presented as a flaky test
rather than as a broken scenario.
One public address is one box. Checked against the thing this models rather than
argued from the model: a bridged modem, a single gateway holding the public
address, one network behind it, and every port forward landing on one host at
that address. Two routers on one address is not a topology, it is a collision.
Gateways to the same segment sharing any address are now one router and their
address lists union, so a v6 address declared on only one of the segments it
serves is still carried. Where such declarations disagree on nat, forwardable or
mapping_ttl, validate refuses — one box cannot behave two ways.
the-ordinary-shape now raises 7 machines instead of 8, and gw0 holds the public
address on eth0 while serving home on eth1 and devices on eth2.
|
||
|
jschoubben
|
54d417fdb2 |
Group each public network with what is behind it
The drawings were confusing, and looking at them showed why: a single stack ordered by depth put a private network far from the public one it sits behind, so a gateway's link to the outside ran the full height of the picture through three networks it had nothing to do with — and two such links overlapped, so they read as one wire. Now each public network is followed by everything behind it, depth first. Every gateway is adjacent to the network it serves, every link is a short stub, and "behind" is shown by INDENTATION rather than by a line to follow. Gaps are sized to what they hold, so a gap with no gateway in it takes no room. Transit is not on a boundary — it reaches every public network at once — so it is stated once at the top instead of drawing a line to each. Both sources now order by name rather than by the order the source yielded. The hypervisor cannot know declaration order, and two pictures laid out differently cannot be compared, which is the whole point of having both. Fixed while testing: the gap size and the box placement each decided separately which network a gateway sat above, and disagreed — reserving the gap above one sibling while drawing the box above the other, which landed a gateway on top of a machine in an unrelated network. Both now read one map. Five new tests, run across every scenario: no link crosses a network it does not touch, no box is drawn inside a network it is not on, a network behind another is indented inside it, a public network is not split apart by another group, and both sources lay the same topology out identically. |
||
|
jschoubben
|
5b8d01eacf |
Lay a gateway on the boundary it serves, not the one it faces
Found by rendering the pictures and looking at them, which is the only way a layout fault shows up. A gateway was placed below its OUTWARD lane, so one serving `home` and `devices` was drawn straddling `hosting` and an unrelated `cafe`, with its connection crossing a network it has nothing to do with. Its first attachment is the segment it faces; the rest are the ones it serves, and it belongs above the topmost of those. Transit faces every lane and serves none, so it keeps the old rule. Also: the live picture kept its attachments sorted alphabetically, which threw away the outside-first order the placement now depends on. A segment holding only gateways-in-the-gaps was counted as occupied and drawn full height with nothing in it. Badges read left to right, in the order the facts are stated. The gap between lanes is wide enough that a straddling node no longer covers the lane's own name and ranges. |
||
|
jschoubben
|
2243618f01 |
Draw a scenario, from the declaration and from the hypervisor
`mesh-lab diagram` renders a scenario as draw.io, from either source, through one layout — so a difference between what was asked for and what exists is a difference you can see. The shape says what a resource is and is fixed per kind. The badges say what is true about that particular one and come entirely from metadata: translation, forwardability, mapping expiry, refuses-inbound, container-or-VM, running. The interesting properties of a network are exactly the ones with no visual consequence — a translated address looks identical to an untranslated one. For the live picture to be a record rather than a restatement, raise now writes down what it applied: a segment's kind, ranges and MTU on the link; a gateway's translation, forwardability and expiry on the gateway; inbound: deny on the machine. Every behavioural tag is written AFTER the thing works, never at creation — a failed raise leaves wreckage standing on purpose, and a picture of that wreckage must not badge translation the router never got. The pairing earned itself immediately: drawn side by side, every virtual machine held no addresses. A container's interface carries the device's name and a VM names its own, so joining them by name silently dropped one whole class of machine. Fixed by joining on MAC. Also brings tests under the typecheck gate, which caught integration timeouts being passed as a 4th argument and therefore ignored entirely. |
||
|
jschoubben
|
ca2bbab836 |
Integration tests, each named for the decision it defends
Reviewed and the criticism was right: 1,072 of 2,128 lines untested, all of it the half that touches the hypervisor, and no gate. The verification I had done was real — pings across NAT, TTL counts, ruleset comparisons — and none of it survived the terminal it ran in, which is 04-ISSUES/005 in miniature. Ten integration tests against a real hypervisor, each named for what it defends. ADR 0031: a raised machine carries no overlay, no wireguard, no mesh config — a scenario that pre-built peering would certify its own work. ADR 0032: exec is the only way in. ADR 0033: routers are containers while machines are virtual machines. And the design's claims: raise waits for usable, snapshots are whole-scenario, NAT hides a private address, published reaches the machine at the gateway's address. Mocking the hypervisor is forbidden, so they skip with a reason on a machine that cannot raise scenarios rather than passing green having checked nothing. The suite earned itself on its first run. It found that a snapshot of a running machine could miss a file written seconds earlier — not stale, absent — because the write was still in the guest's page cache. That is exactly the question the lifecycle design listed as open: does a scenario snapshot need the machines stopped? It does not, but it does need them flushed. snapshot now syncs every machine before capturing, and the design records the answer. The fix buys write-durability, not application-consistency: anything mid-transaction is still captured mid-transaction, and that is now stated rather than assumed. npm run check is the gate — typecheck, 40 unit tests, 10 integration tests. |
||
|
jschoubben
|
5d01006eab |
Transit, host firewalls, and the whole topology raising
The full topology now raises: four machines, three routers, a transit router, six segments, in 35 seconds. Everything the declaration model can express except `place`, which is refused because the node host it would place does not exist yet. Transit was a real gap, not a bug. The design says public networks are unrelated and routed to each other, never bridged — and I built the segments and never built the thing that routes between them, so three public networks were islands and nothing crossed. A transit router now holds an interface on every public segment, forwarding and no translation: the closest thing the lab has to the internet, deliberately dumb. Proven rather than asserted, by ping TTL across the raised topology: within one segment ttl=64 no hops across two unrelated public networks ttl=62 gateway + transit multicast between public networks 0 replies A flat internet would have shown ttl=64 and answered multicast — which would let a node discover a peer it could never reach in production, and report success. That is the fault the as-is layer records the mesh already hitting with multicast name resolution. inbound: deny is implemented as a host firewall on the machine, read back after applying. A declared refusal that silently did not load leaves the machine wide open, which looks exactly like a machine that is working. Established and related traffic is accepted, so a defended machine can still dial out rather than being a disconnected one. Verified by running, all of it: home -> devices (policy allow) reachable devices -> home (policy deny) blocked behind unforwardable NAT -> out reachable in -> behind unforwardable NAT unreachable inbound: deny, dialling out reachable reaching a machine that denies inbound refused The two routers differ exactly as declared: the forwardable one carries the policy rule and no inbound drop, the unforwardable one carries `ct state new drop` and no DNAT. |
||
|
jschoubben
|
a270cd5b02 |
Routers: NAT, port forwarding, policy and mapping expiry
A gateway is the one implicit machine in a declaration — a scenario says a segment sits behind one and never names the thing that serves it. This materialises it. A router is a container, not a virtual machine, because it is scenery rather than something under test (hq ADR 0033). Verified before building that a plain unprivileged container can do all of it: ip_forward and ipv6 forwarding settable, nftables masquerade accepted, and the conntrack timeouts mapping_ttl depends on both writable. No privileged mode. Verified by running, on a machine behind a household gateway reached from one on a routable address: home-server -> anchor 0% loss, through masquerade anchor -> 192.168.1.135 (private, direct) unreachable anchor -> 192.0.2.50:8080 (the GATEWAY) HTTP 200 The last line is the published-but-behind-NAT case research 004 says only exists in production. It is now a 32-second scenario on a workstation. Segments sharing a gateway declaration share ONE router — that is what a VLAN-capable router is, and two routers sharing an external address would not work anyway. mapping_ttl is read back after setting rather than assumed. Those sysctls are not on every kernel, and a scenario that declared an expiring mapping and silently got a permanent one would be exactly the fault being built against. Four bugs found by running it, three of them the same fault — a failure made invisible. The router had no route to a package repository, by design, so installing nftables at raise time could not work. The image is now built once with temporary connectivity and cached; every scenario after that needs no network. That failure was hidden behind `|| true`, which is why it took a raise to find. The builder then failed on DNS: exec works before a container has an address, and I had treated usable as ready. It now waits for the thing actually needed. The stock Alpine image ships `auto eth0 / inet dhcp` and its boot-time networking service flushed the static address the scenario set — on eth0 only, so the outside interface came up bare while inside ones were fine. The image build now neutralises it: a router reconfiguring itself from an image default is the lab overriding the declaration. `ip addr add … || true` had hidden this too, and is now `ip addr replace` with no swallow. And routers were orphaned by destroy, holding their networks open so destroy reported removing zero segments. They now carry the same machine tag as everything else, so one query finds an instance's resources. |
||
|
jschoubben
|
a27d861d3b |
Scenario lifecycle: raise, exec, snapshot, restore, destroy
A declaration goes in and a disposable mesh comes out. Verified on a workstation, not asserted: two machines raised and addressed in 14.6s, snapshot 0.28s, restore-to-usable 11.6s, both families pinging with no loss, and the workstation with no route into any of it. The declaration layer implements the model in full — three positions a machine can be in, keyed on forwardability; gateways carrying the address the world sees them as; both address families; multi-homing; MTU; inter-segment policy. It is validated hard because the failures it prevents are silent: a private range on a public segment produces no error, the mesh simply never forms. Public segments are refused unless they use RFC 5737 or RFC 3849 space, and a range wider than the reserved block is refused too. 33 tests, all offline. The runtime implements less than the model, and refuses the difference. A scenario declaring gateways, published ports, policy, inbound deny or place is rejected at raise with every gap named. Raising it would produce a mesh that silently lacks what it declared, which is the fault this lab exists to catch — 04-ISSUES/003, where a firewall key is declared in five manifests and read by no code. Three bugs found by review and by running it, all of one family: The readiness check truthiness-tested incusOk's return. `exec … true` succeeds with EMPTY output, so every machine reported unreachable while incus exec on it worked perfectly. succeeds() now exists so the mistake is not available, and network delete had the same bug — it counted zero segments removed while removing them. list() split instance from machine on the last dash, so a machine called home-server absorbed half the instance id and destroy found nothing. Resources are now found by the metadata they carry, never by name. restore reported success in 0.79s while the machine's agent was still starting, so the next command failed. Both raise and restore now wait for usable and say how long that took — reporting the earlier number is transport reported as effect, which is the fault the lab is being built to find. Two incus behaviours worth recording. Its CLI reads a YAML definition from stdin when stdin is not a terminal, so a spawned command hangs until the timeout kills it and arrives with empty stderr — a failure with no explanation, on a command that works when typed. And it assigns a MAC at runtime without recording it in device config, so MACs are derived and set explicitly, which the guest needs anyway: it names interfaces by bus position, and matching by name configures the wrong one on a multi-homed machine. No build step; Node strips the types. The lifecycle has no unit tests because a fake hypervisor would assert that the fake behaves as expected, which is the shape of test this project exists to stop shipping. |