jschoubben
5d01006eab
The full topology now raises: four machines, three routers, a transit router, six segments, in 35 seconds. Everything the declaration model can express except `place`, which is refused because the node host it would place does not exist yet. Transit was a real gap, not a bug. The design says public networks are unrelated and routed to each other, never bridged — and I built the segments and never built the thing that routes between them, so three public networks were islands and nothing crossed. A transit router now holds an interface on every public segment, forwarding and no translation: the closest thing the lab has to the internet, deliberately dumb. Proven rather than asserted, by ping TTL across the raised topology: within one segment ttl=64 no hops across two unrelated public networks ttl=62 gateway + transit multicast between public networks 0 replies A flat internet would have shown ttl=64 and answered multicast — which would let a node discover a peer it could never reach in production, and report success. That is the fault the as-is layer records the mesh already hitting with multicast name resolution. inbound: deny is implemented as a host firewall on the machine, read back after applying. A declared refusal that silently did not load leaves the machine wide open, which looks exactly like a machine that is working. Established and related traffic is accepted, so a defended machine can still dial out rather than being a disconnected one. Verified by running, all of it: home -> devices (policy allow) reachable devices -> home (policy deny) blocked behind unforwardable NAT -> out reachable in -> behind unforwardable NAT unreachable inbound: deny, dialling out reachable reaching a machine that denies inbound refused The two routers differ exactly as declared: the forwardable one carries the policy rule and no inbound drop, the unforwardable one carries `ct state new drop` and no DNAT.
196 lines
8.3 KiB
Markdown
196 lines
8.3 KiB
Markdown
# mesh-lab
|
||
|
||
The lab: a disposable Novox Mesh on one machine.
|
||
|
||
It ships to nobody. It runs on a workstation, raises virtual machines, puts things inside
|
||
them, and throws them away.
|
||
|
||
## Why it exists first
|
||
|
||
The node host takes over a machine's packages, services and network. It cannot be developed
|
||
against a machine anyone needs — so the place to develop it has to exist before it does.
|
||
|
||
That makes this repository **phase 0** of the migration, ahead of every tier it will later
|
||
test.
|
||
|
||
## Two classes of scenario
|
||
|
||
| | **Bootstrap** | **Full** |
|
||
|---|---|---|
|
||
| Contains | virtual machines, the node host, a pinned substrate bundle | a complete mesh: forge, control plane, delivery, modules |
|
||
| Verdict from | what the host reports about the state it reconciled | a pipeline result ending in verify |
|
||
| Exercises | tiers 0 and 1 | tier 2 and above, and modules |
|
||
| Exists to | **develop the mesh** | **test what runs on it** |
|
||
|
||
The bootstrap scenario is a **strict subset** — same virtualisation, same networking, same
|
||
lifecycle, stopping before a control plane exists. The full scenario is reached by putting more
|
||
inside the machines, not by building a second thing.
|
||
|
||
## Using it
|
||
|
||
```
|
||
mesh-lab check can this machine run scenarios at all
|
||
mesh-lab validate scenarios/x.yml parse and check, raising nothing
|
||
mesh-lab raise scenarios/x.yml materialise it, wait until the machines are USABLE
|
||
mesh-lab list instances currently standing
|
||
mesh-lab exec <instance> <machine> -- <cmd...>
|
||
mesh-lab snapshot <instance> <label>
|
||
mesh-lab restore <instance> <label>
|
||
mesh-lab destroy <instance>
|
||
```
|
||
|
||
`check` refuses rather than warns. A machine without copy-on-write storage runs scenarios
|
||
correctly and snapshots roughly 76× slower — which does not make the lab slow, it makes it
|
||
unused, and a warning about that is read once and ignored forever.
|
||
|
||
If the incus socket is not reachable as your user — the group was granted to a session that
|
||
already existed — set `MESH_LAB_INCUS="sudo -n incus"`.
|
||
|
||
## What a scenario declares
|
||
|
||
The **underlay**: what a hosting provider and a home router would provide, and nothing the
|
||
mesh is responsible for.
|
||
|
||
```yaml
|
||
segments:
|
||
hosting: # one public network
|
||
kind: public
|
||
cidr: [192.0.2.0/24, "2001:db8:a::/48"]
|
||
isp-home: # another, unrelated — routed to it, never bridged
|
||
kind: public
|
||
cidr: [198.51.100.0/24, "2001:db8:b::/48"]
|
||
home:
|
||
kind: private
|
||
cidr: [192.168.1.0/24, "2001:db8:b:1::/64"]
|
||
mtu: 1492
|
||
gateway:
|
||
to: isp-home
|
||
address: [198.51.100.7] # what the world sees this network as
|
||
nat: [v4] # v4 translated, v6 routed
|
||
forwardable: true
|
||
mapping_ttl: 120s
|
||
machines:
|
||
home-server:
|
||
at: { segment: home, address: [192.168.1.135, "2001:db8:b:1::135"] }
|
||
published: [{ port: 443, on: home }]
|
||
inbound: allow
|
||
```
|
||
|
||
It declares **nothing** about overlay addresses, hubs, peering, names or certificates. Those
|
||
are what the mesh does, and a scenario that supplied them would be certifying its own work.
|
||
|
||
Public segments must use documentation ranges (RFC 5737, RFC 3849) and the validator refuses
|
||
anything else **before raising**. That is not pedantry: the mesh decides public-versus-private
|
||
by matching the address, so a private range on a segment meant to be routable makes the mesh
|
||
silently never form — no error, nothing to notice.
|
||
|
||
## Reaching in
|
||
|
||
Everything goes through incus, never over IP. A scenario is a closed address space, so two
|
||
instances raised from one declaration hold the same addresses and never meet — and the
|
||
workstation has no route into either.
|
||
|
||
So a reachability question is asked **from inside**: *can this machine reach that one* is
|
||
`exec` on the first, testing the second. The workstation's opinion would be a different
|
||
question with a misleadingly similar answer.
|
||
|
||
## What is implemented, and what is not
|
||
|
||
The declaration model is complete — it is the design's shape, and validating against it is
|
||
useful before any of it can be raised. **The runtime is not**, and the gap is refused rather
|
||
than ignored:
|
||
|
||
| | |
|
||
|---|---|
|
||
| segments as isolated links | **works** |
|
||
| machines, multi-homed or detached | **works** |
|
||
| declared addresses, both families | **works** |
|
||
| segment MTU | **works** |
|
||
| raise · exec · snapshot · restore · destroy · list | **works** |
|
||
| gateways, NAT, masquerade | **works** |
|
||
| `published:` ports (DNAT through the gateway's address) | **works** |
|
||
| `mapping_ttl:` (conntrack timeout) | **works**, and verified after setting — a declared expiry that silently did not apply would be the fault this catches |
|
||
| `forwardable: false` | **works** — outbound only, no DNAT, unsolicited inbound dropped |
|
||
| `policy:` between segments | **works**, asymmetric |
|
||
| `inbound: deny` | **works** — host firewall, read back after applying |
|
||
| several public networks, routed not bridged | **works** — a transit router, never a shared bridge |
|
||
| `place:` | **refused at raise** — the node host it would place does not exist yet |
|
||
|
||
`raise` refuses a scenario declaring anything in the lower half, naming every gap. It does not
|
||
raise a mesh that silently lacks what it declared — that is the fault this lab exists to catch
|
||
(`novox/hq` 04-ISSUES/003: a firewall key declared in five manifests and read by no code, so a
|
||
manifest appears to restrict a port and restricts nothing).
|
||
|
||
`the-ordinary-shape.yml` therefore validates and does not raise. That is the intended state:
|
||
it is the topology being built toward, and the tool says exactly what is missing.
|
||
|
||
## Measured on a workstation
|
||
|
||
| | one machine | two machines | two machines + a router |
|
||
|---|---|---|---|
|
||
| raise, to usable | 12.5 s | 14.6 s | 32 s |
|
||
| snapshot | 0.14 s | 0.28 s | — |
|
||
| restore, to usable again | 10.5 s | 11.6 s | — |
|
||
|
||
A router adds seconds, not a boot: it is a container, because it is scenery rather than
|
||
something under test (`novox/hq` ADR 0033).
|
||
|
||
**Verified by running**, not asserted — a machine at `192.168.1.135` behind a household
|
||
gateway, reached from a machine on a routable address:
|
||
|
||
```
|
||
home-server -> anchor 0% loss, through masquerade
|
||
anchor -> 192.168.1.135 (private, direct) unreachable ✓
|
||
anchor -> 192.0.2.50:8080 (the GATEWAY) HTTP 200
|
||
home -> devices (policy allow) reachable ✓
|
||
devices -> home (policy deny) blocked ✓
|
||
roamer behind unforwardable NAT -> anchor reachable ✓ (outbound only)
|
||
anchor -> roamer unreachable ✓
|
||
workstation with inbound: deny, dialling out reachable ✓ (defended, not disconnected)
|
||
home-server -> workstation refused ✓
|
||
```
|
||
|
||
The third line is the case research 004 says only exists in production.
|
||
|
||
**Routed, never bridged**, proven rather than asserted — ping TTL across the full topology:
|
||
|
||
```
|
||
within one segment ttl=64 no hops
|
||
across two unrelated public networks ttl=62 gateway + transit
|
||
multicast between public networks 0 replies
|
||
```
|
||
|
||
A flat "internet" would have shown ttl=64 and answered multicast, which would have let a node
|
||
discover a peer it could never reach in production — and report success.
|
||
|
||
Machines boot concurrently, so a second machine costs seconds rather than doubling the wait.
|
||
Nearly all of the remaining time is boot, which cannot be avoided.
|
||
|
||
These numbers depend entirely on a copy-on-write pool. On `dir` the same snapshot takes 9.9 s
|
||
and a full copy of the disk, and a second one did not finish in two minutes — which is why
|
||
`check` refuses rather than warns.
|
||
|
||
## Where the reasoning lives
|
||
|
||
Design and decisions are in [`novox/hq`](https://git.novox.be/novox/hq), not here:
|
||
|
||
- `03-DESIGN/01-to-be/02-scenario-declaration.md` — what a scenario declares
|
||
- `03-DESIGN/01-to-be/03-scenario-lifecycle.md` — what happens to one
|
||
- `02-DECISIONS/0031-the-lab-provides-the-underlay.md`
|
||
- `02-DECISIONS/0032-a-scenario-is-an-isolated-address-space.md`
|
||
|
||
This repository carries implementation. It does not carry decisions.
|
||
|
||
## Development
|
||
|
||
No build step — Node strips the types.
|
||
|
||
```
|
||
npm test the declaration layer, offline
|
||
npm run typecheck
|
||
```
|
||
|
||
The lifecycle is not unit-tested. It talks to a hypervisor, and a fake one would assert that
|
||
the fake behaves as expected — which is the shape of test this project exists to stop
|
||
shipping. It is exercised by raising real scenarios.
|