novox/mesh-lab
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
Files
555401a7874ab04b5258cffa3bca0c27887b8bf5
mesh-lab/scenarios/whole-mesh-full.yml
T
jschoubben 555401a787 The bed bootstraps through the installer, not around it 
ADR 0067's own acceptance check said the lab must raise its anchor by running the
program a bare machine runs. It did not: whole-mesh-full applied the substrate bundle
by hand and then looped enrolment over all four machines as one continuous operation.
That gets the order right by accident and models the wrong shape — and an install
procedure that exists only as a test fixture is exercised by whoever writes tests and
never by whoever installs, which is why every bootstrap fault this year was found late.

Two acts now, and the first gates the second.

  GENESIS is novox running mesh-bootstrap: the installer is built from source before
  the raise (make bootstrap, carrying the control-plane image built in the same run),
  placed beside the host binary, given the two manifests it reads, and run. The bed
  then asserts a WORKING MESH OF ONE — the control plane answers, the registry replies
  on /v2/, the container called mesh-control is running from a registry-pinned digest
  rather than an image id, the registry agrees it serves it, temp-mesh-control is gone,
  and the mesh has heard from its node. The image-id check is ADR 0067's "the pivot
  completed" verbatim: if it is still an id, nothing was published and this mesh can
  never roll out its own upgrades.

  JOINING is ace, shanks and g14: host binary, token, enrol, run. novox is NOT enrolled
  again — the installer already did it, and a second identity is one the mesh does not
  know.

If genesis stops, the bed prints which of the installer's ten steps it stopped at and
goes no further. A second machine joining a mesh that is not ready is a different
failure, and running it would bury this one underneath it.

The anchor is no longer handed mesh-control:development. Its absence is the point: the
installer carries that image inside itself, and handing it over as well would make the
load say "already held" and leave the carrying untested — the same class of fiction the
lab's own registry used to hide. A unit test asserts the scenario keeps it out.

The registry is reached at 127.0.0.1:5000, which is a finding rather than a shortcut: a
runtime refuses a plain-HTTP registry at any address but a loopback one, so the digest
the control-plane module is pinned to is one only the anchor can pull. Enough here,
because only the anchor runs a control plane. Written down in the bed.

Claude-Session: https://claude.ai/code/session_01LrgweAeERJYBg88c5cKDzF
2026-09-11 11:46:42 +02:00

220 lines
11 KiB
YAML
Raw Blame History

# The FULL mesh in its REAL production shape: two segments, one access point, one overlay.
#
# This is the first multi-segment whole-mesh bed. The earlier flat whole-mesh-full sat every node
# on one public segment with a SEPARATE `anchor` carrying the substrate. Production is not flat, and
# there is no separate anchor: `novox` IS the anchor. It sits on the routable `hosting` segment,
# runs the substrate (store, broker, control) AND its own service set AND is the overlay hub and the
# public ingress. `ace`, `shanks` and `g14` sit on the household `home` segment BEHIND a NAT gateway
# — the access point — reachable from the outside only through what they dial out to.
#
# hosting (public, routable) home (private, behind the access point)
# novox 192.0.2.20 ── anchor ace 10.99.1.10 home server, media/IoT set
# substrate + novox set shanks 10.99.1.20 workstation (light)
# overlay hub, ingress g14 10.99.1.30 workstation (light)
#
# The `home` gateway masquerades v4 outbound and forwards inbound (an ordinary household router).
# Home nodes reach novox's public 192.0.2.20 by dialling OUT through it: the substrate broker (5671),
# the mesh's own artifact store, and — the thing this bed exists to prove — the WireGuard overlay hub
# (51820/udp). The hub keepalive holds the NAT hole open so the tunnel, once formed, stays up. novox
# cannot initiate to a home node at all; every home↔novox path is either the overlay or a forwarded
# port.
#
# EVERY NODE HAS EGRESS, which is not a convenience. Third-party images — postgres, the whole Mailu
# stack, Plex, everything the modules actually run — are pulled from the internet, because that is
# where a real node gets them. The lab used to serve them from a registry it raised inside the
# scenario; no production mesh has one, so a bootstrap that could only work against it went green
# here and would have failed anywhere else. What is loaded onto a machine now is only what exists in
# no registry at all: the mesh's own images, named per machine below.
#
# Egress is a SECOND path, not a replacement for the topology. Each node still reaches the rest of
# the scenario through its declared gateway — that is where the overlay handshake has to survive a
# masquerade — and the uplink carries only what leaves the scenario entirely.
#
# THE UNPROVEN THING (what the flat beds never tested): does the overlay tunnel FORM across the
# access point — a home node dialling novox's public hub endpoint, the handshake completing through
# the gateway's masquerade? The driving test verifies the WireGuard handshake and cross-segment
# reachability over the overlay explicitly, and reports form-vs-break as its headline.
#
# Substrate-on-novox collides on two host ports the separate-anchor beds never hit: the substrate
# store binds 127.0.0.1:5432 and novox's postgres provider publishes 5432; the substrate broker binds
# 5671 + 127.0.0.1:5672 and novox's lavinmq provider publishes 5672. The driving test REMAPS those two
# provider host publishes off the substrate's ports (consumers reach the providers over the mesh
# network on the container port, so the host side is free to move). Reported as a topology finding.
#
# MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock
# MESH_LAB_BOOTSTRAP_BINARY=.../mesh-bootstrap MESH_LAB_CATALOG=.../mesh-catalog/modules
#
# GENESIS AND JOINING ARE TWO DIFFERENT ACTS, and this bed distinguishes them. novox is brought
# into existence by `mesh-bootstrap` — the same program a bare machine runs — and is afterwards a
# working mesh of one, with a registry and a control plane that is an ordinary module pinned to an
# image that registry serves. ace, shanks and g14 then JOIN it: host binary, token, enrol, run. No
# bootstrap, no substrate, no registry. novox is never enrolled twice, because the installer
# already did it.
#
# The images: are the UNION of the novox set (feat/novox-conversions @ 431310f: the slug + roundcube
# fixes, so only-office/de-spiegel/amqp-email-forwarder now resolve) and the ace media/home set, and
# every one of them must already be BUILT on the workstation — nothing can fetch them.
scenario: whole-mesh-full
segments:
# The routable segment. novox lives here, and the overlay hub endpoint is a public address here.
hosting:
kind: public
cidr: [192.0.2.0/24]
# The household segment behind the access point. Its gateway is an ordinary home router: it
# masquerades v4 outbound, forwards inbound, and expires idle mappings after two minutes — which
# is exactly the NAT hole a WireGuard keepalive has to hold open.
home:
kind: private
cidr: [10.99.1.0/24]
gateway:
to: hosting
address: [192.0.2.50] # what the world sees the household as
nat: [v4]
forwardable: true
mapping_ttl: 120s
machines:
# The anchor: substrate (store, broker, control) + the whole novox service set + overlay hub +
# public ingress. Bigger than the flat bed's novox, because it now carries the substrate too.
novox:
at: { segment: hosting, address: [192.0.2.20] }
egress: true
inbound: allow
memory: 24GiB
cpus: 8
disk: 130GiB
# The proxy, and a runtime per module novox is assigned. step-ca, photos, invoicing, novox.be,
# only-office, de-spiegel, amqp-email-forwarder, registry, firewall and fail2ban are not here
# because they carry no runtime image of their own — what they run is third-party or is the
# node itself.
#
# **mesh-control is NOT here, and its absence is the point** (novox/hq ADR 0067). The anchor is
# brought into existence by the installer, and the installer carries the control plane's image
# inside itself — that is the whole reason a machine that can reach no registry can still raise
# a mesh. Handing it over from the workstation as well would mean the bed never found out
# whether the installer really carries it: the load would say "already held" and the fiction
# would be invisible, which is exactly the class of thing the lab's own registry used to hide.
images:
- mesh-route-proxy:development
- mesh-runtime-postgres:development
- mesh-runtime-redis:development
- mesh-runtime-minio:development
- mesh-runtime-mongodb:development
- mesh-runtime-mssql:development
- mesh-runtime-lavinmq:development
- mesh-runtime-keycloak:development
- mesh-runtime-gitea:development
- mesh-runtime-nextcloud:development
- mesh-runtime-umami:development
- mesh-runtime-verdaccio:development
- mesh-runtime-portainer:development
- mesh-runtime-mailu:development
# The home server: the whole ace media/home set — 24 modules, ~50 containers, several heavy
# (Plex, Home Assistant, Letta, Baserow, the UniFi JVM, mssql). Behind the gateway.
ace:
at: { segment: home, address: [10.99.1.10] }
egress: true
inbound: allow
memory: 18GiB
cpus: 6
disk: 120GiB
# A runtime per module ace is assigned, and nothing of novox's. This is the point of saying it
# per machine: ace has no business holding a Keycloak runtime, and an operator's home server
# would not.
images:
- mesh-runtime-postgres:development
- mesh-runtime-redis:development
- mesh-runtime-mssql:development
- mesh-runtime-portainer:development
- mesh-runtime-sonarr:development
- mesh-runtime-radarr:development
- mesh-runtime-lidarr:development
- mesh-runtime-plex:development
- mesh-runtime-bazarr:development
- mesh-runtime-nzbget:development
- mesh-runtime-qbittorrent:development
- mesh-runtime-jackett:development
- mesh-runtime-ombi:development
- mesh-runtime-tautulli:development
- mesh-runtime-bookshelf:development
- mesh-runtime-home-assistant:development
- mesh-runtime-mosquitto:development
- mesh-runtime-influxdb:development
- mesh-runtime-grafana:development
- mesh-runtime-baserow:development
- mesh-runtime-letta:development
- mesh-runtime-nodered:development
- mesh-runtime-searxng:development
- mesh-runtime-unifi:development
# Two workstations on the same home LAN. Light on purpose: they enrol, join the overlay, and run
# one small module (portainer) so a real module converges on each without heavy load. Same-LAN
# nodes with no overlay endpoint of their own hairpin the hub rather than peering directly, which
# is the normal case and is fine.
shanks:
at: { segment: home, address: [10.99.1.20] }
egress: true
inbound: allow
memory: 3GiB
cpus: 2
disk: 30GiB
# One module, one runtime. Handing these two the whole union would put roughly thirty gigabytes
# of images onto a thirty-gigabyte disk, which is how you learn that "the lab loads everything
# everywhere" was never a description of anything real.
images: [mesh-runtime-portainer:development]
g14:
at: { segment: home, address: [10.99.1.30] }
egress: true
inbound: allow
memory: 3GiB
cpus: 2
disk: 30GiB
images: [mesh-runtime-portainer:development]
# The union of what the machines above ask for. This is the list the workstation must be able to
# export — every entry is one of the mesh's own images, built from source and published nowhere, so
# a machine holds it because it was handed it. Everything else the modules run comes from the
# internet and is not named here at all.
images:
# --- per-module runtimes (union) ---
- mesh-runtime-postgres:development
- mesh-runtime-redis:development
- mesh-runtime-mssql:development
- mesh-runtime-portainer:development
- mesh-runtime-minio:development
- mesh-runtime-mongodb:development
- mesh-runtime-lavinmq:development
- mesh-runtime-keycloak:development
- mesh-runtime-gitea:development
- mesh-runtime-nextcloud:development
- mesh-runtime-umami:development
- mesh-runtime-verdaccio:development
- mesh-runtime-mailu:development
- mesh-route-proxy:development
- mesh-runtime-sonarr:development
- mesh-runtime-radarr:development
- mesh-runtime-lidarr:development
- mesh-runtime-plex:development
- mesh-runtime-bazarr:development
- mesh-runtime-nzbget:development
- mesh-runtime-qbittorrent:development
- mesh-runtime-jackett:development
- mesh-runtime-ombi:development
- mesh-runtime-tautulli:development
- mesh-runtime-bookshelf:development
- mesh-runtime-home-assistant:development
- mesh-runtime-mosquitto:development
- mesh-runtime-influxdb:development
- mesh-runtime-grafana:development
- mesh-runtime-baserow:development
- mesh-runtime-letta:development
- mesh-runtime-nodered:development
- mesh-runtime-searxng:development
- mesh-runtime-unifi:development
place:
all: [host, runtime]
Reference in New Issue View Git Blame Copy Permalink