# The FULL mesh in its REAL production shape: two segments, one access point, one overlay. # # This is the first multi-segment whole-mesh bed. The earlier flat whole-mesh-full sat every node # on one public segment with a SEPARATE `anchor` carrying the foundation. Production is not flat, and # there is no separate anchor: `novox` IS the anchor. It sits on the routable `hosting` segment, # runs the foundation (store, broker, control) AND its own service set AND is the overlay hub and the # public ingress. `ace`, `shanks` and `g14` sit on the household `home` segment BEHIND a NAT gateway # — the access point — reachable from the outside only through what they dial out to. # # hosting (public, routable) home (private, behind the access point) # novox 192.0.2.20 ── anchor ace 10.99.1.10 home server, media/IoT set # foundation + novox set shanks 10.99.1.20 workstation (light) # overlay hub, ingress g14 10.99.1.30 workstation (light) # # The `home` gateway masquerades v4 outbound and forwards inbound (an ordinary household router). # Home nodes reach novox's public 192.0.2.20 by dialling OUT through it: the foundation broker (5671), # the mesh's own artifact store, and — the thing this bed exists to prove — the WireGuard overlay hub # (51820/udp). The hub keepalive holds the NAT hole open so the tunnel, once formed, stays up. novox # cannot initiate to a home node at all; every home↔novox path is either the overlay or a forwarded # port. # # EVERY NODE HAS EGRESS, which is not a convenience. Third-party images — postgres, the whole Mailu # stack, Plex, everything the modules actually run — are pulled from the internet, because that is # where a real node gets them. The lab used to serve them from a registry it raised inside the # scenario; no production mesh has one, so a bootstrap that could only work against it went green # here and would have failed anywhere else. What is loaded onto a machine now is only what exists in # no registry at all: the mesh's own images, named per machine below. # # Egress is a SECOND path, not a replacement for the topology. Each node still reaches the rest of # the scenario through its declared gateway — that is where the overlay handshake has to survive a # masquerade — and the uplink carries only what leaves the scenario entirely. # # THE UNPROVEN THING (what the flat beds never tested): does the overlay tunnel FORM across the # access point — a home node dialling novox's public hub endpoint, the handshake completing through # the gateway's masquerade? The driving test verifies the WireGuard handshake and cross-segment # reachability over the overlay explicitly, and reports form-vs-break as its headline. # # Foundation-on-novox collides on two host ports the separate-anchor beds never hit: the foundation # store binds 127.0.0.1:5432 and novox's postgres provider publishes 5432; the foundation broker binds # 5671 + 127.0.0.1:5672 and novox's lavinmq provider publishes 5672. The driving test REMAPS those two # provider host publishes off the foundation's ports (consumers reach the providers over the mesh # network on the container port, so the host side is free to move). Reported as a topology finding. # # MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BUNDLE=.../examples/foundation-first-node.lock # MESH_LAB_BOOTSTRAP_BINARY=.../mesh-bootstrap MESH_LAB_CATALOG=.../mesh-catalog/modules # # GENESIS AND JOINING ARE TWO DIFFERENT ACTS, and this bed distinguishes them. novox is brought # into existence by `mesh-bootstrap` — the same program a bare machine runs — and is afterwards a # working mesh of one, with a registry and a control plane that is an ordinary module pinned to an # image that registry serves. ace, shanks and g14 then JOIN it: host binary, token, enrol, run. No # bootstrap, no foundation, no registry. novox is never enrolled twice, because the installer # already did it. # # The images: are the UNION of the novox set (feat/novox-conversions @ 431310f: the slug + roundcube # fixes, so only-office/de-spiegel/amqp-email-forwarder now resolve) and the ace media/home set, and # every one of them must already be BUILT on the workstation — nothing can fetch them. scenario: whole-mesh-full segments: # The routable segment. novox lives here, and the overlay hub endpoint is a public address here. hosting: kind: public cidr: [192.0.2.0/24] # The household segment behind the access point. Its gateway is an ordinary home router: it # masquerades v4 outbound, forwards inbound, and expires idle mappings after two minutes — which # is exactly the NAT hole a WireGuard keepalive has to hold open. home: kind: private cidr: [10.99.1.0/24] gateway: to: hosting address: [192.0.2.50] # what the world sees the household as nat: [v4] forwardable: true mapping_ttl: 120s machines: # The anchor: foundation (store, broker, control) + the whole novox service set + overlay hub + # public ingress. Bigger than the flat bed's novox, because it now carries the foundation too. novox: at: { segment: hosting, address: [192.0.2.20] } egress: true inbound: allow memory: 24GiB cpus: 8 disk: 130GiB # The proxy, and a runtime per module novox is assigned. step-ca, photos, invoicing, novox.be, # only-office, de-spiegel, amqp-email-forwarder, registry, firewall and fail2ban are not here # because they carry no runtime image of their own — what they run is third-party or is the # node itself. # # **mesh-controller is NOT here, and its absence is the point** (novox/hq ADR 0067). The anchor is # brought into existence by the installer, and the installer carries the control plane's image # inside itself — that is the whole reason a machine that can reach no registry can still raise # a mesh. Handing it over from the workstation as well would mean the bed never found out # whether the installer really carries it: the load would say "already held" and the fiction # would be invisible, which is exactly the class of thing the lab's own registry used to hide. images: - mesh-route-proxy:development - mesh-runtime-postgres:development - mesh-runtime-redis:development - mesh-runtime-minio:development - mesh-runtime-mongodb:development - mesh-runtime-mssql:development - mesh-runtime-lavinmq:development - mesh-runtime-keycloak:development - mesh-runtime-gitea:development - mesh-runtime-nextcloud:development - mesh-runtime-umami:development - mesh-runtime-verdaccio:development - mesh-runtime-portainer:development - mesh-runtime-mailu:development # The home server: the whole ace media/home set — 24 modules, ~50 containers, several heavy # (Plex, Home Assistant, Letta, Baserow, the UniFi JVM, mssql). Behind the gateway. ace: at: { segment: home, address: [10.99.1.10] } egress: true inbound: allow memory: 18GiB cpus: 6 disk: 120GiB # A runtime per module ace is assigned, and nothing of novox's. This is the point of saying it # per machine: ace has no business holding a Keycloak runtime, and an operator's home server # would not. images: - mesh-runtime-postgres:development - mesh-runtime-redis:development - mesh-runtime-mssql:development - mesh-runtime-portainer:development - mesh-runtime-sonarr:development - mesh-runtime-radarr:development - mesh-runtime-lidarr:development - mesh-runtime-plex:development - mesh-runtime-bazarr:development - mesh-runtime-nzbget:development - mesh-runtime-qbittorrent:development - mesh-runtime-jackett:development - mesh-runtime-ombi:development - mesh-runtime-tautulli:development - mesh-runtime-bookshelf:development - mesh-runtime-home-assistant:development - mesh-runtime-mosquitto:development - mesh-runtime-influxdb:development - mesh-runtime-grafana:development - mesh-runtime-baserow:development - mesh-runtime-letta:development - mesh-runtime-nodered:development - mesh-runtime-searxng:development - mesh-runtime-unifi:development # Two workstations on the same home LAN. Light on purpose: they enrol, join the overlay, and run # one small module (portainer) so a real module converges on each without heavy load. Same-LAN # nodes with no overlay endpoint of their own hairpin the hub rather than peering directly, which # is the normal case and is fine. shanks: at: { segment: home, address: [10.99.1.20] } egress: true inbound: allow memory: 3GiB cpus: 2 disk: 30GiB # One module, one runtime. Handing these two the whole union would put roughly thirty gigabytes # of images onto a thirty-gigabyte disk, which is how you learn that "the lab loads everything # everywhere" was never a description of anything real. images: [mesh-runtime-portainer:development] g14: at: { segment: home, address: [10.99.1.30] } egress: true inbound: allow memory: 3GiB cpus: 2 disk: 30GiB images: [mesh-runtime-portainer:development] # The union of what the machines above ask for. This is the list the workstation must be able to # export — every entry is one of the mesh's own images, built from source and published nowhere, so # a machine holds it because it was handed it. Everything else the modules run comes from the # internet and is not named here at all. images: # --- per-module runtimes (union) --- - mesh-runtime-postgres:development - mesh-runtime-redis:development - mesh-runtime-mssql:development - mesh-runtime-portainer:development - mesh-runtime-minio:development - mesh-runtime-mongodb:development - mesh-runtime-lavinmq:development - mesh-runtime-keycloak:development - mesh-runtime-gitea:development - mesh-runtime-nextcloud:development - mesh-runtime-umami:development - mesh-runtime-verdaccio:development - mesh-runtime-mailu:development - mesh-route-proxy:development - mesh-runtime-sonarr:development - mesh-runtime-radarr:development - mesh-runtime-lidarr:development - mesh-runtime-plex:development - mesh-runtime-bazarr:development - mesh-runtime-nzbget:development - mesh-runtime-qbittorrent:development - mesh-runtime-jackett:development - mesh-runtime-ombi:development - mesh-runtime-tautulli:development - mesh-runtime-bookshelf:development - mesh-runtime-home-assistant:development - mesh-runtime-mosquitto:development - mesh-runtime-influxdb:development - mesh-runtime-grafana:development - mesh-runtime-baserow:development - mesh-runtime-letta:development - mesh-runtime-nodered:development - mesh-runtime-searxng:development - mesh-runtime-unifi:development place: all: [host, runtime]