jschoubben
babf08b9f8
Every machine in a bed is a clone of one base image, so all of them booted with the same /etc/machine-id. systemd's DHCP client derives its client identifier from that file and dnsmasq keys leases on the identifier rather than the MAC, so four machines with four distinct MACs were handed one address and the host kept one ARP entry for it. Whichever machine last answered an ARP request received everybody's replies. This is the fault behind every run lost to "flaky lab DNS": resolution that works two times in three, pulls that succeed on a retry, and one machine out of four being fine while the rest have no path at all. It survived an earlier diagnosis that blamed resolver ordering, because reordering resolvers on a machine that has just won the ARP race looks exactly like a fix. Each machine is now given its own machine-id before the uplink lease is asked for, and a check after addresses are applied refuses to go on if two machines took the same one — the positive control this never had, since the fault is invisible where it happens and unrecognisable where it surfaces. The egress check also now demands five consecutive lookups rather than one. A single answer is what let a machine resolving one query in three pass and then die twenty minutes later inside a pull. And fresh-mesh: whole-mesh-full's topology with genesis-single's honesty. The four-machine bed loads thirty-four of the mesh's own images onto its machines from the workstation because it does not build them, which is a shape no real installation has and the same fiction the lab removed when it deleted its own registry. This scenario names no images at all. The machines pull what is public, the installer builds the control plane, and the mesh builds the rest — including, last and deliberately, a module on a machine that did not build it. Claude-Session: https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx
107 lines
4.6 KiB
YAML
107 lines
4.6 KiB
YAML
# FOUR FRESH MACHINES AND THE INSTALLER. Nothing is handed to them.
|
|
#
|
|
# This is `whole-mesh-full`'s topology with `genesis-single`'s honesty. The four-machine bed loads
|
|
# thirty-four of the mesh's own images onto its machines from the workstation, because it does not
|
|
# build them — they are produced by hand beside the bed and copied in. That is a shape no real
|
|
# installation has, and it is the same class of fiction the lab already removed once: it used to
|
|
# raise a registry inside the scenario, and a bootstrap that only worked against it went green here
|
|
# and would have failed on any real machine.
|
|
#
|
|
# So this bed hands over NOTHING. There is no `images:` list. Every machine gets a container
|
|
# runtime and the host binary, which are prerequisites of the machine rather than parts of the
|
|
# mesh, and after that the mesh is on its own:
|
|
#
|
|
# - the substrate, the registry and the builder's own dependencies are PULLED from the internet,
|
|
# which is where a bare machine gets them;
|
|
# - the control plane is BUILT, by the builder the installer carries, from a repository and a
|
|
# commit it is told to use;
|
|
# - every module after that is BUILT by the mesh's own builder and published into the mesh's own
|
|
# registry, and a machine that runs one PULLS it from there.
|
|
#
|
|
# That last clause is the thing no bed has ever checked, and it is why this one has four machines
|
|
# rather than one. Genesis puts the builder, the registry and everything they make on ONE machine.
|
|
# A second machine running a mesh-built module has to fetch it from a registry that asks who it is,
|
|
# and nothing yet gives a joined node an account for it. The bed asks anyway, in its own test, so
|
|
# the gap is a named failure rather than an absence.
|
|
#
|
|
# hosting (public, routable) home (private, behind the access point)
|
|
# novox 192.0.2.20 ── anchor ace 10.99.1.10 home server
|
|
# substrate, registry, shanks 10.99.1.20 workstation
|
|
# builder, control plane g14 10.99.1.30 workstation
|
|
#
|
|
# EGRESS IS NOT OPTIONAL HERE. With nothing loaded, a sealed machine stops at the installer's first
|
|
# pull. Every machine has a way out, and it is a SECOND path: each still reaches the rest of the
|
|
# scenario through its declared gateway, and the uplink carries only what leaves the scenario —
|
|
# the public images, and the forge the control plane is cloned from.
|
|
#
|
|
# MESH_LAB_HOST_BINARY=.../mesh-host MESH_LAB_BOOTSTRAP_BINARY=.../mesh-bootstrap
|
|
# MESH_LAB_BUNDLE=.../examples/substrate-first-node.lock
|
|
# MESH_LAB_CATALOG=.../mesh-catalog/modules
|
|
# MESH_LAB_SOURCE=<forge url> MESH_LAB_SOURCE_REF=<commit>
|
|
scenario: fresh-mesh
|
|
|
|
segments:
|
|
# The routable segment. novox lives here; its public address is the broker endpoint every token
|
|
# carries and the overlay hub the home nodes dial.
|
|
hosting:
|
|
kind: public
|
|
cidr: [192.0.2.0/24]
|
|
|
|
# The household segment behind an ordinary home router: masquerades v4 outbound, forwards
|
|
# inbound, expires idle mappings after two minutes.
|
|
home:
|
|
kind: private
|
|
cidr: [10.99.1.0/24]
|
|
gateway:
|
|
to: hosting
|
|
address: [192.0.2.50]
|
|
nat: [v4]
|
|
forwardable: true
|
|
mapping_ttl: 120s
|
|
|
|
machines:
|
|
# The anchor. Raised by the installer into a mesh of one, and then asked to build.
|
|
#
|
|
# Sized for what it actually does here: the store, the broker, the registry, TWO control planes
|
|
# during the pivot, the builder, and a build workspace holding a Node toolchain image and an npm
|
|
# cache. It is NOT sized for the whole novox service set, because this bed does not run one — it
|
|
# proves the machinery that would produce it.
|
|
novox:
|
|
at: { segment: hosting, address: [192.0.2.20] }
|
|
egress: true
|
|
inbound: allow
|
|
memory: 12GiB
|
|
cpus: 6
|
|
disk: 60GiB
|
|
|
|
# Three machines that JOIN. Host binary and a token, nothing else — no bootstrap, no substrate,
|
|
# no registry. They are deliberately small: what they are here to prove is that a joined machine
|
|
# can be given a module the mesh built, which is a question about credentials and not about load.
|
|
ace:
|
|
at: { segment: home, address: [10.99.1.10] }
|
|
egress: true
|
|
inbound: allow
|
|
memory: 4GiB
|
|
cpus: 2
|
|
disk: 25GiB
|
|
shanks:
|
|
at: { segment: home, address: [10.99.1.20] }
|
|
egress: true
|
|
inbound: allow
|
|
memory: 3GiB
|
|
cpus: 2
|
|
disk: 20GiB
|
|
g14:
|
|
at: { segment: home, address: [10.99.1.30] }
|
|
egress: true
|
|
inbound: allow
|
|
memory: 3GiB
|
|
cpus: 2
|
|
disk: 20GiB
|
|
|
|
# **No `images:` key, and that is the whole point of this file.** Anything a machine holds here, it
|
|
# pulled or the mesh built. See the header.
|
|
|
|
place:
|
|
all: [host, runtime]
|