novox/mesh-lab
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Transit, host firewalls, and the whole topology raising

Browse Source 
The full topology now raises: four machines, three routers, a transit
router, six segments, in 35 seconds. Everything the declaration model can
express except `place`, which is refused because the node host it would
place does not exist yet.

Transit was a real gap, not a bug. The design says public networks are
unrelated and routed to each other, never bridged — and I built the
segments and never built the thing that routes between them, so three
public networks were islands and nothing crossed. A transit router now
holds an interface on every public segment, forwarding and no translation:
the closest thing the lab has to the internet, deliberately dumb.

Proven rather than asserted, by ping TTL across the raised topology:

  within one segment                     ttl=64   no hops
  across two unrelated public networks   ttl=62   gateway + transit
  multicast between public networks      0 replies

A flat internet would have shown ttl=64 and answered multicast — which
would let a node discover a peer it could never reach in production, and
report success. That is the fault the as-is layer records the mesh already
hitting with multicast name resolution.

inbound: deny is implemented as a host firewall on the machine, read back
after applying. A declared refusal that silently did not load leaves the
machine wide open, which looks exactly like a machine that is working.
Established and related traffic is accepted, so a defended machine can
still dial out rather than being a disconnected one.

Verified by running, all of it:

  home -> devices (policy allow)               reachable
  devices -> home (policy deny)                blocked
  behind unforwardable NAT -> out              reachable
  in -> behind unforwardable NAT               unreachable
  inbound: deny, dialling out                  reachable
  reaching a machine that denies inbound       refused

The two routers differ exactly as declared: the forwardable one carries the
policy rule and no inbound drop, the unforwardable one carries `ct state
new drop` and no DNAT.
This commit is contained in:
jschoubben
2026-08-24 01:49:30 +02:00
parent a270cd5b02
commit 5d01006eab
9 changed files with 295 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/lifecycle/router.ts
+92
View File
@@ -91,6 +91,23 @@ export async function ensureRouterImage(log: (message: string) => void = () => {
log(` router image ready`);
}
/**
* The address the transit router holds on a public segment: the last usable host address.
*
* Chosen rather than declared, like a gateway's inside address — a scenario has nothing to
* say about the internet's own routers, only about the networks they connect.
*/
export function transitAddress(cidr: string): string | null {
const slash = cidr.lastIndexOf("/");
if (slash === -1) return null;
const base = cidr.slice(0, slash);
const prefix = cidr.slice(slash);
if (base.includes(":")) return `${base.replace(/::$/, "")}::fffe${prefix}`;
const octets = base.split(".");
octets[3] = "254";
return `${octets.join(".")}${prefix}`;
}
/** The name a router answers to in `list` and `exec` — scenery, but addressable. */
export function routerMachineName(plan: RouterPlan): string {
return `gw-${plan.inside.join("-")}`;
@@ -197,6 +214,70 @@ function insideAddress(scenario: Scenario, segment: string, family: Family): str
return null;
}
/**
* Wire the public segments together.
*
* The internet is not a network — it is unrelated networks that route to each other, many
* hops apart with no shared broadcast domain. So public segments are separate links joined
* by a router, never bridged: bridging them would make ARP adjacency, non-decrementing TTL
* and crossing multicast true in the lab and false in production, and the mesh has already
* been bitten by multicast name resolution.
*
* One transit router, an interface on every public segment, forwarding and no translation.
* It is the closest thing the lab has to "the internet", and it is deliberately dumb.
*/
export async function raiseTransit(
scenario: Scenario,
instanceId: string,
log: (message: string) => void = () => {},
): Promise<string | null> {
const publicSegments = Object.entries(scenario.segments)
.filter(([, segment]) => segment.kind === "public")
.map(([name]) => name);
// One public network needs no transit: everything on it is already adjacent.
if (publicSegments.length < 2) return null;
const name = `mlab-${instanceId}-transit`;
if (!(await succeeds(["config", "show", name], 15_000))) {
await incus([
"init", ROUTER_IMAGE, name,
"-c", `user.mesh-lab.instance=${instanceId}`,
"-c", "user.mesh-lab.machine=transit",
"-c", `user.mesh-lab.transit=${publicSegments.join(",")}`,
], 300_000);
await succeeds(["config", "device", "remove", name, "eth0"], 15_000);
for (const [index, segment] of publicSegments.entries()) {
await incus([
"config", "device", "add", name, `eth${index}`, "nic",
"nictype=bridged",
`parent=${networkName(instanceId, segment)}`,
`hwaddr=${macFor(instanceId, "transit", index)}`,
]);
}
}
await succeeds(["start", name], 60_000);
await waitUntilUsable(name, 120, () => {});
for (const [index, segment] of publicSegments.entries()) {
const device = `eth${index}`;
await sh(name, `ip link set ${device} up`);
for (const cidr of scenario.segments[segment]?.cidr ?? []) {
const address = transitAddress(cidr);
if (address) await sh(name, `ip addr replace ${address} dev ${device}`);
}
const mtu = scenario.segments[segment]?.mtu;
if (mtu) await sh(name, `ip link set ${device} mtu ${mtu}`);
}
await sh(
name,
"sysctl -w net.ipv4.ip_forward=1 >/dev/null; sysctl -w net.ipv6.conf.all.forwarding=1 >/dev/null",
);
log(` transit router across ${publicSegments.join(", ")}`);
return name;
}
export async function raiseRouters(
scenario: Scenario,
instanceId: string,
@@ -289,6 +370,17 @@ async function configureRouter(
"sysctl -w net.ipv4.ip_forward=1 >/dev/null; sysctl -w net.ipv6.conf.all.forwarding=1 >/dev/null",
);
// A gateway reaches other public networks the way anything does: through transit. Without
// this it can only reach its own outside segment, and every scenario with more than one
// public network becomes a set of islands.
for (const cidr of scenario.segments[plan.outside]?.cidr ?? []) {
const via = transitAddress(cidr);
if (!via) continue;
const gateway = via.slice(0, via.lastIndexOf("/"));
const family = gateway.includes(":") ? "-6" : "-4";
await sh(plan.name, `ip ${family} route replace default via ${gateway} dev eth0 2>/dev/null || true`);
}
const ttl = ttlSeconds(plan.mappingTtl);
if (ttl !== undefined) {
// What makes keepalive behaviour testable rather than hoped for: a connection held