A working private network, and four reasons it did not work

Three machines across two sites, two of them behind no reachable address, all
nine paths open. The mesh computes the graph, delivers it as a declaration, and
the nodes bring it up.

Every fault below looked like success from inside the mesh: the graph was
right, the files were right, the services were up, every node reported it had
applied. None was reachable by reasoning.

A running interface does not re-read its configuration. A node joins, every
existing node's peer list changes, the file is replaced -- and the service is
already running, so nothing reloads it. Fixed as declared state rather than a
command: the service must reflect the file. A command to restart would be an
action, and the link may not carry one. The host refused exactly that, which is
how this shape was arrived at.

A hub sharing a site with a spoke appeared twice in that spoke's peer list --
once as a direct peer, once as the route of last resort. WireGuard takes one
entry per key and refuses the file. The ordinary shape of a small mesh, and in
none of the tests written before it ran.

Two nodes at one site that neither can be dialled were peered directly. Nobody
opens the path, and the direct route is more specific than the hub's, so it
wins and blackholes -- this design's own warning arriving in its
implementation. They now route through the hub unless one end can be dialled.

And Docker sets the FORWARD policy to DROP, so a hub with ip_forward enabled
carried nothing between its spokes. The substrate at tier 1 silently breaks the
network at tier 2, and nothing in either tier's state says so. The hub inserts
its own rule above those chains and removes it on the way down.

Two weak tests found by injection along the way: one asserted the keepalive
rule only against the hub, whose peer entries happen not to set that field at
all, so it tested an absence; the other checked the firewall rules by looking
for FORWARD anywhere, which the PostDown line satisfies on its own.

internal/overlay/declaration_test.go

@@ -2,6 +2,7 @@ package overlay
 
 import (
 	"encoding/json"
+	"fmt"
 	"strings"
 	"testing"
 )
@@ -137,3 +138,82 @@ func TestOnlyAReachableNodeListens(t *testing.T) {
 		t.Error("a reachable node does not listen on the port its endpoint names")
 	}
 }
+
+func TestTheServiceIsRestartedWhenThePeerListChanges(t *testing.T) {
+	// The fault this exists to catch, found in the lab the moment a third node arrived: a running
+	// WireGuard interface does not re-read its configuration. The file was replaced, the service
+	// was already running so nothing reloaded it, and every existing node kept a network that no
+	// longer matched the mesh — while looking entirely successful.
+	//
+	// So the declaration has to verify what the interface is *carrying*, not what the file says.
+	//
+	// And it must be declared state rather than a command: the host refuses an action arriving
+	// over the link (novox/hq ADR 0005), correctly, which is how this shape was arrived at. There
+	// is a test below that no action is ever in here.
+	_, resources := declarationFor(t, Node{Name: "laptop", Key: "PUB", Address: "10.42.0.2"},
+		[]Peer{{Name: "anchor", Key: "HUBKEY", Allowed: "10.42.0.0/16"}})
+
+	for _, r := range resources {
+		if r["type"] != "service" {
+			continue
+		}
+		reflects := fmt.Sprint(r["restart-on"])
+		if !strings.Contains(reflects, "overlay-config") {
+			t.Errorf("the interface does not restart when its configuration changes: %v", reflects)
+		}
+		return
+	}
+	t.Error("nothing in the declaration brings the interface up")
+}
+
+func TestTheOverlayDeclarationCarriesNoAction(t *testing.T) {
+	// The link may not carry an action, and the host refuses a declaration containing one — whole,
+	// not in part. An overlay declaration with an action in it does not half-apply: it leaves the
+	// node with no network at all.
+	_, resources := declarationFor(t, Node{Name: "laptop", Key: "PUB", Address: "10.42.0.2"}, nil)
+	for _, r := range resources {
+		if r["type"] == "action" {
+			t.Errorf("the declaration contains an action (%v); the host will refuse the whole "+
+				"thing and the node will have no network", r["id"])
+		}
+	}
+}
+
+func TestTheHubForwardsAndNobodyElseDoes(t *testing.T) {
+	// A hub carries traffic between its spokes, and Linux does not forward packets unless told
+	// to. Without it every spoke reaches the hub perfectly and no spoke reaches any other — which
+	// is how it failed in the lab, and it presents as a peering problem rather than a kernel
+	// setting, so it is worth being sure of.
+	hub, _ := declarationFor(t, Node{Name: "anchor", Key: "HUB", Address: "10.42.0.1",
+		Endpoint: "198.51.100.1:51820", Hub: true}, nil)
+	if !strings.Contains(hub, "ip_forward=1") {
+		t.Error("the hub does not enable forwarding, so its spokes cannot reach each other")
+	}
+	if !strings.Contains(hub, "ip_forward=0") {
+		t.Error("the hub never stops forwarding; a machine that stops being the hub would keep " +
+			"passing traffic it is no longer part of")
+	}
+
+	// And past the machine's own firewall. Any node with a container runtime has FORWARD set to
+	// DROP by Docker, so enabling ip_forward alone changes nothing — which is exactly how it
+	// failed, with every spoke reaching the hub and no spoke reaching any other.
+	// Checked as PostUp specifically. "contains FORWARD" passes on the PostDown line alone,
+	// which would leave a hub that tears down rules it never put up.
+	if !strings.Contains(hub, "PostUp = command -v iptables") {
+		t.Error("the hub does not open its own firewall, so a container runtime's DROP policy " +
+			"silently eats everything it was supposed to carry")
+	}
+	if !strings.Contains(hub, "PostDown = command -v iptables") {
+		t.Error("the hub never removes those rules, so a machine that stops being the hub keeps " +
+			"passing traffic it is no longer part of")
+	}
+
+	spoke, _ := declarationFor(t, Node{Name: "laptop", Key: "PUB", Address: "10.42.0.2"}, nil)
+	if strings.Contains(spoke, "FORWARD") {
+		t.Error("a spoke was given forwarding rules it has no use for")
+	}
+	if strings.Contains(spoke, "ip_forward") {
+		t.Error("a spoke was told to forward packets, which is not its job and widens what a " +
+			"compromised one could do")
+	}
+}