A working private network, and four reasons it did not work

Three machines across two sites, two of them behind no reachable address, all
nine paths open. The mesh computes the graph, delivers it as a declaration, and
the nodes bring it up.

Every fault below looked like success from inside the mesh: the graph was
right, the files were right, the services were up, every node reported it had
applied. None was reachable by reasoning.

A running interface does not re-read its configuration. A node joins, every
existing node's peer list changes, the file is replaced -- and the service is
already running, so nothing reloads it. Fixed as declared state rather than a
command: the service must reflect the file. A command to restart would be an
action, and the link may not carry one. The host refused exactly that, which is
how this shape was arrived at.

A hub sharing a site with a spoke appeared twice in that spoke's peer list --
once as a direct peer, once as the route of last resort. WireGuard takes one
entry per key and refuses the file. The ordinary shape of a small mesh, and in
none of the tests written before it ran.

Two nodes at one site that neither can be dialled were peered directly. Nobody
opens the path, and the direct route is more specific than the hub's, so it
wins and blackholes -- this design's own warning arriving in its
implementation. They now route through the hub unless one end can be dialled.

And Docker sets the FORWARD policy to DROP, so a hub with ip_forward enabled
carried nothing between its spokes. The substrate at tier 1 silently breaks the
network at tier 2, and nothing in either tier's state says so. The hub inserts
its own rule above those chains and removes it on the way down.

Two weak tests found by injection along the way: one asserted the keepalive
rule only against the hub, whose peer entries happen not to set that field at
all, so it tested an absence; the other checked the firewall rules by looking
for FORWARD anywhere, which the PostDown line satisfies on its own.

internal/overlay/graph_test.go

@@ -222,3 +222,85 @@ func TestNobodyPeersWithThemselves(t *testing.T) {
 		}
 	}
 }
+
+func TestAHubAtYourOwnSiteAppearsOnceNotTwice(t *testing.T) {
+	// WireGuard takes one entry per public key. A hub that shares a site with a spoke was being
+	// emitted twice — once as a direct peer and once as the route of last resort — producing a
+	// configuration the interface refuses, from a mesh that thought it had succeeded.
+	//
+	// Found in the lab on the first two machines that shared a site with their hub, which is the
+	// ordinary case for a small mesh and was in none of the tests above.
+	g, err := Compute([]Node{
+		at("anchor", "lab", "10.42.0.1", "192.0.2.10:51820", true),
+		at("laptop", "lab", "10.42.0.2", "", false),
+	}, cidr)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	seen := map[string]int{}
+	for _, p := range g["laptop"] {
+		seen[p.Key]++
+	}
+	for key, count := range seen {
+		if count > 1 {
+			t.Errorf("the key %s appears %d times; WireGuard takes one entry per key", key, count)
+		}
+	}
+
+	if len(g["laptop"]) != 1 {
+		t.Fatalf("laptop has %d peers; the hub at its own site is one peer, not two", len(g["laptop"]))
+	}
+	// And that single entry has to carry everything, or the node keeps a direct path to the hub
+	// and loses its route to everywhere else.
+	if got := g["laptop"][0].Allowed; got != cidr {
+		t.Errorf("the single entry allows %s; it is both the direct path and the route of last "+
+			"resort, so it carries the whole overlay", got)
+	}
+}
+
+func TestTwoUnreachableNodesAtOneSiteDoNotPeerDirectly(t *testing.T) {
+	// Nobody would open the path, and the direct route is more specific than the hub's — so it
+	// wins and blackholes. Both nodes would reach the hub perfectly and be unable to reach each
+	// other, which is the worst arrangement available: it looks configured and is not.
+	//
+	// This is the design's own warning arriving in its implementation, and the lab found it with
+	// two machines at one site behind no reachable address — the ordinary shape of a house.
+	g, err := Compute([]Node{
+		at("anchor", "datacentre", "10.42.0.1", "198.51.100.1:51820", true),
+		at("desk", "house", "10.42.0.2", "", false),
+		at("server", "house", "10.42.0.3", "", false),
+	}, cidr)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if _, ok := peersOf(t, g, "desk")["server"]; ok {
+		t.Error("two nodes that neither can be dialled were peered directly; neither could open " +
+			"the path and the route is more specific than the hub's, so it blackholes")
+	}
+	// And they must still be able to reach each other — through the hub.
+	if hub := peersOf(t, g, "desk")["anchor"]; hub.Allowed != cidr {
+		t.Errorf("desk's route to everything else allows %s", hub.Allowed)
+	}
+}
+
+func TestOneReachableNodeIsEnoughToPeerDirectly(t *testing.T) {
+	// The other side of it: if either end can be dialled, the path can be opened, and using it
+	// keeps their traffic off a hub that may be on another continent.
+	g, err := Compute([]Node{
+		at("anchor", "datacentre", "10.42.0.1", "198.51.100.1:51820", true),
+		at("desk", "house", "10.42.0.2", "192.0.2.2:51820", false),
+		at("server", "house", "10.42.0.3", "", false),
+	}, cidr)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, ok := peersOf(t, g, "server")["desk"]; !ok {
+		t.Error("a node did not peer with a reachable neighbour at its own site")
+	}
+	if _, ok := peersOf(t, g, "desk")["server"]; !ok {
+		t.Error("the reachable node did not hold its unreachable neighbour, so replies have " +
+			"nowhere to go")
+	}
+}