A working private network, and four reasons it did not work

Three machines across two sites, two of them behind no reachable address, all
nine paths open. The mesh computes the graph, delivers it as a declaration, and
the nodes bring it up.

Every fault below looked like success from inside the mesh: the graph was
right, the files were right, the services were up, every node reported it had
applied. None was reachable by reasoning.

A running interface does not re-read its configuration. A node joins, every
existing node's peer list changes, the file is replaced -- and the service is
already running, so nothing reloads it. Fixed as declared state rather than a
command: the service must reflect the file. A command to restart would be an
action, and the link may not carry one. The host refused exactly that, which is
how this shape was arrived at.

A hub sharing a site with a spoke appeared twice in that spoke's peer list --
once as a direct peer, once as the route of last resort. WireGuard takes one
entry per key and refuses the file. The ordinary shape of a small mesh, and in
none of the tests written before it ran.

Two nodes at one site that neither can be dialled were peered directly. Nobody
opens the path, and the direct route is more specific than the hub's, so it
wins and blackholes -- this design's own warning arriving in its
implementation. They now route through the hub unless one end can be dialled.

And Docker sets the FORWARD policy to DROP, so a hub with ip_forward enabled
carried nothing between its spokes. The substrate at tier 1 silently breaks the
network at tier 2, and nothing in either tier's state says so. The hub inserts
its own rule above those chains and removes it on the way down.

Two weak tests found by injection along the way: one asserted the keepalive
rule only against the hub, whose peer entries happen not to set that field at
all, so it tested an absence; the other checked the firewall rules by looking
for FORWARD anywhere, which the PostDown line satisfies on its own.

internal/link/enrolment.go

@@ -32,8 +32,8 @@ type Enrolment struct {
 // single statement that both finds and marks it, so two machines racing on one secret produce one
 // winner. Only then is a key recorded — because recording a key for a node whose token turned out
 // to be spent would leave the mesh believing a machine that never had the right to join.
-func (e Enrolment) Enrol(ctx context.Context, secret string, public ed25519.PublicKey,
-	profile map[string]any) (EnrolReply, error) {
+func (e Enrolment) Enrol(ctx context.Context, request EnrolRequest) (EnrolReply, error) {
+	secret, public, profile := request.Secret, ed25519.PublicKey(request.PublicKey), request.Profile
 
 	if len(public) != ed25519.PublicKeySize {
 		return EnrolReply{}, fmt.Errorf("a node presented a %d-byte key, and an identity is %d",
@@ -84,6 +84,17 @@ func (e Enrolment) Enrol(ctx context.Context, secret string, public ed25519.Publ
 		reply.Password = password
 	}
 
+	// Recorded before the profile because the overlay is the first declaration this node will
+	// receive, and without this key the mesh cannot compose one. A node enrolled with no overlay
+	// key is a node the graph skips — an ordinary in-between state, and one worth leaving as
+	// briefly as possible.
+	if request.OverlayKey != "" {
+		if err := e.Inventory.RecordOverlayKey(ctx, node.ID, request.OverlayKey); err != nil {
+			return EnrolReply{}, fmt.Errorf(
+				"the token was spent and %s's overlay key could not be recorded: %w", node.Name, err)
+		}
+	}
+
 	if profile != nil {
 		// Not fatal if it fails. The profile is what the control plane needs in order to decide
 		// what this machine should run, and it is reported again on every connection — so losing

internal/link/protocol.go

@@ -40,6 +40,10 @@ type EnrolRequest struct {
 	// half has never left that machine (novox/hq ADR 0004).
 	PublicKey []byte `json:"public_key"`
 
+	// OverlayKey is the public half of this node's key on the private network — a different key
+	// from PublicKey, and the mesh only ever sees this half.
+	OverlayKey string `json:"overlay_key,omitempty"`
+
 	// Profile is what this machine can be asked to do. The control plane cannot decide what a
 	// node should run without it, so it arrives with enrolment rather than being asked for after.
 	Profile map[string]any `json:"profile,omitempty"`

internal/link/serve.go

@@ -2,7 +2,6 @@ package link
 
 import (
 	"context"
-	"crypto/ed25519"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -24,7 +23,7 @@ const AMQPVar = "MESH_BROKER_AMQP"
 type Enroller interface {
 	// Enrol spends the token, records the key, and reports the node's name. The error is
 	// returned to the node as a refusal; it must be the same for every reason a token can fail.
-	Enrol(ctx context.Context, secret string, public ed25519.PublicKey, profile map[string]any) (EnrolReply, error)
+	Enrol(ctx context.Context, request EnrolRequest) (EnrolReply, error)
 }
 
 // Server consumes what nodes say.
@@ -194,7 +193,7 @@ func (s *Server) handleEnrol(ctx context.Context, delivery amqp.Delivery) {
 	if err := json.Unmarshal(delivery.Body, &request); err != nil {
 		s.log.Printf("an enrolment request could not be read: %v", err)
 	} else {
-		accepted, err := s.enroller.Enrol(ctx, request.Secret, request.PublicKey, request.Profile)
+		accepted, err := s.enroller.Enrol(ctx, request)
 		if err != nil {
 			// Logged in full here, where an operator can see it; sent back as one refusal, so
 			// that somebody guessing learns nothing from which reason came back.