hq/03-DESIGN/01-to-be/08-connectivity.md

---
layer: to-be
status: designed
code: []
updated: 2026-08-29
decisions:
  - 02-DECISIONS/0005-the-node-host.md
  - 02-DECISIONS/0004-a-node-and-how-it-joins.md
  - 02-DECISIONS/0007-connectivity.md
  - 02-DECISIONS/0007-connectivity.md
  - 02-DECISIONS/0004-a-node-and-how-it-joins.md
  - 02-DECISIONS/0007-connectivity.md
  - 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
---

# Connectivity

One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most
moving parts: **overlay, resolution, exposure, filtering, certificates.**

It is written as a whole because the five are one design. They share inputs, they must agree, and
every one of them today is computed in a different place by a different module from a different
copy of the same facts.

## Why it is control-plane work

Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one
node* — to each responsibility:

| | needs to know | whose |
|---|---|---|
| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane |
| **resolution** — which name is which node | **every node** | control plane |
| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | control plane |
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies |
| **certificates** — who may present which name | which name belongs to which node | control plane |

**Not one of the five can be answered by a machine on its own.** That is the whole reason this is
a context rather than a set of node-local modules — and it is exactly what the current
arrangement gets wrong, by computing all five on the node from a direct database connection.

## The shape: decided centrally, delivered as files

Every one of the five resolves the same way, and it is worth stating once rather than five times:

> **The connectivity context computes the configuration. It arrives over the link as `file`
> resources. The service reads files and knows nothing about the mesh.**

This costs **no new host vocabulary**. `file`, `directory`, `service` and `container` already
exist; WireGuard, the resolver and the proxy are all *a container or a package, plus files*.

It is also what removes the last two upward dependencies.
[Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules
opening a direct connection to the control plane's database — `wireguard` and `traefik` — and
they are the reason every node permanently holds a credential to it
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Both are connectivity
modules. **Closing this context closes that set.**

### And they are modules, not a second mechanism beside the module system

*Written 2026-08-29, from building it. The first version was code beside the module system doing
the module system's job, and the fault it produced is the point of writing this down.*

**A machine was on the private network because it had an address.** Every node that had been
placed got a peer list, whether or not anybody wanted it there, and there was no way to say a
machine should stay off. That is what "special-cased" cost, and it was invisible until somebody
wanted the exception.

**What made it look unavoidable:** a peer list cannot be written in a manifest. It is derived from
every other machine, so it differs on each one and changes when any of them changes. So the
manifest says its resources are **computed** — it names something in the control plane that works
them out per node — and it is a module in every other respect: assigned, resolved, configured by
settings, and absent from a machine nobody gave it to.

**What that made possible immediately** is the arrangement below, which the code has:

| module | provides | requires | claims |
|---|---|---|---|
| the WireGuard one | a private network, **and the mesh's own addressing** | | *the* private network, one per node |
| the names one | name resolution | the mesh's own addressing | |
| `networking` | | both of the above | |

**Three rather than one, because WireGuard is one VPN of several.** Naming the module after the
job — `networking` — and putting WireGuard inside it is the retired *flavor* idea wearing a
generic name: the second VPN has nowhere to go. So a module is named for what it *is* and declares
what it *does*, and `networking` is the third row — requirements and no files
([ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)).

**Names left the WireGuard module for their own.** They had been delivered inside it, on the
argument that a machine with peers and no names is half on the network. True, and the wrong place
to fix it — names are identical over a *different* private network, so bundling them made one
module out of two things. They require the mesh's **addressing** rather than a private network in
general, because that is what they are computed from: over a VPN that hands out its own addresses
the mesh has nothing to write, and refusing is what stops a machine being given a hosts file that
means nothing on it.

**And the claim is not decoration.** Choosing a different VPN still installed WireGuard — dragged
back in by the names, which needed addresses only WireGuard hands out — and nobody was told.
Running two VPNs is not always wrong; being *the* one the mesh runs over is singular. So it is a
claim, and the collision is refused by name.

**What is still not a module, and why that is correct.** The host needs none of this. It has an
address and a route before the mesh exists — that is the machine's own networking — and the
broker's address is carried in the token rather than resolved
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). **The one connection that
carries modules cannot itself be one.** Everything above it can be, and now is.

## The order it comes up in

The one thing to get right, because everything else depends on it:

```
0  the node has an underlay address     the machine's own — DHCP, or a provider gave it one
1  the node dials the mesh              OVER THE UNDERLAY, at the address in its token
2  it proves itself, and is proved to   the link exists      (ADR 0004, ADR 0004)
3  the mesh grants it an identity       and an overlay address
4  the overlay comes up                 peer graph delivered as files
5  names resolve                        resolver config delivered as files
6  filtering is applied                 derived from what is assigned here
7  routes and certificates              once this node has something to expose
```

**Step 1 runs on the underlay and never on the overlay.** This is the circularity that must not
be created: the overlay is configured by the mesh, so a link that required the overlay could
never be established on a new node. The link stays on the underlay permanently — it is
outbound-only and carries its own identity, so it needs nothing the overlay provides.

**Nothing before step 3 can resolve a mesh name**, which is why the token carries an *address*
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Today this is
patched with an `/etc/hosts` floor written underneath the resolver; under this design there is
nothing to patch.

**Step 1 has a precondition this document treated as a fact to record rather than a requirement:
the broker's node must be dialable by every node, at a stable address, and so must the hub**
([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Across the internet that means publicly
reachable; on one network it does not. A mesh whose nodes are all behind NAT cannot be raised, and
a broker node whose address moves invalidates every token issued for it.

**Whether the link should later move onto the overlay, with the underlay as fallback, is
[open](../../02-DECISIONS/0007-connectivity.md).** It is a decision rather than a derivation: the
gain is which network carries bytes, not what an attacker can reach, since the link is already
encrypted against a pinned fingerprint.

## 1 — The overlay

**What is decided:** the peer graph. For every node: its overlay address, which peers it holds,
which of those it may dial, and which must dial it.

**Inputs, all declared:**

- **reachability** — an endpoint, or none
  ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Not
  inferred from the address shape, which is wrong for carrier-grade NAT, wrong for IPv6, and
  wrong for a routable address behind a closed firewall.
- **site** — where the machine physically is, or nothing if it roams.
- **role** — hub or not, **declared**. Today it is inferred from an address prefix, which means a
  renumbering is an outage and nothing can be asked which node is the hub.

**Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the
public key is published to the mesh. This is already true and it is already right — it is
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *a node holds its own
identity* applied to the overlay, and it means the control plane computes a graph it cannot
itself impersonate.

**Shape: a hub, with direct peering between co-located nodes.**

| | |
|---|---|
| two nodes at the same site | peer **directly**, host-routed, with a keepalive |
| everything else | routes through the **hub** |
| a node with no site — it roams | **hub only** |

**Roaming is hub-only deliberately, and the reason is a property of WireGuard rather than a
preference: there is no failover.** A more specific route to a dead endpoint blackholes; it does
not fall back to the general one. So a node whose location changes gets exactly one path, because
two paths would mean one of them silently swallowing traffic.

**What the host receives:** an interface configuration and a peer list, as files. It does not
compute them, and after this it holds no credential to the mesh's database.

### Four things the lab found, none of them visible from the mesh's own state

*Written 2026-08-29, on the first three machines to actually run this.*

Each looked like a working network from every angle the mesh can see: the graph was right, the
files were right, the services were up, and every node reported success.

- **A running interface does not re-read its configuration.** A node joins, every existing node's
  peer list changes, each file is replaced — and the service is already running, so nothing
  reloads it. Every existing node keeps a network that no longer exists. The declaration has to
  say the service must *reflect* the file, which is declared state; a command to restart would be
  an action, and the link may not carry one
  ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)).
- **A hub that shares a site with a spoke was emitted twice** — once as a direct peer and once as
  the route of last resort. WireGuard takes one entry per public key, so the interface refuses the
  file. The ordinary shape of a small mesh, and in none of the tests written before it ran.
- **Two nodes at one site that neither can be dialled must not peer directly.** Nobody opens the
  path, and the direct route is more specific than the hub's, so it wins and blackholes. This
  document's own warning, arriving in its implementation: *a more specific route to a dead
  endpoint blackholes; it does not fall back to the general one.*
- **The container runtime closes the door the overlay needs.** Docker sets the FORWARD policy to
  DROP, so a hub with `ip_forward` enabled still carries nothing between its spokes. The substrate
  at tier 1 silently breaks the network at tier 2, and nothing in either tier's state says so. The
  hub inserts its own rule above those chains and removes it on the way down.

**The pattern in all four:** the mesh's picture of the network was correct and the network did not
work. That is the argument for the lab in one line — none of these is reachable by reasoning, and
each was found within minutes of a real machine trying it.

## 2 — Resolution

**Two name spaces, and they do not mix:**

| | resolves to | certified by |
|---|---|---|
| **internal names** | overlay addresses | the **mesh CA** |
| **public names** | whatever the outside world must reach | a **public authority** |

A node's mesh name is its overlay address. Its public name, if it has one, is a separate fact
used by things outside the mesh — and the separation carries two lessons that were learned
expensively enough to be worth restating:

- **Mesh names are not multicast names.** A name resolved by local multicast discovery introduces
  a delay and a failure mode that appears on one node and not others — the worst shape a fault
  can have.
- **A node must not pin its own public name locally.** The duplicate record breaks resolution of
  that name for everything else that needs it.

**What the host receives:** the resolver's configuration, as files, listing every peer's internal
name and overlay address.

**What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh
database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)
nothing needs a name before the link, and a fallback nothing needs is a path nothing tests.

## 3 — Exposure

Settled by [ADR 0007](../../02-DECISIONS/0007-connectivity.md); summarised here because
this is where it belongs.

**A route is a grant.** A module that must be reachable declares it needs one; the proxy provides
it and hands back the public name. Ordinary
[ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)
vocabulary — the mirror of a database grant, where the consumer supplies a target and receives a
name rather than supplying nothing and receiving credentials.

**A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is
the case is a mesh-level fact, which is the fourth reason exposure is control-plane work.

## 4 — Filtering

**Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a
consequence of what runs on it and who must reach it, not an independent declaration to keep in
step by hand.

**A rule names its source** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)).
A rule with no source is open, and must say so rather than appear to restrict something. `scope:`
is removed rather than implemented: five manifests carry it today, it is referenced by no code,
and it is the clearest instance in the repository of *an unenforced rule is indistinguishable
from a wrong one, and costs more, because people believe it.*

**Unknown keys are refused** — the discipline the host's declaration parser already has
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and
the one manifests lack. `scope:` survived because nothing rejected it.

## 5 — Certificates

**Two authorities, kept separate on purpose.**

| | issued by | for |
|---|---|---|
| **public names** | a public ACME authority | anything outside the mesh reaches |
| **internal names** | the **mesh CA** | node-to-node, over the overlay |

**The split is not collapsed, including in the lab.** A single-CA lab would hide any bug living
in the split, so the lab runs its own ACME issuer on its public segment and keeps the mesh CA
unchanged ([research 004](../../01-RESEARCH/004-lab-network/00-overview.md)).

**Public issuance requires genuine public reachability.** The HTTP-01 challenge must be answered
at the name being certified, so issuance happens through a publicly reachable node regardless of
where the workload runs — the same asymmetry as exposure, for the same reason.

**The issuer must be configurable.** Today it is not: the proxy sets no `caServer` and therefore
defaults to the public authority's *production* endpoint. Two consequences, and the second is
worse than the lab problem that found it — every certificate experiment on a real node consumes
production issuance quota, and a retry loop can exhaust it for a week.

**The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the
fingerprint in its token ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)),
so nothing needs the CA before membership. It certifies internal names afterwards, and that is
all it does.

## What this removes

The list is worth having in one place, because it is most of the argument:

- **The last two direct database connections from nodes** — `wireguard` and `traefik`, the only
  two, both connectivity.
- **Therefore the database credential on every node**, and the object-store credential beside it.
  [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s central claim becomes
  true rather than aspirational.
- **The `/etc/hosts` floor**, and the bootstrap circularity it patched.
- **Hub election by address prefix**, and the silent no-hub failure when nobody knew the
  convention.
- **The RFC1918 inference**, and the lab substitution that existed to satisfy it.
- **`scope:`**, and the class of manifest key that means nothing.

## Open

- ~~**What happens when the hub is down.**~~ **Resolved** by
  [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), together with `06`'s
  matching question — they were one question. Nothing takes over. WireGuard has no failover, the
  hub is declared rather than elected, and non-co-located paths stop while co-located direct peers
  and every already-assigned workload keep running. The recovery path is restore, and its deadline
  is certificate renewal.
- **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from
  an address, but no procedure exists, and a graph delivered node by node has an ordering problem
  while it is half-applied.
- **Revoking a route** when a module is unassigned ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)).
  A stale public name pointing at nothing fails more visibly than a stale grant.
- **IPv6.** [ADR 0007](../../02-DECISIONS/0007-connectivity.md) makes
  it expressible; nothing here says the overlay or the resolver handle it.
- **Reporting declared-versus-observed.** ADR 0007 makes the disagreement detectable and does not
  say who looks or what they are told.