hq/03-DESIGN/01-to-be/08-connectivity.md

---
layer: to-be
status: designed
code:
  - mesh-control internal/catalogue/filtering.go
  - mesh-control internal/identity/authority.go
  - mesh-host internal/identity/serving.go
  - mesh-host internal/apply (the service that reflects a rule set)
updated: 2026-08-31
decisions:
  - 02-DECISIONS/0005-the-node-host.md
  - 02-DECISIONS/0004-a-node-and-how-it-joins.md
  - 02-DECISIONS/0007-connectivity.md
  - 02-DECISIONS/0007-connectivity.md
  - 02-DECISIONS/0004-a-node-and-how-it-joins.md
  - 02-DECISIONS/0007-connectivity.md
  - 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
---

# Connectivity

One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most
moving parts: **overlay, resolution, exposure, filtering, certificates.**

It is written as a whole because the five are one design. They share inputs, they must agree, and
every one of them today is computed in a different place by a different module from a different
copy of the same facts.

## Why it is control-plane work

Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one
node* — to each responsibility:

| | needs to know | whose |
|---|---|---|
| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane |
| **resolution** — which name is which node | **every node** | control plane |
| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | control plane |
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies |
| **certificates** — who may present which name | which name belongs to which node | control plane |

**Not one of the five can be answered by a machine on its own.** That is the whole reason this is
a context rather than a set of node-local modules — and it is exactly what the current
arrangement gets wrong, by computing all five on the node from a direct database connection.

## The shape: decided centrally, delivered as files

Every one of the five resolves the same way, and it is worth stating once rather than five times:

> **The connectivity context computes the configuration. It arrives over the link as `file`
> resources. The service reads files and knows nothing about the mesh.**

This costs **no new host vocabulary**. `file`, `directory`, `service` and `container` already
exist; WireGuard, the resolver and the proxy are all *a container or a package, plus files*.

It is also what removes the last two upward dependencies.
[Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules
opening a direct connection to the control plane's database — `wireguard` and `traefik` — and
they are the reason every node permanently holds a credential to it
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Both are connectivity
modules. **Closing this context closes that set.**

### And they are modules, not a second mechanism beside the module system

*Written 2026-08-29, from building it. The first version was code beside the module system doing
the module system's job, and the fault it produced is the point of writing this down.*

**A machine was on the private network because it had an address.** Every node that had been
placed got a peer list, whether or not anybody wanted it there, and there was no way to say a
machine should stay off. That is what "special-cased" cost, and it was invisible until somebody
wanted the exception.

**What made it look unavoidable:** a peer list cannot be written in a manifest. It is derived from
every other machine, so it differs on each one and changes when any of them changes. So the
manifest says its resources are **computed** — it names something in the control plane that works
them out per node — and it is a module in every other respect: assigned, resolved, configured by
settings, and absent from a machine nobody gave it to.

**What that made possible immediately** is the arrangement below, which the code has:

| module | provides | requires | claims |
|---|---|---|---|
| the WireGuard one | a private network, **and the mesh's own addressing** | | *the* private network, one per node |
| the names one | name resolution | the mesh's own addressing | |
| `networking` | | both of the above | |

**Three rather than one, because WireGuard is one VPN of several.** Naming the module after the
job — `networking` — and putting WireGuard inside it is the retired *flavor* idea wearing a
generic name: the second VPN has nowhere to go. So a module is named for what it *is* and declares
what it *does*, and `networking` is the third row — requirements and no files
([ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)).

**Names left the WireGuard module for their own.** They had been delivered inside it, on the
argument that a machine with peers and no names is half on the network. True, and the wrong place
to fix it — names are identical over a *different* private network, so bundling them made one
module out of two things. They require the mesh's **addressing** rather than a private network in
general, because that is what they are computed from: over a VPN that hands out its own addresses
the mesh has nothing to write, and refusing is what stops a machine being given a hosts file that
means nothing on it.

**And the claim is not decoration.** Choosing a different VPN still installed WireGuard — dragged
back in by the names, which needed addresses only WireGuard hands out — and nobody was told.
Running two VPNs is not always wrong; being *the* one the mesh runs over is singular. So it is a
claim, and the collision is refused by name.

**And the proxy's half, which was the other module reaching into the database.** A web application
requiring a reverse proxy has to say *which name, which port*, and there was nowhere to put it —
`requires` says a thing must exist and never said what to do with it. A module now contributes to
a requirement, the control plane collects every contribution on a node, and the provider is given
them as a file at a path it named. It reloads when that file changes, by the same `restart-on` the
private network needed when a peer list changed under a running interface.

**The proxy's configuration is not written by the mesh.** It is given the facts and turns them
into whatever it runs, which is why swapping Traefik for something else touches nothing that
publishes through it. See [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md) for the
other direction — handing a credential *back* — which is the larger half and is not built.

**What is still not a module, and why that is correct.** The host needs none of this. It has an
address and a route before the mesh exists — that is the machine's own networking — and the
broker's address is carried in the token rather than resolved
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). **The one connection that
carries modules cannot itself be one.** Everything above it can be, and now is.

## The order it comes up in

The one thing to get right, because everything else depends on it:

```
0  the node has an underlay address     the machine's own — DHCP, or a provider gave it one
1  the node dials the mesh              OVER THE UNDERLAY, at the address in its token
2  it proves itself, and is proved to   the link exists      (ADR 0004, ADR 0004)
3  the mesh grants it an identity       and an overlay address
4  the overlay comes up                 peer graph delivered as files
5  names resolve                        resolver config delivered as files
6  filtering is applied                 derived from what is assigned here
7  routes and certificates              once this node has something to expose
```

**Step 1 runs on the underlay and never on the overlay.** This is the circularity that must not
be created: the overlay is configured by the mesh, so a link that required the overlay could
never be established on a new node. The link stays on the underlay permanently — it is
outbound-only and carries its own identity, so it needs nothing the overlay provides.

**Nothing before step 3 can resolve a mesh name**, which is why the token carries an *address*
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Today this is
patched with an `/etc/hosts` floor written underneath the resolver; under this design there is
nothing to patch.

**Step 1 has a precondition this document treated as a fact to record rather than a requirement:
the broker's node must be dialable by every node, at a stable address, and so must the hub**
([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Across the internet that means publicly
reachable; on one network it does not. A mesh whose nodes are all behind NAT cannot be raised, and
a broker node whose address moves invalidates every token issued for it.

**Whether the link should later move onto the overlay, with the underlay as fallback, is
[open](../../02-DECISIONS/0007-connectivity.md).** It is a decision rather than a derivation: the
gain is which network carries bytes, not what an attacker can reach, since the link is already
encrypted against a pinned fingerprint.

## 1 — The overlay

**What is decided:** the peer graph. For every node: its overlay address, which peers it holds,
which of those it may dial, and which must dial it.

**Inputs, all declared:**

- **reachability** — an endpoint, or none
  ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Not
  inferred from the address shape, which is wrong for carrier-grade NAT, wrong for IPv6, and
  wrong for a routable address behind a closed firewall.
- **site** — where the machine physically is, or nothing if it roams.
- **role** — hub or not, **declared**. Today it is inferred from an address prefix, which means a
  renumbering is an outage and nothing can be asked which node is the hub.

**Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the
public key is published to the mesh. This is already true and it is already right — it is
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *a node holds its own
identity* applied to the overlay, and it means the control plane computes a graph it cannot
itself impersonate.

**Shape: a hub, with direct peering between co-located nodes.**

| | |
|---|---|
| two nodes at the same site | peer **directly**, host-routed, with a keepalive |
| everything else | routes through the **hub** |
| a node with no site — it roams | **hub only** |

**Roaming is hub-only deliberately, and the reason is a property of WireGuard rather than a
preference: there is no failover.** A more specific route to a dead endpoint blackholes; it does
not fall back to the general one. So a node whose location changes gets exactly one path, because
two paths would mean one of them silently swallowing traffic.

**What the host receives:** an interface configuration and a peer list, as files. It does not
compute them, and after this it holds no credential to the mesh's database.

### Four things the lab found, none of them visible from the mesh's own state

*Written 2026-08-29, on the first three machines to actually run this.*

Each looked like a working network from every angle the mesh can see: the graph was right, the
files were right, the services were up, and every node reported success.

- **A running interface does not re-read its configuration.** A node joins, every existing node's
  peer list changes, each file is replaced — and the service is already running, so nothing
  reloads it. Every existing node keeps a network that no longer exists. The declaration has to
  say the service must *reflect* the file, which is declared state; a command to restart would be
  an action, and the link may not carry one
  ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)).
- **A hub that shares a site with a spoke was emitted twice** — once as a direct peer and once as
  the route of last resort. WireGuard takes one entry per public key, so the interface refuses the
  file. The ordinary shape of a small mesh, and in none of the tests written before it ran.
- **Two nodes at one site that neither can be dialled must not peer directly.** Nobody opens the
  path, and the direct route is more specific than the hub's, so it wins and blackholes. This
  document's own warning, arriving in its implementation: *a more specific route to a dead
  endpoint blackholes; it does not fall back to the general one.*
- **The container runtime closes the door the overlay needs.** Docker sets the FORWARD policy to
  DROP, so a hub with `ip_forward` enabled still carries nothing between its spokes. The substrate
  at tier 1 silently breaks the network at tier 2, and nothing in either tier's state says so. The
  hub inserts its own rule above those chains and removes it on the way down.

**The pattern in all four:** the mesh's picture of the network was correct and the network did not
work. That is the argument for the lab in one line — none of these is reachable by reasoning, and
each was found within minutes of a real machine trying it.

## 2 — Resolution

**Two name spaces, and they do not mix:**

| | resolves to | certified by |
|---|---|---|
| **internal names** | overlay addresses | the **mesh CA** |
| **public names** | whatever the outside world must reach | a **public authority** |

A node's mesh name is its overlay address. Its public name, if it has one, is a separate fact
used by things outside the mesh — and the separation carries two lessons that were learned
expensively enough to be worth restating:

- **Mesh names are not multicast names.** A name resolved by local multicast discovery introduces
  a delay and a failure mode that appears on one node and not others — the worst shape a fault
  can have.
- **A node must not pin its own public name locally.** The duplicate record breaks resolution of
  that name for everything else that needs it.

**What the host receives:** the resolver's configuration, as files, listing every peer's internal
name and overlay address.

**What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh
database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)
nothing needs a name before the link, and a fallback nothing needs is a path nothing tests.

## 3 — Exposure

Settled by [ADR 0007](../../02-DECISIONS/0007-connectivity.md); summarised here because
this is where it belongs.

**A route is a grant.** A module that must be reachable declares it needs one; the proxy provides
it and hands back the public name. Ordinary
[ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)
vocabulary — the mirror of a database grant, where the consumer supplies a target and receives a
name rather than supplying nothing and receiving credentials.

**A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is
the case is a mesh-level fact, which is the fourth reason exposure is control-plane work.

## 4 — Filtering

**Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a
consequence of what runs on it and who must reach it, not an independent declaration to keep in
step by hand.

**A rule names its source** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)).
A rule with no source is open, and must say so rather than appear to restrict something. `scope:`
is removed rather than implemented: five manifests carry it today, it is referenced by no code,
and it is the clearest instance in the repository of *an unenforced rule is indistinguishable
from a wrong one, and costs more, because people believe it.*

**Unknown keys are refused** — the discipline the host's declaration parser already has
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and
the one manifests lack. `scope:` survived because nothing rejected it.

### What was built

*2026-08-31. Everything above was the intention; this is what exists, and how each part is
checked. [04-ISSUES/003](../../04-ISSUES/003-firewall-scope-is-read-by-no-code/00-report.md) is
resolved by it.*

**A module says what it listens on**, as a port, a protocol and a source — `mesh`, `anywhere`, or
`machine`. The source is required and there is no default, which is the whole of *a rule names its
source*: a manifest that omitted it would read as a restriction and be none. *Checked by a manifest
with a port and no source being refused, and by one naming a source the mesh cannot render being
refused as well — the second is what stops a source becoming a comment.*

**The set is derived per node**, from every module assigned to it, not from the module asking for
it. Where two modules want the same port, the wider source wins and both are still named, because
removing one of them must not read as a reason to close a port the other needs. *Checked by
rendering a node whose firewall module has no ports of its own and asserting another module's port
is in the result; and by giving one port two modules and one source each, and asserting the
narrower rule disappears while both names survive.*

**What is not declared is closed.** The rule set drops by default. *Checked by naming the input
chain in the assertion rather than the policy alone — the first version of that test passed while
input accepted everything, because another chain in the same file also said `policy drop`.*

**From the mesh means the machines the mesh has**, as their addresses on the private network, not
as a subnet. A subnet is a guess that stays wrong quietly; the address set shrinks when a node
leaves and nobody edits anything. A machine that asks for `mesh` where the mesh knows no addresses
is **closed and told so in the file** — widening it would open a port nobody asked to open, and
dropping it silently would close one somebody did.

**Three things it deliberately does not do**, each of which looked right and would have broken
something:

| | why not |
|---|---|
| decide what the machine **forwards** | the container runtime writes its own forwarding rules and a second policy is consulted alongside them, so a drop here stops every container on the node — the control plane included. Nothing in a manifest says what a machine routes, so there is nothing to derive it from either |
| **flush the ruleset** when loading | that empties every table on the machine, the runtime's among them. Only the mesh's own table is replaced, and it is declared empty first so the replacement works on a machine loading one for the first time |
| carry a **command to load itself** | the link may not carry an action ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). A service is declared to reflect the file instead, so replacing it restarts what loads it — the shape that rule leaves, used here for the first time for its real purpose |

**A module that wants a rule set brings the unit that loads it.** Found the hard way: the unit a
distribution packages for nftables runs, applies the rules and exits, so it is neither running nor
stopped — and a host asked for a service that is "running" reports, quite correctly, that it is
stopped. Every packet was filtered exactly as declared and the machine was marked as not doing what
it was told.

**The vocabulary has no word for "ran, did its job, and exited"**, and that is a real gap rather
than a wording problem: the whole class of configuration-applying units — packet filters, sysctl,
tmpfiles — is shaped that way. Until there is one, a module ships a unit that stays, which is also
the better shape: how a machine enforces rules is a fact about the machine, and the mesh has no
business depending on what a distribution happens to package.

**One thing is derived from what is assigned and not yet from the overlay's shape**, and it is
stated here rather than discovered: **a hub's own listening port.** A hub accepts inbound
connections from every node at other sites; a node that is not a hub dials out and needs nothing
open, because a reply to a flow it started is already accepted. So the two want different rules on
an identical module — and `listens` is a static field on a manifest, while the overlay module's
resources are computed per node.

A machine that is not a hub is therefore correct today, and **a hub would have its own port closed
by a rule set derived this way.** The fix is that a computed module contributes listens the way it
contributes resources; until it exists, the firewall belongs on machines that are not hubs, and
this paragraph is the reason rather than an oversight to find later.

**And it is enforced, which is what separates this from `scope:`.** Checked on two real machines:
two ports opened, one declared, and from the other machine the declared one answers and the
undeclared one does not — then the module is removed and the port closes with nobody editing a
rule. *A rule set that is written but never loaded passes every check that reads the file, which
is why the check reads packets.*

## 5 — Certificates

**Two authorities, kept separate on purpose.**

| | issued by | for |
|---|---|---|
| **public names** | a public ACME authority | anything outside the mesh reaches |
| **internal names** | the **mesh CA** | node-to-node, over the overlay |

**The split is not collapsed, including in the lab.** A single-CA lab would hide any bug living
in the split, so the lab runs its own ACME issuer on its public segment and keeps the mesh CA
unchanged ([research 004](../../01-RESEARCH/004-lab-network/00-overview.md)).

**Public issuance requires genuine public reachability.** The HTTP-01 challenge must be answered
at the name being certified, so issuance happens through a publicly reachable node regardless of
where the workload runs — the same asymmetry as exposure, for the same reason.

**The issuer must be configurable.** Today it is not: the proxy sets no `caServer` and therefore
defaults to the public authority's *production* endpoint. Two consequences, and the second is
worse than the lab problem that found it — every certificate experiment on a real node consumes
production issuance quota, and a retry loop can exhaust it for a week.

**The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the
fingerprint in its token ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)),
so nothing needs the CA before membership. It certifies internal names afterwards, and that is
all it does.

### What was built

*2026-08-31.*

**A node generates a fourth key**, and the reason is the one the other three already give: *a key
used for two purposes is one rotation away from breaking the other.* The identity key would work
for TLS and reusing it would mean rotating a node's identity every time its certificate is
replaced. The private half never leaves the machine; the mesh is told the public half at
enrolment.

**So there is no certificate request and nothing to seal.** The mesh signs a statement binding a
public key to a name it alone assigns, which is the whole of what a certificate authority does.
It issues rather than stores: the node's key does not change, so signing again produces an equally
valid certificate and there is nothing to keep in step.

**A machine with no name inside the mesh is refused**, not given a certificate for nothing. A
certificate for a name nothing resolves is a certificate nothing can check.

**And the key is stored in the format a server reads** — PKCS#8 PEM, not the host's own encoding.
That is not an implementation detail of whoever writes the file: the file exists *because
something else reads it*, so the format is the interface
([04-ISSUES/014](../../04-ISSUES/014-a-key-that-is-present-and-unusable/00-report.md)).

*Checked by a real handshake between two machines: one serves on its internal name with the key it
generated, the other verifies against the mesh's authority and nothing else. Every cheaper check
passed while the server could not start — the key was present, the certificate was valid, and
nothing read either the way a server would.*

## What this removes

The list is worth having in one place, because it is most of the argument:

- **The last two direct database connections from nodes** — `wireguard` and `traefik`, the only
  two, both connectivity.
- **Therefore the database credential on every node**, and the object-store credential beside it.
  [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s central claim becomes
  true rather than aspirational.
- **The `/etc/hosts` floor**, and the bootstrap circularity it patched.
- **Hub election by address prefix**, and the silent no-hub failure when nobody knew the
  convention.
- **The RFC1918 inference**, and the lab substitution that existed to satisfy it.
- **`scope:`**, and the class of manifest key that means nothing.

## Open

- ~~**What happens when the hub is down.**~~ **Resolved** by
  [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), together with `06`'s
  matching question — they were one question. Nothing takes over. WireGuard has no failover, the
  hub is declared rather than elected, and non-co-located paths stop while co-located direct peers
  and every already-assigned workload keep running. The recovery path is restore, and its deadline
  is certificate renewal.
- **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from
  an address, but no procedure exists, and a graph delivered node by node has an ordering problem
  while it is half-applied.
- **Revoking a route** when a module is unassigned ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)).
  A stale public name pointing at nothing fails more visibly than a stale grant.
- **IPv6.** [ADR 0007](../../02-DECISIONS/0007-connectivity.md) makes
  it expressible; nothing here says the overlay or the resolver handle it.
- **Reporting declared-versus-observed.** ADR 0007 makes the disagreement detectable and does not
  say who looks or what they are told.