--- layer: to-be status: designed code: [] updated: 2026-08-27 decisions: - 02-DECISIONS/0005-the-node-host.md - 02-DECISIONS/0004-a-node-and-how-it-joins.md - 02-DECISIONS/0007-connectivity.md - 02-DECISIONS/0007-connectivity.md - 02-DECISIONS/0004-a-node-and-how-it-joins.md - 02-DECISIONS/0007-connectivity.md - 02-DECISIONS/0006-the-substrate-and-the-control-plane.md --- # Connectivity One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most moving parts: **overlay, resolution, exposure, filtering, certificates.** It is written as a whole because the five are one design. They share inputs, they must agree, and every one of them today is computed in a different place by a different module from a different copy of the same facts. ## Why it is control-plane work Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one node* — to each responsibility: | | needs to know | whose | |---|---|---| | **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane | | **resolution** — which name is which node | **every node** | control plane | | **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | control plane | | **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies | | **certificates** — who may present which name | which name belongs to which node | control plane | **Not one of the five can be answered by a machine on its own.** That is the whole reason this is a context rather than a set of node-local modules — and it is exactly what the current arrangement gets wrong, by computing all five on the node from a direct database connection. ## The shape: decided centrally, delivered as files Every one of the five resolves the same way, and it is worth stating once rather than five times: > **The connectivity context computes the configuration. It arrives over the link as `file` > resources. The service reads files and knows nothing about the mesh.** This costs **no new host vocabulary**. `file`, `directory`, `service` and `container` already exist; WireGuard, the resolver and the proxy are all *a container or a package, plus files*. It is also what removes the last two upward dependencies. [Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules opening a direct connection to the control plane's database — `wireguard` and `traefik` — and they are the reason every node permanently holds a credential to it ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Both are connectivity modules. **Closing this context closes that set.** ## The order it comes up in The one thing to get right, because everything else depends on it: ``` 0 the node has an underlay address the machine's own — DHCP, or a provider gave it one 1 the node dials the mesh OVER THE UNDERLAY, at the address in its token 2 it proves itself, and is proved to the link exists (ADR 0004, ADR 0004) 3 the mesh grants it an identity and an overlay address 4 the overlay comes up peer graph delivered as files 5 names resolve resolver config delivered as files 6 filtering is applied derived from what is assigned here 7 routes and certificates once this node has something to expose ``` **Step 1 runs on the underlay and never on the overlay.** This is the circularity that must not be created: the overlay is configured by the mesh, so a link that required the overlay could never be established on a new node. The link stays on the underlay permanently — it is outbound-only and carries its own identity, so it needs nothing the overlay provides. **Nothing before step 3 can resolve a mesh name**, which is why the token carries an *address* ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Today this is patched with an `/etc/hosts` floor written underneath the resolver; under this design there is nothing to patch. **Step 1 has a precondition this document treated as a fact to record rather than a requirement: the broker's node must be dialable by every node, at a stable address, and so must the hub** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Across the internet that means publicly reachable; on one network it does not. A mesh whose nodes are all behind NAT cannot be raised, and a broker node whose address moves invalidates every token issued for it. **Whether the link should later move onto the overlay, with the underlay as fallback, is [open](../../02-DECISIONS/0007-connectivity.md).** It is a decision rather than a derivation: the gain is which network carries bytes, not what an attacker can reach, since the link is already encrypted against a pinned fingerprint. ## 1 — The overlay **What is decided:** the peer graph. For every node: its overlay address, which peers it holds, which of those it may dial, and which must dial it. **Inputs, all declared:** - **reachability** — an endpoint, or none ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Not inferred from the address shape, which is wrong for carrier-grade NAT, wrong for IPv6, and wrong for a routable address behind a closed firewall. - **site** — where the machine physically is, or nothing if it roams. - **role** — hub or not, **declared**. Today it is inferred from an address prefix, which means a renumbering is an outage and nothing can be asked which node is the hub. **Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the public key is published to the mesh. This is already true and it is already right — it is [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *a node holds its own identity* applied to the overlay, and it means the control plane computes a graph it cannot itself impersonate. **Shape: a hub, with direct peering between co-located nodes.** | | | |---|---| | two nodes at the same site | peer **directly**, host-routed, with a keepalive | | everything else | routes through the **hub** | | a node with no site — it roams | **hub only** | **Roaming is hub-only deliberately, and the reason is a property of WireGuard rather than a preference: there is no failover.** A more specific route to a dead endpoint blackholes; it does not fall back to the general one. So a node whose location changes gets exactly one path, because two paths would mean one of them silently swallowing traffic. **What the host receives:** an interface configuration and a peer list, as files. It does not compute them, and after this it holds no credential to the mesh's database. ## 2 — Resolution **Two name spaces, and they do not mix:** | | resolves to | certified by | |---|---|---| | **internal names** | overlay addresses | the **mesh CA** | | **public names** | whatever the outside world must reach | a **public authority** | A node's mesh name is its overlay address. Its public name, if it has one, is a separate fact used by things outside the mesh — and the separation carries two lessons that were learned expensively enough to be worth restating: - **Mesh names are not multicast names.** A name resolved by local multicast discovery introduces a delay and a failure mode that appears on one node and not others — the worst shape a fault can have. - **A node must not pin its own public name locally.** The duplicate record breaks resolution of that name for everything else that needs it. **What the host receives:** the resolver's configuration, as files, listing every peer's internal name and overlay address. **What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) nothing needs a name before the link, and a fallback nothing needs is a path nothing tests. ## 3 — Exposure Settled by [ADR 0007](../../02-DECISIONS/0007-connectivity.md); summarised here because this is where it belongs. **A route is a grant.** A module that must be reachable declares it needs one; the proxy provides it and hands back the public name. Ordinary [ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md) vocabulary — the mirror of a database grant, where the consumer supplies a target and receives a name rather than supplying nothing and receiving credentials. **A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is the case is a mesh-level fact, which is the fourth reason exposure is control-plane work. ## 4 — Filtering **Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a consequence of what runs on it and who must reach it, not an independent declaration to keep in step by hand. **A rule names its source** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). A rule with no source is open, and must say so rather than appear to restrict something. `scope:` is removed rather than implemented: five manifests carry it today, it is referenced by no code, and it is the clearest instance in the repository of *an unenforced rule is indistinguishable from a wrong one, and costs more, because people believe it.* **Unknown keys are refused** — the discipline the host's declaration parser already has ([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and the one manifests lack. `scope:` survived because nothing rejected it. ## 5 — Certificates **Two authorities, kept separate on purpose.** | | issued by | for | |---|---|---| | **public names** | a public ACME authority | anything outside the mesh reaches | | **internal names** | the **mesh CA** | node-to-node, over the overlay | **The split is not collapsed, including in the lab.** A single-CA lab would hide any bug living in the split, so the lab runs its own ACME issuer on its public segment and keeps the mesh CA unchanged ([research 004](../../01-RESEARCH/004-lab-network/00-overview.md)). **Public issuance requires genuine public reachability.** The HTTP-01 challenge must be answered at the name being certified, so issuance happens through a publicly reachable node regardless of where the workload runs — the same asymmetry as exposure, for the same reason. **The issuer must be configurable.** Today it is not: the proxy sets no `caServer` and therefore defaults to the public authority's *production* endpoint. Two consequences, and the second is worse than the lab problem that found it — every certificate experiment on a real node consumes production issuance quota, and a retry loop can exhaust it for a week. **The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the fingerprint in its token ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)), so nothing needs the CA before membership. It certifies internal names afterwards, and that is all it does. ## What this removes The list is worth having in one place, because it is most of the argument: - **The last two direct database connections from nodes** — `wireguard` and `traefik`, the only two, both connectivity. - **Therefore the database credential on every node**, and the object-store credential beside it. [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s central claim becomes true rather than aspirational. - **The `/etc/hosts` floor**, and the bootstrap circularity it patched. - **Hub election by address prefix**, and the silent no-hub failure when nobody knew the convention. - **The RFC1918 inference**, and the lab substitution that existed to satisfy it. - **`scope:`**, and the class of manifest key that means nothing. ## Open - ~~**What happens when the hub is down.**~~ **Resolved** by [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), together with `06`'s matching question — they were one question. Nothing takes over. WireGuard has no failover, the hub is declared rather than elected, and non-co-located paths stop while co-located direct peers and every already-assigned workload keep running. The recovery path is restore, and its deadline is certificate renewal. - **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from an address, but no procedure exists, and a graph delivered node by node has an ordering problem while it is half-applied. - **Revoking a route** when a module is unassigned ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). A stale public name pointing at nothing fails more visibly than a stale grant. - **IPv6.** [ADR 0007](../../02-DECISIONS/0007-connectivity.md) makes it expressible; nothing here says the overlay or the resolver handle it. - **Reporting declared-versus-observed.** ADR 0007 makes the disagreement detectable and does not say who looks or what they are told.