novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Design connectivity in full: overlay, resolution, exposure, filtering, certificates

Browse Source 
Written as one document because the five are one design. They share inputs,
they must agree, and every one of them today is computed in a different place
by a different module from a different copy of the same facts.

The through-line is that none of the five can be answered by a machine alone,
so all five are decided centrally and delivered as `file` resources. That costs
no new host vocabulary and removes both remaining direct database connections
from nodes -- wireguard and traefik are the only two, and both are connectivity.

Three decisions fall out, all proposed:

0050 -- reachability is declared, not inferred from an address. The RFC1918
regex is wrong for carrier-grade NAT (100.64/10 tests as public, so an endpoint
is written to an address nothing can reach), wrong for IPv6, and wrong for a
routable address behind a closed firewall. The lab needing TEST-NET-3 to
satisfy the regex is the same bug from the other side. Also kills hub election
by address prefix, which fails silently and makes renumbering an outage.

0051 -- the enrolment token carries where the mesh is and how to recognise it.
Closes two circles with one mechanism: verifying the mesh needed the CA, and
obtaining the CA meant trusting whoever handed it over; and a node had to reach
the mesh before it could resolve any mesh name. An address plus a fingerprint,
carried out of band, resolves both -- and closes the CA question 0049 deferred.

0052 -- a filter rule names its source. `scope:` is declared in five manifests,
is part of no rule type, and is referenced by no code, so those manifests
appear to restrict ports and restrict nothing. Removed rather than implemented;
the general fix is refusing unknown keys, which the host already does and
manifests do not.

Also corrects two claims in 0049 asserting wireguard was already handled.
Research 006 says both modules still reach upward; neither is.
This commit is contained in:
jschoubben
2026-08-27 00:36:49 +02:00
parent 8d9282d86b
commit 4e80820e2f
7 changed files with 557 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
03-DESIGN/01-to-be/06-the-control-plane.md
+1 -1
View File
@@ -49,7 +49,7 @@ Ten contexts and one interface, from the skeleton
| **record** | the event log every other context integrates through |
| **inventory** | nodes, modules, assignments, versions |
| **config** | settings, secrets, and deriving them onto nodes |
| **connectivity** | overlay, resolution, exposure, filtering, certificates — it *decides* routes; the proxy on a node applies them ([ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md)) |
| **connectivity** | overlay, resolution, exposure, filtering, certificates — **specified in full in [`08-connectivity.md`](08-connectivity.md)** |
| **provisioning** | resource grants between modules |
| **delivery** | source to artifact to node |
| **observability** | health, logs, metrics, alerts |
03-DESIGN/01-to-be/08-connectivity.md
+233
View File
@@ -0,0 +1,233 @@
---
layer: to-be
status: designed
code: []
updated: 2026-08-27
decisions:
- 02-DECISIONS/0037-the-host-applies-it-does-not-decide.md
- 02-DECISIONS/0039-the-link-is-the-security-boundary.md
- 02-DECISIONS/0049-a-route-is-a-grant.md
- 02-DECISIONS/0050-reachability-is-a-property-of-the-address.md
- 02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md
- 02-DECISIONS/0052-a-filter-rule-names-its-source.md
---
# Connectivity
One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most
moving parts: **overlay, resolution, exposure, filtering, certificates.**
It is written as a whole because the five are one design. They share inputs, they must agree, and
every one of them today is computed in a different place by a different module from a different
copy of the same facts.
## Why it is control-plane work
Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one
node* — to each responsibility:
| | needs to know | whose |
|---|---|---|
| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane |
| **resolution** — which name is which node | **every node** | control plane |
| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md)) | control plane |
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies |
| **certificates** — who may present which name | which name belongs to which node | control plane |
**Not one of the five can be answered by a machine on its own.** That is the whole reason this is
a context rather than a set of node-local modules — and it is exactly what the current
arrangement gets wrong, by computing all five on the node from a direct database connection.
## The shape: decided centrally, delivered as files
Every one of the five resolves the same way, and it is worth stating once rather than five times:
> **The connectivity context computes the configuration. It arrives over the link as `file`
> resources. The service reads files and knows nothing about the mesh.**
This costs **no new host vocabulary**. `file`, `directory`, `service` and `container` already
exist; WireGuard, the resolver and the proxy are all *a container or a package, plus files*.
It is also what removes the last two upward dependencies.
[Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules
opening a direct connection to the control plane's database — `wireguard` and `traefik` — and
they are the reason every node permanently holds a credential to it
([ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)). Both are connectivity
modules. **Closing this context closes that set.**
## The order it comes up in
The one thing to get right, because everything else depends on it:
```
0 the node has an underlay address the machine's own — DHCP, or a provider gave it one
1 the node dials the mesh OVER THE UNDERLAY, at the address in its token
2 it proves itself, and is proved to the link exists (ADR 0039, ADR 0051)
3 the mesh grants it an identity and an overlay address
4 the overlay comes up peer graph delivered as files
5 names resolve resolver config delivered as files
6 filtering is applied derived from what is assigned here
7 routes and certificates once this node has something to expose
```
**Step 1 runs on the underlay and never on the overlay.** This is the circularity that must not
be created: the overlay is configured by the mesh, so a link that required the overlay could
never be established on a new node. The link stays on the underlay permanently — it is
outbound-only and carries its own identity, so it needs nothing the overlay provides.
**Nothing before step 3 can resolve a mesh name**, which is why the token carries an *address*
([ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md)). Today this is
patched with an `/etc/hosts` floor written underneath the resolver; under this design there is
nothing to patch.
## 1 — The overlay
**What is decided:** the peer graph. For every node: its overlay address, which peers it holds,
which of those it may dial, and which must dial it.
**Inputs, all declared:**
- **reachability** — an endpoint, or none
([ADR 0050](../../02-DECISIONS/0050-reachability-is-a-property-of-the-address.md)). Not
inferred from the address shape, which is wrong for carrier-grade NAT, wrong for IPv6, and
wrong for a routable address behind a closed firewall.
- **site** — where the machine physically is, or nothing if it roams.
- **role** — hub or not, **declared**. Today it is inferred from an address prefix, which means a
renumbering is an outage and nothing can be asked which node is the hub.
**Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the
public key is published to the mesh. This is already true and it is already right — it is
[ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)'s *a node holds its own
identity* applied to the overlay, and it means the control plane computes a graph it cannot
itself impersonate.
**Shape: a hub, with direct peering between co-located nodes.**
| | |
|---|---|
| two nodes at the same site | peer **directly**, host-routed, with a keepalive |
| everything else | routes through the **hub** |
| a node with no site — it roams | **hub only** |
**Roaming is hub-only deliberately, and the reason is a property of WireGuard rather than a
preference: there is no failover.** A more specific route to a dead endpoint blackholes; it does
not fall back to the general one. So a node whose location changes gets exactly one path, because
two paths would mean one of them silently swallowing traffic.
**What the host receives:** an interface configuration and a peer list, as files. It does not
compute them, and after this it holds no credential to the mesh's database.
## 2 — Resolution
**Two name spaces, and they do not mix:**
| | resolves to | certified by |
|---|---|---|
| **internal names** | overlay addresses | the **mesh CA** |
| **public names** | whatever the outside world must reach | a **public authority** |
A node's mesh name is its overlay address. Its public name, if it has one, is a separate fact
used by things outside the mesh — and the separation carries two lessons that were learned
expensively enough to be worth restating:
- **Mesh names are not multicast names.** A name resolved by local multicast discovery introduces
a delay and a failure mode that appears on one node and not others — the worst shape a fault
can have.
- **A node must not pin its own public name locally.** The duplicate record breaks resolution of
that name for everything else that needs it.
**What the host receives:** the resolver's configuration, as files, listing every peer's internal
name and overlay address.
**What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh
database before its own DNS existed; with [ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md)
nothing needs a name before the link, and a fallback nothing needs is a path nothing tests.
## 3 — Exposure
Settled by [ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md); summarised here because
this is where it belongs.
**A route is a grant.** A module that must be reachable declares it needs one; the proxy provides
it and hands back the public name. Ordinary
[ADR 0044](../../02-DECISIONS/0044-a-module-declares-presence-instantiation-and-exclusion.md)
vocabulary — the mirror of a database grant, where the consumer supplies a target and receives a
name rather than supplying nothing and receiving credentials.
**A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is
the case is a mesh-level fact, which is the fourth reason exposure is control-plane work.
## 4 — Filtering
**Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a
consequence of what runs on it and who must reach it, not an independent declaration to keep in
step by hand.
**A rule names its source** ([ADR 0052](../../02-DECISIONS/0052-a-filter-rule-names-its-source.md)).
A rule with no source is open, and must say so rather than appear to restrict something. `scope:`
is removed rather than implemented: five manifests carry it today, it is referenced by no code,
and it is the clearest instance in the repository of *an unenforced rule is indistinguishable
from a wrong one, and costs more, because people believe it.*
**Unknown keys are refused** — the discipline the host's declaration parser already has
([ADR 0043](../../02-DECISIONS/0043-a-declaration-is-an-ordered-list-of-owned-resources.md)), and
the one manifests lack. `scope:` survived because nothing rejected it.
## 5 — Certificates
**Two authorities, kept separate on purpose.**
| | issued by | for |
|---|---|---|
| **public names** | a public ACME authority | anything outside the mesh reaches |
| **internal names** | the **mesh CA** | node-to-node, over the overlay |
**The split is not collapsed, including in the lab.** A single-CA lab would hide any bug living
in the split, so the lab runs its own ACME issuer on its public segment and keeps the mesh CA
unchanged ([research 004](../../01-RESEARCH/004-lab-network/00-overview.md)).
**Public issuance requires genuine public reachability.** The HTTP-01 challenge must be answered
at the name being certified, so issuance happens through a publicly reachable node regardless of
where the workload runs — the same asymmetry as exposure, for the same reason.
**The issuer must be configurable.** Today it is not: the proxy sets no `caServer` and therefore
defaults to the public authority's *production* endpoint. Two consequences, and the second is
worse than the lab problem that found it — every certificate experiment on a real node consumes
production issuance quota, and a retry loop can exhaust it for a week.
**The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the
fingerprint in its token ([ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md)),
so nothing needs the CA before membership. It certifies internal names afterwards, and that is
all it does.
## What this removes
The list is worth having in one place, because it is most of the argument:
- **The last two direct database connections from nodes** — `wireguard` and `traefik`, the only
two, both connectivity.
- **Therefore the database credential on every node**, and the object-store credential beside it.
[ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)'s central claim becomes
true rather than aspirational.
- **The `/etc/hosts` floor**, and the bootstrap circularity it patched.
- **Hub election by address prefix**, and the silent no-hub failure when nobody knew the
convention.
- **The RFC1918 inference**, and the lab substitution that existed to satisfy it.
- **`scope:`**, and the class of manifest key that means nothing.
## Open
- **What happens when the hub is down.** WireGuard has no failover, and the hub is a single point
through which every non-co-located pair routes. This design does not add a second hub and does
not pretend the first one is redundant. It is the same shape as
[`06`](06-the-control-plane.md)'s *how many run, and what a node does without one*, and it
should be answered with it rather than separately.
- **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from
an address, but no procedure exists, and a graph delivered node by node has an ordering problem
while it is half-applied.
- **Revoking a route** when a module is unassigned ([ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md)).
A stale public name pointing at nothing fails more visibly than a stale grant.
- **IPv6.** [ADR 0050](../../02-DECISIONS/0050-reachability-is-a-property-of-the-address.md) makes
it expressible; nothing here says the overlay or the resolver handle it.
- **Reporting declared-versus-observed.** ADR 0050 makes the disagreement detectable and does not
say who looks or what they are told.
03-DESIGN/01-to-be/README.md
+6 -4
View File
@@ -16,13 +16,15 @@ document is written and this one's status becomes `implemented`.
| [`04-lab-installation.md`](04-lab-installation.md) | Getting the lab onto a clean machine, and why it verifies capability rather than installation | [ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md) |
| [`05-the-node-host.md`](05-the-node-host.md) | Tier 0 — the one thing installed by hand, and the only thing that changes a machine | [ADR 0037](../../02-DECISIONS/0037-the-host-applies-it-does-not-decide.md) |
| [`06-the-control-plane.md`](06-the-control-plane.md) | Tier 2 — what the term means, and the test for what belongs in it | [ADR 0037](../../02-DECISIONS/0037-the-host-applies-it-does-not-decide.md) |
| [`07-the-substrate.md`](07-the-substrate.md) | Tier 1 — what the control plane consumes and cannot grant itself | [ADR 0038](../../02-DECISIONS/0038-a-node-joins-by-linking-first.md) |
| [`07-the-substrate.md`](07-the-substrate.md) | Tier 1 — what the control plane consumes and cannot grant itself | [ADR 0038](../../02-DECISIONS/0038-a-node-joins-by-linking-first.md), [0048](../../02-DECISIONS/0048-the-substrate-is-named.md) |
| [`08-connectivity.md`](08-connectivity.md) | One context in full — overlay, resolution, exposure, filtering, certificates | [ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md), [0050](../../02-DECISIONS/0050-reachability-is-a-property-of-the-address.md), [0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md) |
## Not yet written
- **The eight bounded contexts.** [ADR 0015](../../02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md)
decides the decomposition; the per-context specifications do not exist yet. The work
breakdown says in what order they are needed.
- **The remaining contexts.** [ADR 0015](../../02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md)
decides the decomposition; `connectivity` is the first written in full
([`08`](08-connectivity.md)) and the others do not exist yet. The work breakdown says in what
order they are needed.
- **Domain grouping outside the core.** [ADR 0017](../../02-DECISIONS/0017-modules-outside-the-core-are-grouped-by-domain.md)
settles the principle and explicitly does not settle the domain list. That is a research
effort, not a design document, until it concludes.