jschoubben
918dc04916
Four things settled by talking them through, all of which had been true in somebody's head and written nowhere. It is not a mesh in the peer-to-peer sense and will not become one. 0001 now says what it is instead: machines linked by a private network, one node holding knowledge of all of them, modules as the way anything is built and delivered, and agents hired onto nodes to do the work. The word describes what machines can reach, not how they are governed. "Master" overstates it the other way -- nothing needs that node to keep running, only to change. 0006 gains the option that would make it a real mesh, recorded as considered rather than rejected by silence: every node holding the whole inventory, a replication process, an elected master with promotion on failure. What settles it is not the complexity but that it still would not deliver the name, because application databases are not replicated -- so a genuine peer-to-peer mesh means becoming a replicated database system for every consumer's data too. That is a larger product than the thing it would support. Also in 0006: three central roles, not one. Losing the control plane costs change, losing the broker costs being told anything, and losing the hub costs nodes in different places reaching each other at all -- which is operation, not administration. Whether they are one node is not decided. And SSH access is identity's. It appeared three times as something that uses the overlay and never as something the mesh provides, which reads as settled when nothing decided it. Nobody else could: the mesh is the only thing that knows which humans and agents exist and which nodes they may reach. Node to node SSH stays out -- the host has no inbound control surface by decision, and nodes reaching each other that way is a second control path through the back door. 0007 gains the requirement underneath all of it. Reachability was recorded as a fact to track and never as a thing some node must have. The broker's node and the hub must be dialable by every node at a stable address, or nothing can join and a disconnected node cannot return. A mesh entirely behind NAT cannot be raised. That is a precondition and it belongs with the others. The link staying on the underlay is also argued now rather than asserted. At join time it is forced; afterwards it is a choice, and the reason is that a repair channel carried over the thing being repaired is not one. Moving it onto the overlay, with fallback, is recorded as open with what it would have to get right -- a WireGuard interface has no link state to test, and a silent fallback is this repository's recurring fault in a new place. 0010 says in one line what was the intention throughout: the module system is the CI/CD. Not a pipeline beside the mesh. Build, test, publish and deploy are one reconciliation seen at four points, which is why a thing that cannot be a module cannot be delivered.
247 lines
13 KiB
Markdown
247 lines
13 KiB
Markdown
---
|
|
layer: to-be
|
|
status: designed
|
|
code: []
|
|
updated: 2026-08-27
|
|
decisions:
|
|
- 02-DECISIONS/0005-the-node-host.md
|
|
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
|
- 02-DECISIONS/0007-connectivity.md
|
|
- 02-DECISIONS/0007-connectivity.md
|
|
- 02-DECISIONS/0004-a-node-and-how-it-joins.md
|
|
- 02-DECISIONS/0007-connectivity.md
|
|
- 02-DECISIONS/0006-the-substrate-and-the-control-plane.md
|
|
---
|
|
|
|
# Connectivity
|
|
|
|
One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most
|
|
moving parts: **overlay, resolution, exposure, filtering, certificates.**
|
|
|
|
It is written as a whole because the five are one design. They share inputs, they must agree, and
|
|
every one of them today is computed in a different place by a different module from a different
|
|
copy of the same facts.
|
|
|
|
## Why it is control-plane work
|
|
|
|
Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one
|
|
node* — to each responsibility:
|
|
|
|
| | needs to know | whose |
|
|
|---|---|---|
|
|
| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane |
|
|
| **resolution** — which name is which node | **every node** | control plane |
|
|
| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)) | control plane |
|
|
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies |
|
|
| **certificates** — who may present which name | which name belongs to which node | control plane |
|
|
|
|
**Not one of the five can be answered by a machine on its own.** That is the whole reason this is
|
|
a context rather than a set of node-local modules — and it is exactly what the current
|
|
arrangement gets wrong, by computing all five on the node from a direct database connection.
|
|
|
|
## The shape: decided centrally, delivered as files
|
|
|
|
Every one of the five resolves the same way, and it is worth stating once rather than five times:
|
|
|
|
> **The connectivity context computes the configuration. It arrives over the link as `file`
|
|
> resources. The service reads files and knows nothing about the mesh.**
|
|
|
|
This costs **no new host vocabulary**. `file`, `directory`, `service` and `container` already
|
|
exist; WireGuard, the resolver and the proxy are all *a container or a package, plus files*.
|
|
|
|
It is also what removes the last two upward dependencies.
|
|
[Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules
|
|
opening a direct connection to the control plane's database — `wireguard` and `traefik` — and
|
|
they are the reason every node permanently holds a credential to it
|
|
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Both are connectivity
|
|
modules. **Closing this context closes that set.**
|
|
|
|
## The order it comes up in
|
|
|
|
The one thing to get right, because everything else depends on it:
|
|
|
|
```
|
|
0 the node has an underlay address the machine's own — DHCP, or a provider gave it one
|
|
1 the node dials the mesh OVER THE UNDERLAY, at the address in its token
|
|
2 it proves itself, and is proved to the link exists (ADR 0004, ADR 0004)
|
|
3 the mesh grants it an identity and an overlay address
|
|
4 the overlay comes up peer graph delivered as files
|
|
5 names resolve resolver config delivered as files
|
|
6 filtering is applied derived from what is assigned here
|
|
7 routes and certificates once this node has something to expose
|
|
```
|
|
|
|
**Step 1 runs on the underlay and never on the overlay.** This is the circularity that must not
|
|
be created: the overlay is configured by the mesh, so a link that required the overlay could
|
|
never be established on a new node. The link stays on the underlay permanently — it is
|
|
outbound-only and carries its own identity, so it needs nothing the overlay provides.
|
|
|
|
**Nothing before step 3 can resolve a mesh name**, which is why the token carries an *address*
|
|
([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)). Today this is
|
|
patched with an `/etc/hosts` floor written underneath the resolver; under this design there is
|
|
nothing to patch.
|
|
|
|
**Step 1 has a precondition this document treated as a fact to record rather than a requirement:
|
|
the broker's node must be dialable by every node, at a stable address, and so must the hub**
|
|
([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Across the internet that means publicly
|
|
reachable; on one network it does not. A mesh whose nodes are all behind NAT cannot be raised, and
|
|
a broker node whose address moves invalidates every token issued for it.
|
|
|
|
**Whether the link should later move onto the overlay, with the underlay as fallback, is
|
|
[open](../../02-DECISIONS/0007-connectivity.md).** It is a decision rather than a derivation: the
|
|
gain is which network carries bytes, not what an attacker can reach, since the link is already
|
|
encrypted against a pinned fingerprint.
|
|
|
|
## 1 — The overlay
|
|
|
|
**What is decided:** the peer graph. For every node: its overlay address, which peers it holds,
|
|
which of those it may dial, and which must dial it.
|
|
|
|
**Inputs, all declared:**
|
|
|
|
- **reachability** — an endpoint, or none
|
|
([ADR 0007](../../02-DECISIONS/0007-connectivity.md)). Not
|
|
inferred from the address shape, which is wrong for carrier-grade NAT, wrong for IPv6, and
|
|
wrong for a routable address behind a closed firewall.
|
|
- **site** — where the machine physically is, or nothing if it roams.
|
|
- **role** — hub or not, **declared**. Today it is inferred from an address prefix, which means a
|
|
renumbering is an outage and nothing can be asked which node is the hub.
|
|
|
|
**Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the
|
|
public key is published to the mesh. This is already true and it is already right — it is
|
|
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s *a node holds its own
|
|
identity* applied to the overlay, and it means the control plane computes a graph it cannot
|
|
itself impersonate.
|
|
|
|
**Shape: a hub, with direct peering between co-located nodes.**
|
|
|
|
| | |
|
|
|---|---|
|
|
| two nodes at the same site | peer **directly**, host-routed, with a keepalive |
|
|
| everything else | routes through the **hub** |
|
|
| a node with no site — it roams | **hub only** |
|
|
|
|
**Roaming is hub-only deliberately, and the reason is a property of WireGuard rather than a
|
|
preference: there is no failover.** A more specific route to a dead endpoint blackholes; it does
|
|
not fall back to the general one. So a node whose location changes gets exactly one path, because
|
|
two paths would mean one of them silently swallowing traffic.
|
|
|
|
**What the host receives:** an interface configuration and a peer list, as files. It does not
|
|
compute them, and after this it holds no credential to the mesh's database.
|
|
|
|
## 2 — Resolution
|
|
|
|
**Two name spaces, and they do not mix:**
|
|
|
|
| | resolves to | certified by |
|
|
|---|---|---|
|
|
| **internal names** | overlay addresses | the **mesh CA** |
|
|
| **public names** | whatever the outside world must reach | a **public authority** |
|
|
|
|
A node's mesh name is its overlay address. Its public name, if it has one, is a separate fact
|
|
used by things outside the mesh — and the separation carries two lessons that were learned
|
|
expensively enough to be worth restating:
|
|
|
|
- **Mesh names are not multicast names.** A name resolved by local multicast discovery introduces
|
|
a delay and a failure mode that appears on one node and not others — the worst shape a fault
|
|
can have.
|
|
- **A node must not pin its own public name locally.** The duplicate record breaks resolution of
|
|
that name for everything else that needs it.
|
|
|
|
**What the host receives:** the resolver's configuration, as files, listing every peer's internal
|
|
name and overlay address.
|
|
|
|
**What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh
|
|
database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)
|
|
nothing needs a name before the link, and a fallback nothing needs is a path nothing tests.
|
|
|
|
## 3 — Exposure
|
|
|
|
Settled by [ADR 0007](../../02-DECISIONS/0007-connectivity.md); summarised here because
|
|
this is where it belongs.
|
|
|
|
**A route is a grant.** A module that must be reachable declares it needs one; the proxy provides
|
|
it and hands back the public name. Ordinary
|
|
[ADR 0009](../../02-DECISIONS/0009-modules-and-the-graph.md)
|
|
vocabulary — the mirror of a database grant, where the consumer supplies a target and receives a
|
|
name rather than supplying nothing and receiving credentials.
|
|
|
|
**A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is
|
|
the case is a mesh-level fact, which is the fourth reason exposure is control-plane work.
|
|
|
|
## 4 — Filtering
|
|
|
|
**Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a
|
|
consequence of what runs on it and who must reach it, not an independent declaration to keep in
|
|
step by hand.
|
|
|
|
**A rule names its source** ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)).
|
|
A rule with no source is open, and must say so rather than appear to restrict something. `scope:`
|
|
is removed rather than implemented: five manifests carry it today, it is referenced by no code,
|
|
and it is the clearest instance in the repository of *an unenforced rule is indistinguishable
|
|
from a wrong one, and costs more, because people believe it.*
|
|
|
|
**Unknown keys are refused** — the discipline the host's declaration parser already has
|
|
([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)), and
|
|
the one manifests lack. `scope:` survived because nothing rejected it.
|
|
|
|
## 5 — Certificates
|
|
|
|
**Two authorities, kept separate on purpose.**
|
|
|
|
| | issued by | for |
|
|
|---|---|---|
|
|
| **public names** | a public ACME authority | anything outside the mesh reaches |
|
|
| **internal names** | the **mesh CA** | node-to-node, over the overlay |
|
|
|
|
**The split is not collapsed, including in the lab.** A single-CA lab would hide any bug living
|
|
in the split, so the lab runs its own ACME issuer on its public segment and keeps the mesh CA
|
|
unchanged ([research 004](../../01-RESEARCH/004-lab-network/00-overview.md)).
|
|
|
|
**Public issuance requires genuine public reachability.** The HTTP-01 challenge must be answered
|
|
at the name being certified, so issuance happens through a publicly reachable node regardless of
|
|
where the workload runs — the same asymmetry as exposure, for the same reason.
|
|
|
|
**The issuer must be configurable.** Today it is not: the proxy sets no `caServer` and therefore
|
|
defaults to the public authority's *production* endpoint. Two consequences, and the second is
|
|
worse than the lab problem that found it — every certificate experiment on a real node consumes
|
|
production issuance quota, and a retry loop can exhaust it for a week.
|
|
|
|
**The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the
|
|
fingerprint in its token ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)),
|
|
so nothing needs the CA before membership. It certifies internal names afterwards, and that is
|
|
all it does.
|
|
|
|
## What this removes
|
|
|
|
The list is worth having in one place, because it is most of the argument:
|
|
|
|
- **The last two direct database connections from nodes** — `wireguard` and `traefik`, the only
|
|
two, both connectivity.
|
|
- **Therefore the database credential on every node**, and the object-store credential beside it.
|
|
[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)'s central claim becomes
|
|
true rather than aspirational.
|
|
- **The `/etc/hosts` floor**, and the bootstrap circularity it patched.
|
|
- **Hub election by address prefix**, and the silent no-hub failure when nobody knew the
|
|
convention.
|
|
- **The RFC1918 inference**, and the lab substitution that existed to satisfy it.
|
|
- **`scope:`**, and the class of manifest key that means nothing.
|
|
|
|
## Open
|
|
|
|
- ~~**What happens when the hub is down.**~~ **Resolved** by
|
|
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), together with `06`'s
|
|
matching question — they were one question. Nothing takes over. WireGuard has no failover, the
|
|
hub is declared rather than elected, and non-co-located paths stop while co-located direct peers
|
|
and every already-assigned workload keep running. The recovery path is restore, and its deadline
|
|
is certificate renewal.
|
|
- **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from
|
|
an address, but no procedure exists, and a graph delivered node by node has an ordering problem
|
|
while it is half-applied.
|
|
- **Revoking a route** when a module is unassigned ([ADR 0007](../../02-DECISIONS/0007-connectivity.md)).
|
|
A stale public name pointing at nothing fails more visibly than a stale grant.
|
|
- **IPv6.** [ADR 0007](../../02-DECISIONS/0007-connectivity.md) makes
|
|
it expressible; nothing here says the overlay or the resolver handle it.
|
|
- **Reporting declared-versus-observed.** ADR 0007 makes the disagreement detectable and does not
|
|
say who looks or what they are told.
|