hq/03-DESIGN/01-to-be/08-connectivity.md

layer, status, code, updated, decisions

layer	status	code	updated	decisions
to-be	designed	mesh-control internal/catalogue/filtering.go mesh-control examples/route-proxy mesh-control internal/identity/authority.go mesh-host internal/identity/serving.go mesh-host internal/apply (the service that reflects a rule set)	2026-08-31	02-DECISIONS/0005-the-node-host.md 02-DECISIONS/0004-a-node-and-how-it-joins.md 02-DECISIONS/0007-connectivity.md 02-DECISIONS/0007-connectivity.md 02-DECISIONS/0004-a-node-and-how-it-joins.md 02-DECISIONS/0007-connectivity.md 02-DECISIONS/0006-the-substrate-and-the-control-plane.md

Connectivity

One of the control plane's ten contexts, and the one with the most moving parts: overlay, resolution, exposure, filtering, certificates.

It is written as a whole because the five are one design. They share inputs, they must agree, and every one of them today is computed in a different place by a different module from a different copy of the same facts.

Why it is control-plane work

Apply the test — everything that needs to know about more than one node — to each responsibility:

	needs to know	whose
overlay — who peers with whom, at what address	every node, and which of them can be dialled	control plane
resolution — which name is which node	every node	control plane
exposure — which public name reaches which container	which node is publicly reachable (ADR 0007)	control plane
filtering — which port is open, to whom	what is assigned here, and the overlay's shape	control plane decides, host applies
certificates — who may present which name	which name belongs to which node	control plane

Not one of the five can be answered by a machine on its own. That is the whole reason this is a context rather than a set of node-local modules — and it is exactly what the current arrangement gets wrong, by computing all five on the node from a direct database connection.

The shape: decided centrally, delivered as files

Every one of the five resolves the same way, and it is worth stating once rather than five times:

The connectivity context computes the configuration. It arrives over the link as file resources. The service reads files and knows nothing about the mesh.

This costs no new host vocabulary. file, directory, service and container already exist; WireGuard, the resolver and the proxy are all a container or a package, plus files.

It is also what removes the last two upward dependencies. Research 006 counted exactly two modules opening a direct connection to the control plane's database — wireguard and traefik — and they are the reason every node permanently holds a credential to it (ADR 0004). Both are connectivity modules. Closing this context closes that set.

And they are modules, not a second mechanism beside the module system

Written 2026-08-29, from building it. The first version was code beside the module system doing the module system's job, and the fault it produced is the point of writing this down.

A machine was on the private network because it had an address. Every node that had been placed got a peer list, whether or not anybody wanted it there, and there was no way to say a machine should stay off. That is what "special-cased" cost, and it was invisible until somebody wanted the exception.

What made it look unavoidable: a peer list cannot be written in a manifest. It is derived from every other machine, so it differs on each one and changes when any of them changes. So the manifest says its resources are computed — it names something in the control plane that works them out per node — and it is a module in every other respect: assigned, resolved, configured by settings, and absent from a machine nobody gave it to.

What that made possible immediately is the arrangement below, which the code has:

module	provides	requires	claims
the WireGuard one	a private network, and the mesh's own addressing		the private network, one per node
the names one	name resolution	the mesh's own addressing
networking		both of the above

Three rather than one, because WireGuard is one VPN of several. Naming the module after the job — networking — and putting WireGuard inside it is the retired flavor idea wearing a generic name: the second VPN has nowhere to go. So a module is named for what it is and declares what it does, and networking is the third row — requirements and no files (ADR 0009).

Names left the WireGuard module for their own. They had been delivered inside it, on the argument that a machine with peers and no names is half on the network. True, and the wrong place to fix it — names are identical over a different private network, so bundling them made one module out of two things. They require the mesh's addressing rather than a private network in general, because that is what they are computed from: over a VPN that hands out its own addresses the mesh has nothing to write, and refusing is what stops a machine being given a hosts file that means nothing on it.

And the claim is not decoration. Choosing a different VPN still installed WireGuard — dragged back in by the names, which needed addresses only WireGuard hands out — and nobody was told. Running two VPNs is not always wrong; being the one the mesh runs over is singular. So it is a claim, and the collision is refused by name.

And the proxy's half, which was the other module reaching into the database. A web application requiring a reverse proxy has to say which name, which port, and there was nowhere to put it — requires says a thing must exist and never said what to do with it. A module now contributes to a requirement, the control plane collects every contribution on a node, and the provider is given them as a file at a path it named. It reloads when that file changes, by the same restart-on the private network needed when a peer list changed under a running interface.

The proxy's configuration is not written by the mesh. It is given the facts and turns them into whatever it runs, which is why swapping Traefik for something else touches nothing that publishes through it. See ADR 0009 for the other direction — handing a credential back — which is the larger half and is not built.

What is still not a module, and why that is correct. The host needs none of this. It has an address and a route before the mesh exists — that is the machine's own networking — and the broker's address is carried in the token rather than resolved (ADR 0004). The one connection that carries modules cannot itself be one. Everything above it can be, and now is.

The order it comes up in

The one thing to get right, because everything else depends on it:

0  the node has an underlay address     the machine's own — DHCP, or a provider gave it one
1  the node dials the mesh              OVER THE UNDERLAY, at the address in its token
2  it proves itself, and is proved to   the link exists      (ADR 0004, ADR 0004)
3  the mesh grants it an identity       and an overlay address
4  the overlay comes up                 peer graph delivered as files
5  names resolve                        resolver config delivered as files
6  filtering is applied                 derived from what is assigned here
7  routes and certificates              once this node has something to expose

Step 1 runs on the underlay and never on the overlay. This is the circularity that must not be created: the overlay is configured by the mesh, so a link that required the overlay could never be established on a new node. The link stays on the underlay permanently — it is outbound-only and carries its own identity, so it needs nothing the overlay provides.

Nothing before step 3 can resolve a mesh name, which is why the token carries an address (ADR 0004). Today this is patched with an /etc/hosts floor written underneath the resolver; under this design there is nothing to patch.

Step 1 has a precondition this document treated as a fact to record rather than a requirement: the broker's node must be dialable by every node, at a stable address, and so must the hub (ADR 0007). Across the internet that means publicly reachable; on one network it does not. A mesh whose nodes are all behind NAT cannot be raised, and a broker node whose address moves invalidates every token issued for it.

Whether the link should later move onto the overlay, with the underlay as fallback, is open. It is a decision rather than a derivation: the gain is which network carries bytes, not what an attacker can reach, since the link is already encrypted against a pinned fingerprint.

1 — The overlay

What is decided: the peer graph. For every node: its overlay address, which peers it holds, which of those it may dial, and which must dial it.

Inputs, all declared:

• reachability — an endpoint, or none (ADR 0007). Not inferred from the address shape, which is wrong for carrier-grade NAT, wrong for IPv6, and wrong for a routable address behind a closed firewall.
• site — where the machine physically is, or nothing if it roams.
• role — hub or not, declared. Today it is inferred from an address prefix, which means a renumbering is an outage and nothing can be asked which node is the hub.

Keys. Each node generates its own keypair. The private key never leaves the machine; the public key is published to the mesh. This is already true and it is already right — it is ADR 0004's a node holds its own identity applied to the overlay, and it means the control plane computes a graph it cannot itself impersonate.

Shape: a hub, with direct peering between co-located nodes.

two nodes at the same site	peer directly, host-routed, with a keepalive
everything else	routes through the hub
a node with no site — it roams	hub only

Roaming is hub-only deliberately, and the reason is a property of WireGuard rather than a preference: there is no failover. A more specific route to a dead endpoint blackholes; it does not fall back to the general one. So a node whose location changes gets exactly one path, because two paths would mean one of them silently swallowing traffic.

What the host receives: an interface configuration and a peer list, as files. It does not compute them, and after this it holds no credential to the mesh's database.

Four things the lab found, none of them visible from the mesh's own state

Written 2026-08-29, on the first three machines to actually run this.

Each looked like a working network from every angle the mesh can see: the graph was right, the files were right, the services were up, and every node reported success.

• A running interface does not re-read its configuration. A node joins, every existing node's peer list changes, each file is replaced — and the service is already running, so nothing reloads it. Every existing node keeps a network that no longer exists. The declaration has to say the service must reflect the file, which is declared state; a command to restart would be an action, and the link may not carry one (ADR 0005).
• A hub that shares a site with a spoke was emitted twice — once as a direct peer and once as the route of last resort. WireGuard takes one entry per public key, so the interface refuses the file. The ordinary shape of a small mesh, and in none of the tests written before it ran.
• Two nodes at one site that neither can be dialled must not peer directly. Nobody opens the path, and the direct route is more specific than the hub's, so it wins and blackholes. This document's own warning, arriving in its implementation: a more specific route to a dead endpoint blackholes; it does not fall back to the general one.
• The container runtime closes the door the overlay needs. Docker sets the FORWARD policy to DROP, so a hub with ip_forward enabled still carries nothing between its spokes. The substrate at tier 1 silently breaks the network at tier 2, and nothing in either tier's state says so. The hub inserts its own rule above those chains and removes it on the way down.

The pattern in all four: the mesh's picture of the network was correct and the network did not work. That is the argument for the lab in one line — none of these is reachable by reasoning, and each was found within minutes of a real machine trying it.

2 — Resolution

Two name spaces, and they do not mix:

	resolves to	certified by
internal names	overlay addresses	the mesh CA
public names	whatever the outside world must reach	a public authority

A node's mesh name is its overlay address. Its public name, if it has one, is a separate fact used by things outside the mesh — and the separation carries two lessons that were learned expensively enough to be worth restating:

• Mesh names are not multicast names. A name resolved by local multicast discovery introduces a delay and a failure mode that appears on one node and not others — the worst shape a fault can have.
• A node must not pin its own public name locally. The duplicate record breaks resolution of that name for everything else that needs it.

What the host receives: the resolver's configuration, as files, listing every peer's internal name and overlay address.

What goes away: the /etc/hosts floor. It exists because a node had to reach the mesh database before its own DNS existed; with ADR 0004 nothing needs a name before the link, and a fallback nothing needs is a path nothing tests.

Names, and what a container can see

2026-08-31, from a container that could not resolve a name every machine could.

Internal names are <node>.internal — the suffix is the one IANA reserved in 2024, so a name that leaks into a public resolver fails rather than reaching a stranger's machine. They are computed centrally, because a name set needs every node at once, and written to each machine's hosts file.

A file rather than a resolver, and the reasoning holds: it works on every Linux, needs no package, and has no failure mode of its own. The stated trigger for a daemon was names that are not one-per-node — service names, wildcards.

But a container does not inherit the machine's names. It gets its own hosts file holding only its own hostname. So every name the mesh wrote was invisible to the majority of things that need one — and on the machine it always worked, which is exactly what made it easy to miss. It was found by a database client on one node failing to resolve another node, on a mesh where both names were correct and present on both machines.

So the mesh gives its names to the containers it declares, written into each container's own hosts file by the runtime. That extends the file decision rather than overturning it. Given by the mesh and not chosen by a module: a module that listed the machines would go stale the day one joins, and a module that did not would be one whose containers cannot reach anything by name.

The boundary, which is deliberate and worth stating: declared containers. A container somebody starts by hand is not the mesh's to configure, and reaching into every container on a machine — declared or not — is what a nameserver in resolv.conf would be for.

That is now the second reason to want a resolver, and it is a different one from the trigger above. Both remain unmet needs rather than plans:

names that are not one-per-node	postgres.internal meaning wherever the database is
containers the mesh did not declare	anything a person or another tool starts on a node

3 — Exposure

Settled by ADR 0007; summarised here because this is where it belongs.

A route is a grant. A module that must be reachable declares it needs one; the proxy provides it and hands back the public name. Ordinary ADR 0009 vocabulary — the mirror of a database grant, where the consumer supplies a target and receives a name rather than supplying nothing and receiving credentials.

A workload on an unreachable node is proxied by a reachable one, across the overlay. Which is the case is a mesh-level fact, which is the fourth reason exposure is control-plane work.

What was built

2026-08-31. Nothing new in the vocabulary, which was the claim and is now the fact: a route is a provision, a proxy provides it, and a module that must be reachable requires it. The consumer contributes the name it wants and the port it listens on; the proxy receives every consumer that asked; the consumer is told what the provider serves, which is how it knows its own name.

One field was missing, and it is the one anything reaching back needs. A contribution now carries where the mesh says that machine is. A database is reached by its consumer, so the mesh never had to tell a provider where anybody was; a proxy is the other direction — it is told to send traffic to a consumer and has to open a connection. Without it every provider implementing a provision would have to know how the mesh names machines, which is a convention leaking into every module.

Exposure and filtering are different questions and a module answers both. A workload says what it listens on and who may reach it; separately, it says it wants a route. A module that asked for a route and not for the port is unreachable by the proxy it just asked for — which the lab demonstrates, because the machine is already filtering by the time this runs.

Withdrawal, which was open above. The file the proxy is given is the whole truth about who has a route, so a proxy replaces its table rather than merging. Merging would keep serving a name whose module was unassigned — and a stale public name pointing at nothing fails more visibly than a stale grant is the reason it must not survive, not a reason to tolerate it.

A name a proxy does not serve is refused by saying which it does. A route that was withdrawn and a name that never existed are different things, and a bare 404 makes an operator go and read the mesh to tell them apart.

Checked in the lab by a request to the name reaching the workload across the private network and returning the workload's own answer, then by unassigning the module and requiring the same request to stop working.

4 — Filtering

Derived from what is assigned here, and from the overlay's shape — a node's open ports are a consequence of what runs on it and who must reach it, not an independent declaration to keep in step by hand.

A rule names its source (ADR 0007). A rule with no source is open, and must say so rather than appear to restrict something. scope: is removed rather than implemented: five manifests carry it today, it is referenced by no code, and it is the clearest instance in the repository of an unenforced rule is indistinguishable from a wrong one, and costs more, because people believe it.

Unknown keys are refused — the discipline the host's declaration parser already has (ADR 0005), and the one manifests lack. scope: survived because nothing rejected it.

What was built

2026-08-31. Everything above was the intention; this is what exists, and how each part is checked. 04-ISSUES/003 is resolved by it.

A module says what it listens on, as a port, a protocol and a source — mesh, anywhere, or machine. The source is required and there is no default, which is the whole of a rule names its source: a manifest that omitted it would read as a restriction and be none. Checked by a manifest with a port and no source being refused, and by one naming a source the mesh cannot render being refused as well — the second is what stops a source becoming a comment.

The set is derived per node, from every module assigned to it, not from the module asking for it. Where two modules want the same port, the wider source wins and both are still named, because removing one of them must not read as a reason to close a port the other needs. Checked by rendering a node whose firewall module has no ports of its own and asserting another module's port is in the result; and by giving one port two modules and one source each, and asserting the narrower rule disappears while both names survive.

What is not declared is closed. The rule set drops by default. Checked by naming the input chain in the assertion rather than the policy alone — the first version of that test passed while input accepted everything, because another chain in the same file also said policy drop.

From the mesh means the machines the mesh has, as their addresses on the private network, not as a subnet. A subnet is a guess that stays wrong quietly; the address set shrinks when a node leaves and nobody edits anything. A machine that asks for mesh where the mesh knows no addresses is closed and told so in the file — widening it would open a port nobody asked to open, and dropping it silently would close one somebody did.

Three things it deliberately does not do, each of which looked right and would have broken something:

	why not
decide what the machine forwards	the container runtime writes its own forwarding rules and a second policy is consulted alongside them, so a drop here stops every container on the node — the control plane included. Nothing in a manifest says what a machine routes, so there is nothing to derive it from either
flush the ruleset when loading	that empties every table on the machine, the runtime's among them. Only the mesh's own table is replaced, and it is declared empty first so the replacement works on a machine loading one for the first time
carry a command to load itself	the link may not carry an action (ADR 0005). A service is declared to reflect the file instead, so replacing it restarts what loads it — the shape that rule leaves, used here for the first time for its real purpose

A module that wants a rule set brings the unit that loads it. Found the hard way: the unit a distribution packages for nftables runs, applies the rules and exits, so it is neither running nor stopped — and a host asked for a service that is "running" reports, quite correctly, that it is stopped. Every packet was filtered exactly as declared and the machine was marked as not doing what it was told.

The vocabulary has no word for "ran, did its job, and exited", and that is a real gap rather than a wording problem: the whole class of configuration-applying units — packet filters, sysctl, tmpfiles — is shaped that way. Until there is one, a module ships a unit that stays, which is also the better shape: how a machine enforces rules is a fact about the machine, and the mesh has no business depending on what a distribution happens to package.

One thing is derived from what is assigned and not yet from the overlay's shape, and it is stated here rather than discovered: a hub's own listening port. A hub accepts inbound connections from every node at other sites; a node that is not a hub dials out and needs nothing open, because a reply to a flow it started is already accepted. So the two want different rules on an identical module — and listens is a static field on a manifest, while the overlay module's resources are computed per node.

A machine that is not a hub is therefore correct today, and a hub would have its own port closed by a rule set derived this way. The fix is that a computed module contributes listens the way it contributes resources; until it exists, the firewall belongs on machines that are not hubs, and this paragraph is the reason rather than an oversight to find later.

And it is enforced, which is what separates this from scope:. Checked on two real machines: two ports opened, one declared, and from the other machine the declared one answers and the undeclared one does not — then the module is removed and the port closes with nobody editing a rule. A rule set that is written but never loaded passes every check that reads the file, which is why the check reads packets.

5 — Certificates

Two authorities, kept separate on purpose.

	issued by	for
public names	a public ACME authority	anything outside the mesh reaches
internal names	the mesh CA	node-to-node, over the overlay

The split is not collapsed, including in the lab. A single-CA lab would hide any bug living in the split, so the lab runs its own ACME issuer on its public segment and keeps the mesh CA unchanged (research 004).

Public issuance requires genuine public reachability. The HTTP-01 challenge must be answered at the name being certified, so issuance happens through a publicly reachable node regardless of where the workload runs — the same asymmetry as exposure, for the same reason.

The issuer must be configurable. Today it is not: the proxy sets no caServer and therefore defaults to the public authority's production endpoint. Two consequences, and the second is worse than the lab problem that found it — every certificate experiment on a real node consumes production issuance quota, and a retry loop can exhaust it for a week.

The mesh CA is not a bootstrap concern. A joining node verifies the control plane against the fingerprint in its token (ADR 0004), so nothing needs the CA before membership. It certifies internal names afterwards, and that is all it does.

What was built

2026-08-31.

A node generates a fourth key, and the reason is the one the other three already give: a key used for two purposes is one rotation away from breaking the other. The identity key would work for TLS and reusing it would mean rotating a node's identity every time its certificate is replaced. The private half never leaves the machine; the mesh is told the public half at enrolment.

So there is no certificate request and nothing to seal. The mesh signs a statement binding a public key to a name it alone assigns, which is the whole of what a certificate authority does. It issues rather than stores: the node's key does not change, so signing again produces an equally valid certificate and there is nothing to keep in step.

A machine with no name inside the mesh is refused, not given a certificate for nothing. A certificate for a name nothing resolves is a certificate nothing can check.

And the key is stored in the format a server reads — PKCS#8 PEM, not the host's own encoding. That is not an implementation detail of whoever writes the file: the file exists because something else reads it, so the format is the interface (04-ISSUES/014).

Checked by a real handshake between two machines: one serves on its internal name with the key it generated, the other verifies against the mesh's authority and nothing else. Every cheaper check passed while the server could not start — the key was present, the certificate was valid, and nothing read either the way a server would.

What this removes

The list is worth having in one place, because it is most of the argument:

• The last two direct database connections from nodes — wireguard and traefik, the only two, both connectivity.
• Therefore the database credential on every node, and the object-store credential beside it. ADR 0004's central claim becomes true rather than aspirational.
• The /etc/hosts floor, and the bootstrap circularity it patched.
• Hub election by address prefix, and the silent no-hub failure when nobody knew the convention.
• The RFC1918 inference, and the lab substitution that existed to satisfy it.
• scope:, and the class of manifest key that means nothing.

Open

• What happens when the hub is down. Resolved by ADR 0006, together with 06's matching question — they were one question. Nothing takes over. WireGuard has no failover, the hub is declared rather than elected, and non-co-located paths stop while co-located direct peers and every already-assigned workload keep running. The recovery path is restore, and its deadline is certificate renewal.
• Renumbering the overlay. Made possible by declaring the hub rather than inferring it from an address, but no procedure exists, and a graph delivered node by node has an ordering problem while it is half-applied.
• Revoking a route when a module is unassigned. Resolved 2026-08-31 — see §3. The file a proxy is given is the whole truth about who has a route, so a route does not outlive the module that asked for it.
• IPv6. ADR 0007 makes it expressible; nothing here says the overlay or the resolver handle it.
• Reporting declared-versus-observed. ADR 0007 makes the disagreement detectable and does not say who looks or what they are told.