novox/hq

Files

T

jschoubben 4e80820e2f Design connectivity in full: overlay, resolution, exposure, filtering, certificates

Written as one document because the five are one design. They share inputs,
they must agree, and every one of them today is computed in a different place
by a different module from a different copy of the same facts.

The through-line is that none of the five can be answered by a machine alone,
so all five are decided centrally and delivered as `file` resources. That costs
no new host vocabulary and removes both remaining direct database connections
from nodes -- wireguard and traefik are the only two, and both are connectivity.

Three decisions fall out, all proposed:

0050 -- reachability is declared, not inferred from an address. The RFC1918
regex is wrong for carrier-grade NAT (100.64/10 tests as public, so an endpoint
is written to an address nothing can reach), wrong for IPv6, and wrong for a
routable address behind a closed firewall. The lab needing TEST-NET-3 to
satisfy the regex is the same bug from the other side. Also kills hub election
by address prefix, which fails silently and makes renumbering an outage.

0051 -- the enrolment token carries where the mesh is and how to recognise it.
Closes two circles with one mechanism: verifying the mesh needed the CA, and
obtaining the CA meant trusting whoever handed it over; and a node had to reach
the mesh before it could resolve any mesh name. An address plus a fingerprint,
carried out of band, resolves both -- and closes the CA question 0049 deferred.

0052 -- a filter rule names its source. `scope:` is declared in five manifests,
is part of no rule type, and is referenced by no code, so those manifests
appear to restrict ports and restrict nothing. Removed rather than implemented;
the general fix is refusing unknown keys, which the host already does and
manifests do not.

Also corrects two claims in 0049 asserting wireguard was already handled.
Research 006 says both modules still reach upward; neither is.

2026-08-27 00:36:49 +02:00

6.4 KiB

Raw Blame History

status, date, deciders, reconstructed, extends

status	date	deciders	reconstructed	extends
proposed	2026-08-27	jochen	false	0039-the-link-is-the-security-boundary.md

51. The enrolment token carries where the mesh is and how to recognise it

Context

ADR 0039 requires mutual authority: the node proves it may join, and the control plane proves it is the mesh. It states why one-way is not enough — the host applies whatever the link delivers, so an attacker who can answer a joining node's first call owns the machine.

It does not say how the control plane proves itself, and ADR 0049 deferred the question again, noting only that internal identity and public exposure are two different certificate stories.

Left unanswered it produces a circle. Verifying the mesh needs the mesh's CA. Obtaining the CA means trusting whatever hands it over — which is the thing being verified.

There is a second circle in the same place, and today they are solved by the same unfortunate mechanism. A node must reach the mesh before the mesh has configured it, so it cannot yet resolve any mesh name. Research 004 records the workaround:

dnsmasq-app generates .internal names on each node and writes an /etc/hosts block as a floor underneath, because a node must reach the mesh DB before its own DNS exists.

and, separately, that a joining node must have "registry database and object-store host and credentials, plus an npm token" placed on disk beforehand — the credentials ADR 0039 exists to remove.

Both circles are the same shape: a node needs some fact about the mesh before it has any trustworthy way to obtain one. Whatever supplies that fact must arrive by a path other than the mesh.

Considered options

Ship the CA with the host binary. Then the binary is mesh-specific, which ADR 0041 and ADR 0046 both work to avoid, and rotating the CA means rebuilding and redistributing the host everywhere. Rejected.
Trust on first use, plainly. Accept whatever answers the first call and pin it. Rejected: it is exactly the attack ADR 0039 names, and the first call is the one moment the node has no way to tell.
A public certificate authority for the control plane's own endpoint. Workable, and it makes joining depend on public DNS and public issuance for a link that is otherwise entirely the mesh's business. Rejected as a dependency, not as a technique — a mesh whose nodes cannot join because an unrelated public authority is having a bad day has bought nothing.
The token carries it. Chosen.

Decision

The enrolment token carries three things, and it is the only thing a joining node needs:


where	an address, not a name	there is no resolution yet, and this is why none is needed
who	the fingerprint of the identity to expect	what makes the mesh provable rather than assumed
the right to join	the one-time secret ADR 0039 already specifies	useless once used, useless after it expires

The token is issued by the mesh for one enrolment and carried out of band — by the person adopting the machine. That is what breaks both circles: its authenticity comes from the channel it travelled, not from anything the node can check afterwards.

This is trust-on-first-use with the first use moved out of band, which is the difference between a pin and a guess. The node does not accept whatever answers; it accepts the one thing it was told to expect, before it spoke to anything.

What this settles

The mesh CA is not a bootstrap concern. It is how .internal names are certified once a node is a member, and nothing needs it earlier. The open item ADR 0049 left — what the control plane presents to a node that trusts nothing yet — is closed: it presents the identity whose fingerprint the token named.

Nothing needs name resolution before the link exists, because the token carries an address. The /etc/hosts floor exists to solve a problem that stops existing, and it should go rather than be carried forward — a fallback nothing needs is a path nothing tests.

Nothing is placed on disk beforehand except the token. No database credential, no object-store credential, no registry token. That is ADR 0039's central claim finally made true at the one moment it was still false, and it is the difference between a node holds only its own identity being a design statement and being a fact.

Consequences

The token becomes security-critical in a way it was not, because it now carries the pin. Tampering with it in transit substitutes the mesh. That is a real exposure and it is strictly better than the alternative: without a pin there is nothing to tamper with, and the node trusts the first answer unconditionally. The exposure moves to a channel a person controls and can verify, from one nobody could.
Token delivery is now a designed step, not an incidental one. It is short-lived and single-use, so interception is bounded — but how it reaches a machine is part of adoption and needs saying. Research 012 is where that belongs.
Rotating the control plane's identity invalidates outstanding tokens, which is correct and needs to fail legibly. A node presenting a token with a stale fingerprint must be told that, not left to time out.
The address in the token can go stale. If the control plane moves, unissued tokens point somewhere wrong. Tokens are short-lived, which bounds it; moving the control plane is 06's undesigned territory regardless.
A rejoining node is an ordinary case, not a special one. A node that has lost its identity gets a new token. There is no recovery path to design because there is no long-lived secret to recover.

References

ADR 0039 — the mutual authority this implements.
ADR 0038 — the join this is the first step of.
ADR 0049 — the open item this closes.
Research 004 — the /etc/hosts floor and the credentials placed beforehand.

6.4 KiB Raw Blame History