novox/hq
3
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Design connectivity in full: overlay, resolution, exposure, filtering, certificates

Browse Source 
Written as one document because the five are one design. They share inputs,
they must agree, and every one of them today is computed in a different place
by a different module from a different copy of the same facts.

The through-line is that none of the five can be answered by a machine alone,
so all five are decided centrally and delivered as `file` resources. That costs
no new host vocabulary and removes both remaining direct database connections
from nodes -- wireguard and traefik are the only two, and both are connectivity.

Three decisions fall out, all proposed:

0050 -- reachability is declared, not inferred from an address. The RFC1918
regex is wrong for carrier-grade NAT (100.64/10 tests as public, so an endpoint
is written to an address nothing can reach), wrong for IPv6, and wrong for a
routable address behind a closed firewall. The lab needing TEST-NET-3 to
satisfy the regex is the same bug from the other side. Also kills hub election
by address prefix, which fails silently and makes renumbering an outage.

0051 -- the enrolment token carries where the mesh is and how to recognise it.
Closes two circles with one mechanism: verifying the mesh needed the CA, and
obtaining the CA meant trusting whoever handed it over; and a node had to reach
the mesh before it could resolve any mesh name. An address plus a fingerprint,
carried out of band, resolves both -- and closes the CA question 0049 deferred.

0052 -- a filter rule names its source. `scope:` is declared in five manifests,
is part of no rule type, and is referenced by no code, so those manifests
appear to restrict ports and restrict nothing. Removed rather than implemented;
the general fix is refusing unknown keys, which the host already does and
manifests do not.

Also corrects two claims in 0049 asserting wireguard was already handled.
Research 006 says both modules still reach upward; neither is.
This commit is contained in:
jschoubben
2026-08-27 00:36:49 +02:00
parent 8d9282d86b
commit 4e80820e2f
7 changed files with 557 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02-DECISIONS/0049-a-route-is-a-grant.md
+10 -4
View File
@@ -95,8 +95,9 @@ shapes built. The proxy becomes a `container` with `file` configuration, which t
knows how to apply and read back.
And it removes a database credential from every node, which is half of what
[ADR 0039](0039-the-link-is-the-security-boundary.md) is for. The same move was already made for
`wireguard`; this applies it to the other offender.
[ADR 0039](0039-the-link-is-the-security-boundary.md) is for.
[ADR 0037](0037-the-host-applies-it-does-not-decide.md) decided this in principle; **neither
offender has actually been changed**, and this applies it to one of the two.
### A node without a public address is routed through one that has
@@ -117,8 +118,10 @@ behind NAT* — as the case the lab exists to get right.
- **Traefik's upward dependency is removed, and the module mostly disappears.** It computed its
own configuration; now it is an image plus files somebody else derived. Of the 426 lines
research 006 counted, what survives is a declaration.
- **One of two remaining database credentials leaves the node.** `wireguard` was the other, and
it is already handled — so this closes the set that ADR 0039 identified.
- **One of the two direct database connections goes.** `wireguard` is the other and is *not*
yet handled — [ADR 0050](0050-reachability-is-a-property-of-the-address.md) is what handles it,
and only both together close the set ADR 0039 identified. Until then a node still holds the
credential, so this record alone changes the design and not the exposure.
- **Certificate issuance becomes a control-plane responsibility with a real constraint**: an
ACME challenge can only be answered at a publicly reachable address, so issuance happens
through a public node regardless of where the workload runs. The lab already runs its own
@@ -130,6 +133,9 @@ behind NAT* — as the case the lab exists to get right.
mesh, which needs something a joining node can verify before it trusts anything. **Internal
identity and public exposure are two certificate stories and this record only closes the
second.** Conflating them is what made the gap hard to see.
**Since closed** by [ADR 0051](0051-the-enrolment-token-carries-the-mesh.md): the joining node
verifies the control plane against a fingerprint carried in its enrolment token, so nothing
needs the CA before membership.
- **Nothing says how a route is revoked** when a module is unassigned. The grant model implies
it — removing a consumer drops what it was granted
([ADR 0045](0045-a-context-owns-its-store.md)) — but a stale public name pointing at nothing is
02-DECISIONS/0050-reachability-is-a-property-of-the-address.md
+119
View File
@@ -0,0 +1,119 @@
---
status: proposed
date: 2026-08-27
deciders: jochen
reconstructed: false
extends: 0031-the-lab-provides-the-underlay.md
---
# 50. Reachability is declared, not inferred from an address
## Context
The overlay's peer graph is computed on each node by the `wireguard` module, which decides —
per pair — whether to write an `Endpoint` for a peer. The rule
([research 004](../01-RESEARCH/004-lab-network/analysis.md)) is a regular expression:
```js
const isPrivate = (a) => /^(10\.|127\.|192\.168\.|172\.(1[6-9]|2\d|3[01])\.)/.test(a);
```
Not private ⇒ assumed reachable ⇒ an `Endpoint` is written. Private ⇒ no endpoint, and the peer
must initiate.
The module's own comments record what this cost to arrive at: testing `profile === "server"`
was tried and was wrong, because a home-hosted node **is** a server and is not publicly
reachable — *"role does not imply reachability; the address does."*
**That lesson is right and the implementation of it is not.** The address is evidence of
reachability; it is not the fact itself, and the gap between the two has already caused failures
and will cause more:
| Address | The regex says | Actually |
|---|---|---|
| `100.64.0.0/10` — carrier-grade NAT ([RFC 6598](https://www.rfc-editor.org/rfc/rfc6598)) | **public** | **not reachable.** An endpoint is written to an address nothing can reach. |
| any IPv6 address, including `fd00::/8` unique-local | **public** | unique-local is not reachable; the regex tests v4 shapes only |
| a routable address behind a closed firewall | public | not reachable |
| `10.200.0.0/24` standing in for a public segment | private | reachable — this is the lab bug |
The CGNAT row is the serious one. A node on a carrier-grade NAT address presents exactly the
failure already recorded for the hairpin case: *1.77 MiB sent, 0 B received, no handshake.* The
mesh silently never forms, and it presents as a WireGuard fault rather than an addressing one.
The lab row is the same bug seen from the other side. [Research 004](../01-RESEARCH/004-lab-network/00-overview.md)
calls the required substitution *"the single most important fact in this document"* — a
simulated public segment must use TEST-NET-3, or nothing can ever initiate. **A test environment
having to choose its addresses to satisfy a regex is the regex telling us it is not a fact.**
There is a second inference in the same code, and it is worse because nothing records it:
> **Hub election is by convention.** The hub is the node whose `profile='server'` *and* whose
> overlay address begins `10.10.0.1`. A lab must assign that address to the node it intends as
> hub **or there will be no hub** — and nothing says so.
An election decided by the first four characters of an address is not an election. It fails
silently, it cannot be queried, and it makes a renumbering into an outage.
## Considered options
1. **Fix the regex.** Add CGNAT, add IPv6, add the ranges as they are discovered. Rejected: the
list is unbounded, and each addition is written after the outage that revealed it. The
firewall case cannot be fixed at all — no address shape encodes it.
2. **Probe for reachability and cache the answer.** Attractive, and wrong as the *primary*
source: at the moment the graph is computed a node may be legitimately down, and a probe
cannot distinguish *unreachable* from *asleep* ([ADR 0036](0036-a-node-is-a-managed-machine.md)).
Deriving topology from a liveness check makes the overlay flap with the network.
3. **Declare it, and let observation contradict it.** Chosen.
## Decision
**A node's reachability is a declared fact on its record, not an inference from its address.**
Two facts, and the mesh stores both:
- **an endpoint, or none** — where peers may reach this node, if anywhere. Absent means *this
node initiates and is never dialled*, which is the safe default and the common case.
- **its role in the overlay** — whether it is a hub. **Declared, never derived from an address.**
**The address remains evidence and stops being the fact.** When a node's observed endpoint
disagrees with its declared one, that is a **reportable condition**, not a silent correction —
the same discipline as [ADR 0035](0035-a-picture-is-read-from-what-runs.md): a picture is read
from the system, and where the reading disagrees with the intent, the disagreement is the
finding.
**The peer graph is computed by the control plane**, from these declared facts, and delivered to
each node as configuration. It is not computed on the node, which is
[ADR 0037](0037-the-host-applies-it-does-not-decide.md) and is what removes `wireguard`'s direct
database connection ([ADR 0049](0049-a-route-is-a-grant.md) does the same for the proxy).
**What does not change** is the rule the comment was defending. Role still does not imply
reachability — a home-hosted node is still a server that cannot be dialled. This record keeps
that lesson and stops encoding it as a pattern match.
## Consequences
- **CGNAT and IPv6 nodes become expressible**, which today they are not. Neither needs a code
change to support; they need a field that says what is true.
- **The lab stops needing its substitution.** TEST-NET-3 remains the right choice for a
documentation range, but the scenario now says *this segment is reachable* rather than relying
on an address shape to imply it. The constraint research 004 calls its most important fact
becomes an ordinary declaration, and the validator's enforcement of it becomes unnecessary
rather than load-bearing.
- **Hub election becomes queryable and renumbering becomes safe.** Both follow from the same
change and neither is possible today.
- **Two facts can now disagree, and something must say so.** Declared-versus-observed is a new
reportable condition and a new way to be wrong — a node declared reachable that is not will
fail exactly as it does today until somebody looks. The gain is that *looking* is now possible;
the regex offered nothing to compare against.
- **Somebody must set these when a node joins.** [ADR 0038](0038-a-node-joins-by-linking-first.md)
has the mesh finish the job after the link, so this is one more thing it finishes — and the
honest default (no endpoint, not a hub) is correct for every node except the ones somebody
deliberately publishes.
## References
- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) — the regex, the hub convention, and
the hairpin failure.
- [ADR 0037](0037-the-host-applies-it-does-not-decide.md) — why the graph is computed centrally.
- [ADR 0035](0035-a-picture-is-read-from-what-runs.md) — declared versus observed.
- [`08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md) — the design this serves.
02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md
+117
View File
@@ -0,0 +1,117 @@
---
status: proposed
date: 2026-08-27
deciders: jochen
reconstructed: false
extends: 0039-the-link-is-the-security-boundary.md
---
# 51. The enrolment token carries where the mesh is and how to recognise it
## Context
[ADR 0039](0039-the-link-is-the-security-boundary.md) requires **mutual** authority: the node
proves it may join, and **the control plane proves it is the mesh**. It states why one-way is not
enough — the host applies whatever the link delivers, so *an attacker who can answer a joining
node's first call owns the machine.*
It does not say **how** the control plane proves itself, and
[ADR 0049](0049-a-route-is-a-grant.md) deferred the question again, noting only that internal
identity and public exposure are two different certificate stories.
Left unanswered it produces a circle. Verifying the mesh needs the mesh's CA. Obtaining the CA
means trusting whatever hands it over — which is the thing being verified.
There is a second circle in the same place, and today they are solved by the same unfortunate
mechanism. A node must reach the mesh before the mesh has configured it, so it cannot yet
resolve any mesh name. [Research 004](../01-RESEARCH/004-lab-network/analysis.md) records the
workaround:
> `dnsmasq-app` generates `.internal` names on each node and writes an `/etc/hosts` block **as a
> floor underneath, because a node must reach the mesh DB before its own DNS exists.**
and, separately, that a joining node must have *"registry database and object-store host and
credentials, plus an npm token"* placed on disk beforehand — the credentials
[ADR 0039](0039-the-link-is-the-security-boundary.md) exists to remove.
**Both circles are the same shape: a node needs some fact about the mesh before it has any
trustworthy way to obtain one.** Whatever supplies that fact must arrive by a path other than the
mesh.
## Considered options
1. **Ship the CA with the host binary.** Then the binary is mesh-specific, which
[ADR 0041](0041-the-host-depends-on-nothing.md) and
[ADR 0046](0046-the-installer-fetches-what-it-pins.md) both work to avoid, and rotating the
CA means rebuilding and redistributing the host everywhere. Rejected.
2. **Trust on first use, plainly.** Accept whatever answers the first call and pin it. Rejected:
it is exactly the attack ADR 0039 names, and the first call is the one moment the node has no
way to tell.
3. **A public certificate authority for the control plane's own endpoint.** Workable, and it
makes joining depend on public DNS and public issuance for a link that is otherwise entirely
the mesh's business. Rejected as a dependency, not as a technique — a mesh whose nodes cannot
join because an unrelated public authority is having a bad day has bought nothing.
4. **The token carries it.** Chosen.
## Decision
**The enrolment token carries three things**, and it is the only thing a joining node needs:
| | | |
|---|---|---|
| **where** | an **address**, not a name | there is no resolution yet, and this is why none is needed |
| **who** | the fingerprint of the identity to expect | what makes the mesh provable rather than assumed |
| **the right to join** | the one-time secret ADR 0039 already specifies | useless once used, useless after it expires |
The token is issued by the mesh for one enrolment and **carried out of band** — by the person
adopting the machine. That is what breaks both circles: its authenticity comes from the channel
it travelled, not from anything the node can check afterwards.
**This is trust-on-first-use with the first use moved out of band**, which is the difference
between a pin and a guess. The node does not accept whatever answers; it accepts the one thing
it was told to expect, before it spoke to anything.
### What this settles
**The mesh CA is not a bootstrap concern.** It is how `.internal` names are certified once a node
is a member, and nothing needs it earlier. The open item
[ADR 0049](0049-a-route-is-a-grant.md) left — *what the control plane presents to a node that
trusts nothing yet* — is closed: it presents the identity whose fingerprint the token named.
**Nothing needs name resolution before the link exists**, because the token carries an address.
The `/etc/hosts` floor exists to solve a problem that stops existing, and it should go rather than
be carried forward — a fallback nothing needs is a path nothing tests.
**Nothing is placed on disk beforehand except the token.** No database credential, no object-store
credential, no registry token. That is ADR 0039's central claim finally made true at the one
moment it was still false, and it is the difference between *a node holds only its own identity*
being a design statement and being a fact.
## Consequences
- **The token becomes security-critical in a way it was not**, because it now carries the pin.
Tampering with it in transit substitutes the mesh. That is a real exposure and it is strictly
better than the alternative: without a pin there is nothing to tamper *with*, and the node
trusts the first answer unconditionally. The exposure moves to a channel a person controls and
can verify, from one nobody could.
- **Token delivery is now a designed step, not an incidental one.** It is short-lived and
single-use, so interception is bounded — but how it reaches a machine is part of adoption and
needs saying. [Research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md) is
where that belongs.
- **Rotating the control plane's identity invalidates outstanding tokens**, which is correct and
needs to fail legibly. A node presenting a token with a stale fingerprint must be told that,
not left to time out.
- **The address in the token can go stale.** If the control plane moves, unissued tokens point
somewhere wrong. Tokens are short-lived, which bounds it; moving the control plane is
[`06`](../03-DESIGN/01-to-be/06-the-control-plane.md)'s undesigned territory regardless.
- **A rejoining node is an ordinary case, not a special one.** A node that has lost its identity
gets a new token. There is no recovery path to design because there is no long-lived secret to
recover.
## References
- [ADR 0039](0039-the-link-is-the-security-boundary.md) — the mutual authority this implements.
- [ADR 0038](0038-a-node-joins-by-linking-first.md) — the join this is the first step of.
- [ADR 0049](0049-a-route-is-a-grant.md) — the open item this closes.
- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) — the `/etc/hosts` floor and the
credentials placed beforehand.
02-DECISIONS/0052-a-filter-rule-names-its-source.md
+71
View File
@@ -0,0 +1,71 @@
---
status: proposed
date: 2026-08-27
deciders: jochen
reconstructed: false
extends: 0043-a-declaration-is-an-ordered-list-of-owned-resources.md
---
# 52. A filter rule names its source, or it is not a rule
## Context
[Research 004](../01-RESEARCH/004-lab-network/analysis.md) found this while looking for something
else:
> Several manifests declare `scope: public` on firewall rules — `wireguard`, `traefik`, `gitea`,
> `mailu`, `qbittorrent`. It is **not part of the rule type** and is **referenced by no code** in
> the firewall path. Real scoping is done with `from:`.
**So five manifests appear to restrict a port and restrict nothing.** Anyone reading them —
including whoever wrote the next one by copying — sees an access control that does not exist.
Research 004 already names the shape: it is `how-we-build`'s *an unenforced rule is
indistinguishable from a wrong one, and costs more, because people believe it.* This is that
rule, in the firewall, on the modules most worth restricting.
The mechanism that let it happen is worth more than the instance. `scope:` was accepted because
**unknown keys were ignored**. Nothing rejected it, nothing warned, and it spread by copying for
long enough to reach five manifests.
## Decision
**A rule names its source.** `from:` is the only way to scope a rule, and a rule without one is
open — which it must therefore say plainly rather than imply otherwise.
**`scope:` is removed, not implemented.** Giving it meaning would leave two ways to express one
thing, and a manifest carrying both would need a precedence rule nobody would remember. The five
manifests are corrected to `from:` where they meant to restrict something, and left open where
they did not — and finding out which is which is part of the work, not a formality.
**An unknown key is refused.** This is the general fix and the reason to bother:
> A manifest carrying a key the schema does not define is **rejected**, naming the key.
The host already works this way — its declaration parser sets `DisallowUnknownFields` and
collects every problem into one refusal
([ADR 0043](0043-a-declaration-is-an-ordered-list-of-owned-resources.md)). **Manifests are the
layer where that discipline is missing**, and `scope:` is what missing looks like: not a wrong
value, an invented one, silently accepted for months.
## Consequences
- **This class of fiction stops at validation** rather than at an audit. A misspelled `form:`,
an invented `scope:`, a key from a different schema — each fails on the manifest that
introduces it, once, instead of spreading.
- **Existing manifests will fail validation**, and some of those failures will be keys somebody
believed were doing something. That is the finding, not the cost — but it means the refusal
cannot be switched on without reading every manifest first.
- **The firewall becomes reviewable.** Today a rule's real effect is only visible by knowing
which keys are fictional. Afterwards the manifest says what happens.
- **Five manifests need a decision each**, and `wireguard` and `traefik` genuinely are open to
the world — they must be, which the manifest should state rather than appear to deny.
- **It does not make the rules correct**, only honest. A rule that says `from:` anywhere is
legitimately open; this record ensures it says so.
## References
- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) §7 — the finding.
- [`how-we-build.md`](../00-META/how-we-build.md) — the rule about unenforced rules.
- [ADR 0043](0043-a-declaration-is-an-ordered-list-of-owned-resources.md) — unknown-is-refused,
where it already holds.