Design connectivity in full: overlay, resolution, exposure, filtering, certificates
Written as one document because the five are one design. They share inputs, they must agree, and every one of them today is computed in a different place by a different module from a different copy of the same facts. The through-line is that none of the five can be answered by a machine alone, so all five are decided centrally and delivered as `file` resources. That costs no new host vocabulary and removes both remaining direct database connections from nodes -- wireguard and traefik are the only two, and both are connectivity. Three decisions fall out, all proposed: 0050 -- reachability is declared, not inferred from an address. The RFC1918 regex is wrong for carrier-grade NAT (100.64/10 tests as public, so an endpoint is written to an address nothing can reach), wrong for IPv6, and wrong for a routable address behind a closed firewall. The lab needing TEST-NET-3 to satisfy the regex is the same bug from the other side. Also kills hub election by address prefix, which fails silently and makes renumbering an outage. 0051 -- the enrolment token carries where the mesh is and how to recognise it. Closes two circles with one mechanism: verifying the mesh needed the CA, and obtaining the CA meant trusting whoever handed it over; and a node had to reach the mesh before it could resolve any mesh name. An address plus a fingerprint, carried out of band, resolves both -- and closes the CA question 0049 deferred. 0052 -- a filter rule names its source. `scope:` is declared in five manifests, is part of no rule type, and is referenced by no code, so those manifests appear to restrict ports and restrict nothing. Removed rather than implemented; the general fix is refusing unknown keys, which the host already does and manifests do not. Also corrects two claims in 0049 asserting wireguard was already handled. Research 006 says both modules still reach upward; neither is.
This commit is contained in:
@@ -95,8 +95,9 @@ shapes built. The proxy becomes a `container` with `file` configuration, which t
|
||||
knows how to apply and read back.
|
||||
|
||||
And it removes a database credential from every node, which is half of what
|
||||
[ADR 0039](0039-the-link-is-the-security-boundary.md) is for. The same move was already made for
|
||||
`wireguard`; this applies it to the other offender.
|
||||
[ADR 0039](0039-the-link-is-the-security-boundary.md) is for.
|
||||
[ADR 0037](0037-the-host-applies-it-does-not-decide.md) decided this in principle; **neither
|
||||
offender has actually been changed**, and this applies it to one of the two.
|
||||
|
||||
### A node without a public address is routed through one that has
|
||||
|
||||
@@ -117,8 +118,10 @@ behind NAT* — as the case the lab exists to get right.
|
||||
- **Traefik's upward dependency is removed, and the module mostly disappears.** It computed its
|
||||
own configuration; now it is an image plus files somebody else derived. Of the 426 lines
|
||||
research 006 counted, what survives is a declaration.
|
||||
- **One of two remaining database credentials leaves the node.** `wireguard` was the other, and
|
||||
it is already handled — so this closes the set that ADR 0039 identified.
|
||||
- **One of the two direct database connections goes.** `wireguard` is the other and is *not*
|
||||
yet handled — [ADR 0050](0050-reachability-is-a-property-of-the-address.md) is what handles it,
|
||||
and only both together close the set ADR 0039 identified. Until then a node still holds the
|
||||
credential, so this record alone changes the design and not the exposure.
|
||||
- **Certificate issuance becomes a control-plane responsibility with a real constraint**: an
|
||||
ACME challenge can only be answered at a publicly reachable address, so issuance happens
|
||||
through a public node regardless of where the workload runs. The lab already runs its own
|
||||
@@ -130,6 +133,9 @@ behind NAT* — as the case the lab exists to get right.
|
||||
mesh, which needs something a joining node can verify before it trusts anything. **Internal
|
||||
identity and public exposure are two certificate stories and this record only closes the
|
||||
second.** Conflating them is what made the gap hard to see.
|
||||
**Since closed** by [ADR 0051](0051-the-enrolment-token-carries-the-mesh.md): the joining node
|
||||
verifies the control plane against a fingerprint carried in its enrolment token, so nothing
|
||||
needs the CA before membership.
|
||||
- **Nothing says how a route is revoked** when a module is unassigned. The grant model implies
|
||||
it — removing a consumer drops what it was granted
|
||||
([ADR 0045](0045-a-context-owns-its-store.md)) — but a stale public name pointing at nothing is
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
status: proposed
|
||||
date: 2026-08-27
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0031-the-lab-provides-the-underlay.md
|
||||
---
|
||||
|
||||
# 50. Reachability is declared, not inferred from an address
|
||||
|
||||
## Context
|
||||
|
||||
The overlay's peer graph is computed on each node by the `wireguard` module, which decides —
|
||||
per pair — whether to write an `Endpoint` for a peer. The rule
|
||||
([research 004](../01-RESEARCH/004-lab-network/analysis.md)) is a regular expression:
|
||||
|
||||
```js
|
||||
const isPrivate = (a) => /^(10\.|127\.|192\.168\.|172\.(1[6-9]|2\d|3[01])\.)/.test(a);
|
||||
```
|
||||
|
||||
Not private ⇒ assumed reachable ⇒ an `Endpoint` is written. Private ⇒ no endpoint, and the peer
|
||||
must initiate.
|
||||
|
||||
The module's own comments record what this cost to arrive at: testing `profile === "server"`
|
||||
was tried and was wrong, because a home-hosted node **is** a server and is not publicly
|
||||
reachable — *"role does not imply reachability; the address does."*
|
||||
|
||||
**That lesson is right and the implementation of it is not.** The address is evidence of
|
||||
reachability; it is not the fact itself, and the gap between the two has already caused failures
|
||||
and will cause more:
|
||||
|
||||
| Address | The regex says | Actually |
|
||||
|---|---|---|
|
||||
| `100.64.0.0/10` — carrier-grade NAT ([RFC 6598](https://www.rfc-editor.org/rfc/rfc6598)) | **public** | **not reachable.** An endpoint is written to an address nothing can reach. |
|
||||
| any IPv6 address, including `fd00::/8` unique-local | **public** | unique-local is not reachable; the regex tests v4 shapes only |
|
||||
| a routable address behind a closed firewall | public | not reachable |
|
||||
| `10.200.0.0/24` standing in for a public segment | private | reachable — this is the lab bug |
|
||||
|
||||
The CGNAT row is the serious one. A node on a carrier-grade NAT address presents exactly the
|
||||
failure already recorded for the hairpin case: *1.77 MiB sent, 0 B received, no handshake.* The
|
||||
mesh silently never forms, and it presents as a WireGuard fault rather than an addressing one.
|
||||
|
||||
The lab row is the same bug seen from the other side. [Research 004](../01-RESEARCH/004-lab-network/00-overview.md)
|
||||
calls the required substitution *"the single most important fact in this document"* — a
|
||||
simulated public segment must use TEST-NET-3, or nothing can ever initiate. **A test environment
|
||||
having to choose its addresses to satisfy a regex is the regex telling us it is not a fact.**
|
||||
|
||||
There is a second inference in the same code, and it is worse because nothing records it:
|
||||
|
||||
> **Hub election is by convention.** The hub is the node whose `profile='server'` *and* whose
|
||||
> overlay address begins `10.10.0.1`. A lab must assign that address to the node it intends as
|
||||
> hub **or there will be no hub** — and nothing says so.
|
||||
|
||||
An election decided by the first four characters of an address is not an election. It fails
|
||||
silently, it cannot be queried, and it makes a renumbering into an outage.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Fix the regex.** Add CGNAT, add IPv6, add the ranges as they are discovered. Rejected: the
|
||||
list is unbounded, and each addition is written after the outage that revealed it. The
|
||||
firewall case cannot be fixed at all — no address shape encodes it.
|
||||
2. **Probe for reachability and cache the answer.** Attractive, and wrong as the *primary*
|
||||
source: at the moment the graph is computed a node may be legitimately down, and a probe
|
||||
cannot distinguish *unreachable* from *asleep* ([ADR 0036](0036-a-node-is-a-managed-machine.md)).
|
||||
Deriving topology from a liveness check makes the overlay flap with the network.
|
||||
3. **Declare it, and let observation contradict it.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**A node's reachability is a declared fact on its record, not an inference from its address.**
|
||||
|
||||
Two facts, and the mesh stores both:
|
||||
|
||||
- **an endpoint, or none** — where peers may reach this node, if anywhere. Absent means *this
|
||||
node initiates and is never dialled*, which is the safe default and the common case.
|
||||
- **its role in the overlay** — whether it is a hub. **Declared, never derived from an address.**
|
||||
|
||||
**The address remains evidence and stops being the fact.** When a node's observed endpoint
|
||||
disagrees with its declared one, that is a **reportable condition**, not a silent correction —
|
||||
the same discipline as [ADR 0035](0035-a-picture-is-read-from-what-runs.md): a picture is read
|
||||
from the system, and where the reading disagrees with the intent, the disagreement is the
|
||||
finding.
|
||||
|
||||
**The peer graph is computed by the control plane**, from these declared facts, and delivered to
|
||||
each node as configuration. It is not computed on the node, which is
|
||||
[ADR 0037](0037-the-host-applies-it-does-not-decide.md) and is what removes `wireguard`'s direct
|
||||
database connection ([ADR 0049](0049-a-route-is-a-grant.md) does the same for the proxy).
|
||||
|
||||
**What does not change** is the rule the comment was defending. Role still does not imply
|
||||
reachability — a home-hosted node is still a server that cannot be dialled. This record keeps
|
||||
that lesson and stops encoding it as a pattern match.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **CGNAT and IPv6 nodes become expressible**, which today they are not. Neither needs a code
|
||||
change to support; they need a field that says what is true.
|
||||
- **The lab stops needing its substitution.** TEST-NET-3 remains the right choice for a
|
||||
documentation range, but the scenario now says *this segment is reachable* rather than relying
|
||||
on an address shape to imply it. The constraint research 004 calls its most important fact
|
||||
becomes an ordinary declaration, and the validator's enforcement of it becomes unnecessary
|
||||
rather than load-bearing.
|
||||
- **Hub election becomes queryable and renumbering becomes safe.** Both follow from the same
|
||||
change and neither is possible today.
|
||||
- **Two facts can now disagree, and something must say so.** Declared-versus-observed is a new
|
||||
reportable condition and a new way to be wrong — a node declared reachable that is not will
|
||||
fail exactly as it does today until somebody looks. The gain is that *looking* is now possible;
|
||||
the regex offered nothing to compare against.
|
||||
- **Somebody must set these when a node joins.** [ADR 0038](0038-a-node-joins-by-linking-first.md)
|
||||
has the mesh finish the job after the link, so this is one more thing it finishes — and the
|
||||
honest default (no endpoint, not a hub) is correct for every node except the ones somebody
|
||||
deliberately publishes.
|
||||
|
||||
## References
|
||||
|
||||
- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) — the regex, the hub convention, and
|
||||
the hairpin failure.
|
||||
- [ADR 0037](0037-the-host-applies-it-does-not-decide.md) — why the graph is computed centrally.
|
||||
- [ADR 0035](0035-a-picture-is-read-from-what-runs.md) — declared versus observed.
|
||||
- [`08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md) — the design this serves.
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
status: proposed
|
||||
date: 2026-08-27
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0039-the-link-is-the-security-boundary.md
|
||||
---
|
||||
|
||||
# 51. The enrolment token carries where the mesh is and how to recognise it
|
||||
|
||||
## Context
|
||||
|
||||
[ADR 0039](0039-the-link-is-the-security-boundary.md) requires **mutual** authority: the node
|
||||
proves it may join, and **the control plane proves it is the mesh**. It states why one-way is not
|
||||
enough — the host applies whatever the link delivers, so *an attacker who can answer a joining
|
||||
node's first call owns the machine.*
|
||||
|
||||
It does not say **how** the control plane proves itself, and
|
||||
[ADR 0049](0049-a-route-is-a-grant.md) deferred the question again, noting only that internal
|
||||
identity and public exposure are two different certificate stories.
|
||||
|
||||
Left unanswered it produces a circle. Verifying the mesh needs the mesh's CA. Obtaining the CA
|
||||
means trusting whatever hands it over — which is the thing being verified.
|
||||
|
||||
There is a second circle in the same place, and today they are solved by the same unfortunate
|
||||
mechanism. A node must reach the mesh before the mesh has configured it, so it cannot yet
|
||||
resolve any mesh name. [Research 004](../01-RESEARCH/004-lab-network/analysis.md) records the
|
||||
workaround:
|
||||
|
||||
> `dnsmasq-app` generates `.internal` names on each node and writes an `/etc/hosts` block **as a
|
||||
> floor underneath, because a node must reach the mesh DB before its own DNS exists.**
|
||||
|
||||
and, separately, that a joining node must have *"registry database and object-store host and
|
||||
credentials, plus an npm token"* placed on disk beforehand — the credentials
|
||||
[ADR 0039](0039-the-link-is-the-security-boundary.md) exists to remove.
|
||||
|
||||
**Both circles are the same shape: a node needs some fact about the mesh before it has any
|
||||
trustworthy way to obtain one.** Whatever supplies that fact must arrive by a path other than the
|
||||
mesh.
|
||||
|
||||
## Considered options
|
||||
|
||||
1. **Ship the CA with the host binary.** Then the binary is mesh-specific, which
|
||||
[ADR 0041](0041-the-host-depends-on-nothing.md) and
|
||||
[ADR 0046](0046-the-installer-fetches-what-it-pins.md) both work to avoid, and rotating the
|
||||
CA means rebuilding and redistributing the host everywhere. Rejected.
|
||||
2. **Trust on first use, plainly.** Accept whatever answers the first call and pin it. Rejected:
|
||||
it is exactly the attack ADR 0039 names, and the first call is the one moment the node has no
|
||||
way to tell.
|
||||
3. **A public certificate authority for the control plane's own endpoint.** Workable, and it
|
||||
makes joining depend on public DNS and public issuance for a link that is otherwise entirely
|
||||
the mesh's business. Rejected as a dependency, not as a technique — a mesh whose nodes cannot
|
||||
join because an unrelated public authority is having a bad day has bought nothing.
|
||||
4. **The token carries it.** Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**The enrolment token carries three things**, and it is the only thing a joining node needs:
|
||||
|
||||
| | | |
|
||||
|---|---|---|
|
||||
| **where** | an **address**, not a name | there is no resolution yet, and this is why none is needed |
|
||||
| **who** | the fingerprint of the identity to expect | what makes the mesh provable rather than assumed |
|
||||
| **the right to join** | the one-time secret ADR 0039 already specifies | useless once used, useless after it expires |
|
||||
|
||||
The token is issued by the mesh for one enrolment and **carried out of band** — by the person
|
||||
adopting the machine. That is what breaks both circles: its authenticity comes from the channel
|
||||
it travelled, not from anything the node can check afterwards.
|
||||
|
||||
**This is trust-on-first-use with the first use moved out of band**, which is the difference
|
||||
between a pin and a guess. The node does not accept whatever answers; it accepts the one thing
|
||||
it was told to expect, before it spoke to anything.
|
||||
|
||||
### What this settles
|
||||
|
||||
**The mesh CA is not a bootstrap concern.** It is how `.internal` names are certified once a node
|
||||
is a member, and nothing needs it earlier. The open item
|
||||
[ADR 0049](0049-a-route-is-a-grant.md) left — *what the control plane presents to a node that
|
||||
trusts nothing yet* — is closed: it presents the identity whose fingerprint the token named.
|
||||
|
||||
**Nothing needs name resolution before the link exists**, because the token carries an address.
|
||||
The `/etc/hosts` floor exists to solve a problem that stops existing, and it should go rather than
|
||||
be carried forward — a fallback nothing needs is a path nothing tests.
|
||||
|
||||
**Nothing is placed on disk beforehand except the token.** No database credential, no object-store
|
||||
credential, no registry token. That is ADR 0039's central claim finally made true at the one
|
||||
moment it was still false, and it is the difference between *a node holds only its own identity*
|
||||
being a design statement and being a fact.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The token becomes security-critical in a way it was not**, because it now carries the pin.
|
||||
Tampering with it in transit substitutes the mesh. That is a real exposure and it is strictly
|
||||
better than the alternative: without a pin there is nothing to tamper *with*, and the node
|
||||
trusts the first answer unconditionally. The exposure moves to a channel a person controls and
|
||||
can verify, from one nobody could.
|
||||
- **Token delivery is now a designed step, not an incidental one.** It is short-lived and
|
||||
single-use, so interception is bounded — but how it reaches a machine is part of adoption and
|
||||
needs saying. [Research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md) is
|
||||
where that belongs.
|
||||
- **Rotating the control plane's identity invalidates outstanding tokens**, which is correct and
|
||||
needs to fail legibly. A node presenting a token with a stale fingerprint must be told that,
|
||||
not left to time out.
|
||||
- **The address in the token can go stale.** If the control plane moves, unissued tokens point
|
||||
somewhere wrong. Tokens are short-lived, which bounds it; moving the control plane is
|
||||
[`06`](../03-DESIGN/01-to-be/06-the-control-plane.md)'s undesigned territory regardless.
|
||||
- **A rejoining node is an ordinary case, not a special one.** A node that has lost its identity
|
||||
gets a new token. There is no recovery path to design because there is no long-lived secret to
|
||||
recover.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0039](0039-the-link-is-the-security-boundary.md) — the mutual authority this implements.
|
||||
- [ADR 0038](0038-a-node-joins-by-linking-first.md) — the join this is the first step of.
|
||||
- [ADR 0049](0049-a-route-is-a-grant.md) — the open item this closes.
|
||||
- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) — the `/etc/hosts` floor and the
|
||||
credentials placed beforehand.
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
status: proposed
|
||||
date: 2026-08-27
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 0043-a-declaration-is-an-ordered-list-of-owned-resources.md
|
||||
---
|
||||
|
||||
# 52. A filter rule names its source, or it is not a rule
|
||||
|
||||
## Context
|
||||
|
||||
[Research 004](../01-RESEARCH/004-lab-network/analysis.md) found this while looking for something
|
||||
else:
|
||||
|
||||
> Several manifests declare `scope: public` on firewall rules — `wireguard`, `traefik`, `gitea`,
|
||||
> `mailu`, `qbittorrent`. It is **not part of the rule type** and is **referenced by no code** in
|
||||
> the firewall path. Real scoping is done with `from:`.
|
||||
|
||||
**So five manifests appear to restrict a port and restrict nothing.** Anyone reading them —
|
||||
including whoever wrote the next one by copying — sees an access control that does not exist.
|
||||
|
||||
Research 004 already names the shape: it is `how-we-build`'s *an unenforced rule is
|
||||
indistinguishable from a wrong one, and costs more, because people believe it.* This is that
|
||||
rule, in the firewall, on the modules most worth restricting.
|
||||
|
||||
The mechanism that let it happen is worth more than the instance. `scope:` was accepted because
|
||||
**unknown keys were ignored**. Nothing rejected it, nothing warned, and it spread by copying for
|
||||
long enough to reach five manifests.
|
||||
|
||||
## Decision
|
||||
|
||||
**A rule names its source.** `from:` is the only way to scope a rule, and a rule without one is
|
||||
open — which it must therefore say plainly rather than imply otherwise.
|
||||
|
||||
**`scope:` is removed, not implemented.** Giving it meaning would leave two ways to express one
|
||||
thing, and a manifest carrying both would need a precedence rule nobody would remember. The five
|
||||
manifests are corrected to `from:` where they meant to restrict something, and left open where
|
||||
they did not — and finding out which is which is part of the work, not a formality.
|
||||
|
||||
**An unknown key is refused.** This is the general fix and the reason to bother:
|
||||
|
||||
> A manifest carrying a key the schema does not define is **rejected**, naming the key.
|
||||
|
||||
The host already works this way — its declaration parser sets `DisallowUnknownFields` and
|
||||
collects every problem into one refusal
|
||||
([ADR 0043](0043-a-declaration-is-an-ordered-list-of-owned-resources.md)). **Manifests are the
|
||||
layer where that discipline is missing**, and `scope:` is what missing looks like: not a wrong
|
||||
value, an invented one, silently accepted for months.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **This class of fiction stops at validation** rather than at an audit. A misspelled `form:`,
|
||||
an invented `scope:`, a key from a different schema — each fails on the manifest that
|
||||
introduces it, once, instead of spreading.
|
||||
- **Existing manifests will fail validation**, and some of those failures will be keys somebody
|
||||
believed were doing something. That is the finding, not the cost — but it means the refusal
|
||||
cannot be switched on without reading every manifest first.
|
||||
- **The firewall becomes reviewable.** Today a rule's real effect is only visible by knowing
|
||||
which keys are fictional. Afterwards the manifest says what happens.
|
||||
- **Five manifests need a decision each**, and `wireguard` and `traefik` genuinely are open to
|
||||
the world — they must be, which the manifest should state rather than appear to deny.
|
||||
- **It does not make the rules correct**, only honest. A rule that says `from:` anywhere is
|
||||
legitimately open; this record ensures it says so.
|
||||
|
||||
## References
|
||||
|
||||
- [Research 004](../01-RESEARCH/004-lab-network/analysis.md) §7 — the finding.
|
||||
- [`how-we-build.md`](../00-META/how-we-build.md) — the rule about unenforced rules.
|
||||
- [ADR 0043](0043-a-declaration-is-an-ordered-list-of-owned-resources.md) — unknown-is-refused,
|
||||
where it already holds.
|
||||
@@ -49,7 +49,7 @@ Ten contexts and one interface, from the skeleton
|
||||
| **record** | the event log every other context integrates through |
|
||||
| **inventory** | nodes, modules, assignments, versions |
|
||||
| **config** | settings, secrets, and deriving them onto nodes |
|
||||
| **connectivity** | overlay, resolution, exposure, filtering, certificates — it *decides* routes; the proxy on a node applies them ([ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md)) |
|
||||
| **connectivity** | overlay, resolution, exposure, filtering, certificates — **specified in full in [`08-connectivity.md`](08-connectivity.md)** |
|
||||
| **provisioning** | resource grants between modules |
|
||||
| **delivery** | source to artifact to node |
|
||||
| **observability** | health, logs, metrics, alerts |
|
||||
|
||||
@@ -0,0 +1,233 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-08-27
|
||||
decisions:
|
||||
- 02-DECISIONS/0037-the-host-applies-it-does-not-decide.md
|
||||
- 02-DECISIONS/0039-the-link-is-the-security-boundary.md
|
||||
- 02-DECISIONS/0049-a-route-is-a-grant.md
|
||||
- 02-DECISIONS/0050-reachability-is-a-property-of-the-address.md
|
||||
- 02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md
|
||||
- 02-DECISIONS/0052-a-filter-rule-names-its-source.md
|
||||
---
|
||||
|
||||
# Connectivity
|
||||
|
||||
One of [the control plane's](06-the-control-plane.md) ten contexts, and the one with the most
|
||||
moving parts: **overlay, resolution, exposure, filtering, certificates.**
|
||||
|
||||
It is written as a whole because the five are one design. They share inputs, they must agree, and
|
||||
every one of them today is computed in a different place by a different module from a different
|
||||
copy of the same facts.
|
||||
|
||||
## Why it is control-plane work
|
||||
|
||||
Apply [the test](06-the-control-plane.md) — *everything that needs to know about more than one
|
||||
node* — to each responsibility:
|
||||
|
||||
| | needs to know | whose |
|
||||
|---|---|---|
|
||||
| **overlay** — who peers with whom, at what address | **every node**, and which of them can be dialled | control plane |
|
||||
| **resolution** — which name is which node | **every node** | control plane |
|
||||
| **exposure** — which public name reaches which container | **which node is publicly reachable** ([ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md)) | control plane |
|
||||
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape | control plane decides, host applies |
|
||||
| **certificates** — who may present which name | which name belongs to which node | control plane |
|
||||
|
||||
**Not one of the five can be answered by a machine on its own.** That is the whole reason this is
|
||||
a context rather than a set of node-local modules — and it is exactly what the current
|
||||
arrangement gets wrong, by computing all five on the node from a direct database connection.
|
||||
|
||||
## The shape: decided centrally, delivered as files
|
||||
|
||||
Every one of the five resolves the same way, and it is worth stating once rather than five times:
|
||||
|
||||
> **The connectivity context computes the configuration. It arrives over the link as `file`
|
||||
> resources. The service reads files and knows nothing about the mesh.**
|
||||
|
||||
This costs **no new host vocabulary**. `file`, `directory`, `service` and `container` already
|
||||
exist; WireGuard, the resolver and the proxy are all *a container or a package, plus files*.
|
||||
|
||||
It is also what removes the last two upward dependencies.
|
||||
[Research 006](../../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted exactly two modules
|
||||
opening a direct connection to the control plane's database — `wireguard` and `traefik` — and
|
||||
they are the reason every node permanently holds a credential to it
|
||||
([ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)). Both are connectivity
|
||||
modules. **Closing this context closes that set.**
|
||||
|
||||
## The order it comes up in
|
||||
|
||||
The one thing to get right, because everything else depends on it:
|
||||
|
||||
```
|
||||
0 the node has an underlay address the machine's own — DHCP, or a provider gave it one
|
||||
1 the node dials the mesh OVER THE UNDERLAY, at the address in its token
|
||||
2 it proves itself, and is proved to the link exists (ADR 0039, ADR 0051)
|
||||
3 the mesh grants it an identity and an overlay address
|
||||
4 the overlay comes up peer graph delivered as files
|
||||
5 names resolve resolver config delivered as files
|
||||
6 filtering is applied derived from what is assigned here
|
||||
7 routes and certificates once this node has something to expose
|
||||
```
|
||||
|
||||
**Step 1 runs on the underlay and never on the overlay.** This is the circularity that must not
|
||||
be created: the overlay is configured by the mesh, so a link that required the overlay could
|
||||
never be established on a new node. The link stays on the underlay permanently — it is
|
||||
outbound-only and carries its own identity, so it needs nothing the overlay provides.
|
||||
|
||||
**Nothing before step 3 can resolve a mesh name**, which is why the token carries an *address*
|
||||
([ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md)). Today this is
|
||||
patched with an `/etc/hosts` floor written underneath the resolver; under this design there is
|
||||
nothing to patch.
|
||||
|
||||
## 1 — The overlay
|
||||
|
||||
**What is decided:** the peer graph. For every node: its overlay address, which peers it holds,
|
||||
which of those it may dial, and which must dial it.
|
||||
|
||||
**Inputs, all declared:**
|
||||
|
||||
- **reachability** — an endpoint, or none
|
||||
([ADR 0050](../../02-DECISIONS/0050-reachability-is-a-property-of-the-address.md)). Not
|
||||
inferred from the address shape, which is wrong for carrier-grade NAT, wrong for IPv6, and
|
||||
wrong for a routable address behind a closed firewall.
|
||||
- **site** — where the machine physically is, or nothing if it roams.
|
||||
- **role** — hub or not, **declared**. Today it is inferred from an address prefix, which means a
|
||||
renumbering is an outage and nothing can be asked which node is the hub.
|
||||
|
||||
**Keys.** Each node generates its own keypair. **The private key never leaves the machine**; the
|
||||
public key is published to the mesh. This is already true and it is already right — it is
|
||||
[ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)'s *a node holds its own
|
||||
identity* applied to the overlay, and it means the control plane computes a graph it cannot
|
||||
itself impersonate.
|
||||
|
||||
**Shape: a hub, with direct peering between co-located nodes.**
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| two nodes at the same site | peer **directly**, host-routed, with a keepalive |
|
||||
| everything else | routes through the **hub** |
|
||||
| a node with no site — it roams | **hub only** |
|
||||
|
||||
**Roaming is hub-only deliberately, and the reason is a property of WireGuard rather than a
|
||||
preference: there is no failover.** A more specific route to a dead endpoint blackholes; it does
|
||||
not fall back to the general one. So a node whose location changes gets exactly one path, because
|
||||
two paths would mean one of them silently swallowing traffic.
|
||||
|
||||
**What the host receives:** an interface configuration and a peer list, as files. It does not
|
||||
compute them, and after this it holds no credential to the mesh's database.
|
||||
|
||||
## 2 — Resolution
|
||||
|
||||
**Two name spaces, and they do not mix:**
|
||||
|
||||
| | resolves to | certified by |
|
||||
|---|---|---|
|
||||
| **internal names** | overlay addresses | the **mesh CA** |
|
||||
| **public names** | whatever the outside world must reach | a **public authority** |
|
||||
|
||||
A node's mesh name is its overlay address. Its public name, if it has one, is a separate fact
|
||||
used by things outside the mesh — and the separation carries two lessons that were learned
|
||||
expensively enough to be worth restating:
|
||||
|
||||
- **Mesh names are not multicast names.** A name resolved by local multicast discovery introduces
|
||||
a delay and a failure mode that appears on one node and not others — the worst shape a fault
|
||||
can have.
|
||||
- **A node must not pin its own public name locally.** The duplicate record breaks resolution of
|
||||
that name for everything else that needs it.
|
||||
|
||||
**What the host receives:** the resolver's configuration, as files, listing every peer's internal
|
||||
name and overlay address.
|
||||
|
||||
**What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh
|
||||
database before its own DNS existed; with [ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md)
|
||||
nothing needs a name before the link, and a fallback nothing needs is a path nothing tests.
|
||||
|
||||
## 3 — Exposure
|
||||
|
||||
Settled by [ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md); summarised here because
|
||||
this is where it belongs.
|
||||
|
||||
**A route is a grant.** A module that must be reachable declares it needs one; the proxy provides
|
||||
it and hands back the public name. Ordinary
|
||||
[ADR 0044](../../02-DECISIONS/0044-a-module-declares-presence-instantiation-and-exclusion.md)
|
||||
vocabulary — the mirror of a database grant, where the consumer supplies a target and receives a
|
||||
name rather than supplying nothing and receiving credentials.
|
||||
|
||||
**A workload on an unreachable node is proxied by a reachable one, across the overlay.** Which is
|
||||
the case is a mesh-level fact, which is the fourth reason exposure is control-plane work.
|
||||
|
||||
## 4 — Filtering
|
||||
|
||||
**Derived from what is assigned here, and from the overlay's shape** — a node's open ports are a
|
||||
consequence of what runs on it and who must reach it, not an independent declaration to keep in
|
||||
step by hand.
|
||||
|
||||
**A rule names its source** ([ADR 0052](../../02-DECISIONS/0052-a-filter-rule-names-its-source.md)).
|
||||
A rule with no source is open, and must say so rather than appear to restrict something. `scope:`
|
||||
is removed rather than implemented: five manifests carry it today, it is referenced by no code,
|
||||
and it is the clearest instance in the repository of *an unenforced rule is indistinguishable
|
||||
from a wrong one, and costs more, because people believe it.*
|
||||
|
||||
**Unknown keys are refused** — the discipline the host's declaration parser already has
|
||||
([ADR 0043](../../02-DECISIONS/0043-a-declaration-is-an-ordered-list-of-owned-resources.md)), and
|
||||
the one manifests lack. `scope:` survived because nothing rejected it.
|
||||
|
||||
## 5 — Certificates
|
||||
|
||||
**Two authorities, kept separate on purpose.**
|
||||
|
||||
| | issued by | for |
|
||||
|---|---|---|
|
||||
| **public names** | a public ACME authority | anything outside the mesh reaches |
|
||||
| **internal names** | the **mesh CA** | node-to-node, over the overlay |
|
||||
|
||||
**The split is not collapsed, including in the lab.** A single-CA lab would hide any bug living
|
||||
in the split, so the lab runs its own ACME issuer on its public segment and keeps the mesh CA
|
||||
unchanged ([research 004](../../01-RESEARCH/004-lab-network/00-overview.md)).
|
||||
|
||||
**Public issuance requires genuine public reachability.** The HTTP-01 challenge must be answered
|
||||
at the name being certified, so issuance happens through a publicly reachable node regardless of
|
||||
where the workload runs — the same asymmetry as exposure, for the same reason.
|
||||
|
||||
**The issuer must be configurable.** Today it is not: the proxy sets no `caServer` and therefore
|
||||
defaults to the public authority's *production* endpoint. Two consequences, and the second is
|
||||
worse than the lab problem that found it — every certificate experiment on a real node consumes
|
||||
production issuance quota, and a retry loop can exhaust it for a week.
|
||||
|
||||
**The mesh CA is not a bootstrap concern.** A joining node verifies the control plane against the
|
||||
fingerprint in its token ([ADR 0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md)),
|
||||
so nothing needs the CA before membership. It certifies internal names afterwards, and that is
|
||||
all it does.
|
||||
|
||||
## What this removes
|
||||
|
||||
The list is worth having in one place, because it is most of the argument:
|
||||
|
||||
- **The last two direct database connections from nodes** — `wireguard` and `traefik`, the only
|
||||
two, both connectivity.
|
||||
- **Therefore the database credential on every node**, and the object-store credential beside it.
|
||||
[ADR 0039](../../02-DECISIONS/0039-the-link-is-the-security-boundary.md)'s central claim becomes
|
||||
true rather than aspirational.
|
||||
- **The `/etc/hosts` floor**, and the bootstrap circularity it patched.
|
||||
- **Hub election by address prefix**, and the silent no-hub failure when nobody knew the
|
||||
convention.
|
||||
- **The RFC1918 inference**, and the lab substitution that existed to satisfy it.
|
||||
- **`scope:`**, and the class of manifest key that means nothing.
|
||||
|
||||
## Open
|
||||
|
||||
- **What happens when the hub is down.** WireGuard has no failover, and the hub is a single point
|
||||
through which every non-co-located pair routes. This design does not add a second hub and does
|
||||
not pretend the first one is redundant. It is the same shape as
|
||||
[`06`](06-the-control-plane.md)'s *how many run, and what a node does without one*, and it
|
||||
should be answered with it rather than separately.
|
||||
- **Renumbering the overlay.** Made *possible* by declaring the hub rather than inferring it from
|
||||
an address, but no procedure exists, and a graph delivered node by node has an ordering problem
|
||||
while it is half-applied.
|
||||
- **Revoking a route** when a module is unassigned ([ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md)).
|
||||
A stale public name pointing at nothing fails more visibly than a stale grant.
|
||||
- **IPv6.** [ADR 0050](../../02-DECISIONS/0050-reachability-is-a-property-of-the-address.md) makes
|
||||
it expressible; nothing here says the overlay or the resolver handle it.
|
||||
- **Reporting declared-versus-observed.** ADR 0050 makes the disagreement detectable and does not
|
||||
say who looks or what they are told.
|
||||
@@ -16,13 +16,15 @@ document is written and this one's status becomes `implemented`.
|
||||
| [`04-lab-installation.md`](04-lab-installation.md) | Getting the lab onto a clean machine, and why it verifies capability rather than installation | [ADR 0008](../../02-DECISIONS/0008-a-failed-step-fails-the-job.md) |
|
||||
| [`05-the-node-host.md`](05-the-node-host.md) | Tier 0 — the one thing installed by hand, and the only thing that changes a machine | [ADR 0037](../../02-DECISIONS/0037-the-host-applies-it-does-not-decide.md) |
|
||||
| [`06-the-control-plane.md`](06-the-control-plane.md) | Tier 2 — what the term means, and the test for what belongs in it | [ADR 0037](../../02-DECISIONS/0037-the-host-applies-it-does-not-decide.md) |
|
||||
| [`07-the-substrate.md`](07-the-substrate.md) | Tier 1 — what the control plane consumes and cannot grant itself | [ADR 0038](../../02-DECISIONS/0038-a-node-joins-by-linking-first.md) |
|
||||
| [`07-the-substrate.md`](07-the-substrate.md) | Tier 1 — what the control plane consumes and cannot grant itself | [ADR 0038](../../02-DECISIONS/0038-a-node-joins-by-linking-first.md), [0048](../../02-DECISIONS/0048-the-substrate-is-named.md) |
|
||||
| [`08-connectivity.md`](08-connectivity.md) | One context in full — overlay, resolution, exposure, filtering, certificates | [ADR 0049](../../02-DECISIONS/0049-a-route-is-a-grant.md), [0050](../../02-DECISIONS/0050-reachability-is-a-property-of-the-address.md), [0051](../../02-DECISIONS/0051-the-enrolment-token-carries-the-mesh.md) |
|
||||
|
||||
## Not yet written
|
||||
|
||||
- **The eight bounded contexts.** [ADR 0015](../../02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md)
|
||||
decides the decomposition; the per-context specifications do not exist yet. The work
|
||||
breakdown says in what order they are needed.
|
||||
- **The remaining contexts.** [ADR 0015](../../02-DECISIONS/0015-mesh-brokers-nodes-host-agents-think.md)
|
||||
decides the decomposition; `connectivity` is the first written in full
|
||||
([`08`](08-connectivity.md)) and the others do not exist yet. The work breakdown says in what
|
||||
order they are needed.
|
||||
- **Domain grouping outside the core.** [ADR 0017](../../02-DECISIONS/0017-modules-outside-the-core-are-grouped-by-domain.md)
|
||||
settles the principle and explicitly does not settle the domain list. That is a research
|
||||
effort, not a design document, until it concludes.
|
||||
|
||||
Reference in New Issue
Block a user