jschoubben
4e80820e2f
Written as one document because the five are one design. They share inputs, they must agree, and every one of them today is computed in a different place by a different module from a different copy of the same facts. The through-line is that none of the five can be answered by a machine alone, so all five are decided centrally and delivered as `file` resources. That costs no new host vocabulary and removes both remaining direct database connections from nodes -- wireguard and traefik are the only two, and both are connectivity. Three decisions fall out, all proposed: 0050 -- reachability is declared, not inferred from an address. The RFC1918 regex is wrong for carrier-grade NAT (100.64/10 tests as public, so an endpoint is written to an address nothing can reach), wrong for IPv6, and wrong for a routable address behind a closed firewall. The lab needing TEST-NET-3 to satisfy the regex is the same bug from the other side. Also kills hub election by address prefix, which fails silently and makes renumbering an outage. 0051 -- the enrolment token carries where the mesh is and how to recognise it. Closes two circles with one mechanism: verifying the mesh needed the CA, and obtaining the CA meant trusting whoever handed it over; and a node had to reach the mesh before it could resolve any mesh name. An address plus a fingerprint, carried out of band, resolves both -- and closes the CA question 0049 deferred. 0052 -- a filter rule names its source. `scope:` is declared in five manifests, is part of no rule type, and is referenced by no code, so those manifests appear to restrict ports and restrict nothing. Removed rather than implemented; the general fix is refusing unknown keys, which the host already does and manifests do not. Also corrects two claims in 0049 asserting wireguard was already handled. Research 006 says both modules still reach upward; neither is.
154 lines
8.2 KiB
Markdown
154 lines
8.2 KiB
Markdown
---
|
|
status: proposed
|
|
date: 2026-08-27
|
|
deciders: jochen
|
|
reconstructed: false
|
|
extends: 0044-a-module-declares-presence-instantiation-and-exclusion.md
|
|
---
|
|
|
|
# 49. A route is a grant
|
|
|
|
## Context
|
|
|
|
[ADR 0048](0048-the-substrate-is-named.md) named the substrate's products and, in doing so,
|
|
found a hole rather than a naming problem: the `connectivity` context lists *exposure* and
|
|
*certificates* among its responsibilities, and **no document says what terminates TLS, how a
|
|
public name reaches a container, or which tier owns any of it.**
|
|
|
|
Traefik is what does it today, and how it does it is the problem.
|
|
[Research 006](../01-RESEARCH/006-mesh-from-scratch/host-size.md) counted it as one of only two
|
|
modules that **open a direct Postgres connection to the control plane's database** — reading
|
|
`nodes` and `mesh_ca` and computing its own configuration from them.
|
|
|
|
That is three violations in one module:
|
|
|
|
- **[ADR 0037](0037-the-host-applies-it-does-not-decide.md)** — it decides, on the node, from
|
|
mesh-wide knowledge.
|
|
- **[ADR 0045](0045-a-context-owns-its-store.md)** — it reads another context's tables directly.
|
|
- **[ADR 0039](0039-the-link-is-the-security-boundary.md)** — it is the reason every node
|
|
permanently holds a credential to the control plane's database.
|
|
|
|
So exposure was never designed; it was accreted, and it is one of the two things standing
|
|
between the current arrangement and the security boundary ADR 0039 describes.
|
|
|
|
## Decision
|
|
|
|
### Ingress is not substrate
|
|
|
|
The test from [`07-the-substrate.md`](../03-DESIGN/01-to-be/07-the-substrate.md), applied
|
|
honestly:
|
|
|
|
| | |
|
|
|---|---|
|
|
| Does the control plane need a route to **start**? | **No.** It listens locally. |
|
|
| Does a **node** need one to reach it? | **No** — the node dials out over AMQP, and has no listening control surface at all ([ADR 0039](0039-the-link-is-the-security-boundary.md)). |
|
|
| Does anything need one **before the control plane runs**? | **No.** |
|
|
|
|
**So ingress is an ordinary provider module**, provisioned like anything else once a mesh exists.
|
|
|
|
The strongest objection deserves stating rather than dodging: the control plane's `api` is the
|
|
one interface every surface speaks to, so eventually it *does* want a public name. But *wanting
|
|
one later* is not *needing one to start* — that distinction is the entire substrate test, and
|
|
ingress is on the ordinary side of it. At bootstrap the first node's surface is reached locally,
|
|
and the mesh grants itself a route afterwards, the same way it grants itself a bucket.
|
|
|
|
### A route is an instantiation edge
|
|
|
|
A module that must be reachable declares it needs a **route**, and the proxy module provides
|
|
one. This is [ADR 0044](0044-a-module-declares-presence-instantiation-and-exclusion.md)'s
|
|
instantiation edge with no extension: a provider makes something for a consumer and hands back
|
|
an identity.
|
|
|
|
The direction is worth noticing, because it is the mirror of a database and could be mistaken
|
|
for a different kind of thing:
|
|
|
|
| | a database grant | a route grant |
|
|
|---|---|---|
|
|
| consumer supplies | nothing | where to send traffic |
|
|
| consumer receives | credentials | **the public name it is reachable at** |
|
|
|
|
Both are still *the provider made something and told the consumer how to use it*, which is what
|
|
the edge means. Nothing new is required.
|
|
|
|
### Exposure is three facts at two scopes, and that is why it is a context
|
|
|
|
The reason this cannot simply live on the node:
|
|
|
|
| The fact | Scope | Whose |
|
|
|---|---|---|
|
|
| the public name resolves to an address | **mesh** — which node is publicly reachable | the **control plane**, `connectivity` |
|
|
| a certificate valid for that name exists | **mesh** — issued once, for a name, used on one node | the **control plane**, `connectivity` |
|
|
| the proxy maps that name to that container | **node** | the **host**, applying a declaration |
|
|
|
|
Two of the three need to know about more than one node, which is exactly
|
|
[the control plane's definition](../03-DESIGN/01-to-be/06-the-control-plane.md). The third is a
|
|
single machine's business. The line falls where the tier rule already puts it, and the current
|
|
arrangement is wrong precisely because Traefik does all three on the node.
|
|
|
|
### The proxy reads files; it does not read the mesh
|
|
|
|
The connectivity context computes the proxy's configuration and the certificate, and they arrive
|
|
over the link as **`file` resources**.
|
|
|
|
**This costs zero new host vocabulary.** `file` already exists — it was one of the first three
|
|
shapes built. The proxy becomes a `container` with `file` configuration, which the host already
|
|
knows how to apply and read back.
|
|
|
|
And it removes a database credential from every node, which is half of what
|
|
[ADR 0039](0039-the-link-is-the-security-boundary.md) is for.
|
|
[ADR 0037](0037-the-host-applies-it-does-not-decide.md) decided this in principle; **neither
|
|
offender has actually been changed**, and this applies it to one of the two.
|
|
|
|
### A node without a public address is routed through one that has
|
|
|
|
Most nodes sit behind a connection with no forwarded port
|
|
([research 004](../01-RESEARCH/004-lab-network/00-overview.md)), so exposure cannot assume the
|
|
workload's node is reachable. Two cases, and the mesh must handle both because the difference is
|
|
invisible until it matters:
|
|
|
|
- **the node is publicly reachable** — the proxy runs there and the route is direct;
|
|
- **it is not** — a publicly reachable node proxies to it across the overlay.
|
|
|
|
Which case applies is a mesh-level fact, which is the fourth reason exposure is control-plane
|
|
work. Research 004 already records the hard variant — *a node that is publicly named but sits
|
|
behind NAT* — as the case the lab exists to get right.
|
|
|
|
## Consequences
|
|
|
|
- **Traefik's upward dependency is removed, and the module mostly disappears.** It computed its
|
|
own configuration; now it is an image plus files somebody else derived. Of the 426 lines
|
|
research 006 counted, what survives is a declaration.
|
|
- **One of the two direct database connections goes.** `wireguard` is the other and is *not*
|
|
yet handled — [ADR 0050](0050-reachability-is-a-property-of-the-address.md) is what handles it,
|
|
and only both together close the set ADR 0039 identified. Until then a node still holds the
|
|
credential, so this record alone changes the design and not the exposure.
|
|
- **Certificate issuance becomes a control-plane responsibility with a real constraint**: an
|
|
ACME challenge can only be answered at a publicly reachable address, so issuance happens
|
|
through a public node regardless of where the workload runs. The lab already runs its own
|
|
issuer, so this is testable ([research 004](../01-RESEARCH/004-lab-network/00-overview.md)).
|
|
- **The proxy is a presence edge for anything exposed.** A module with a route needs the proxy on
|
|
its node — ordinary ADR 0044 vocabulary, no special case.
|
|
- **This does not settle the mesh's internal CA.** `mesh_ca` is the *other* thing Traefik reads,
|
|
and it belongs to a different question: ADR 0039 requires the control plane to prove it is the
|
|
mesh, which needs something a joining node can verify before it trusts anything. **Internal
|
|
identity and public exposure are two certificate stories and this record only closes the
|
|
second.** Conflating them is what made the gap hard to see.
|
|
**Since closed** by [ADR 0051](0051-the-enrolment-token-carries-the-mesh.md): the joining node
|
|
verifies the control plane against a fingerprint carried in its enrolment token, so nothing
|
|
needs the CA before membership.
|
|
- **Nothing says how a route is revoked** when a module is unassigned. The grant model implies
|
|
it — removing a consumer drops what it was granted
|
|
([ADR 0045](0045-a-context-owns-its-store.md)) — but a stale public name pointing at nothing is
|
|
a more visible failure than a stale database, and it is not designed.
|
|
|
|
## References
|
|
|
|
- [ADR 0044](0044-a-module-declares-presence-instantiation-and-exclusion.md) — the edge a route is.
|
|
- [ADR 0037](0037-the-host-applies-it-does-not-decide.md) — why the proxy may not decide.
|
|
- [ADR 0039](0039-the-link-is-the-security-boundary.md) — the credential this removes.
|
|
- [ADR 0048](0048-the-substrate-is-named.md) — which named this gap and left it open.
|
|
- [Research 006](../01-RESEARCH/006-mesh-from-scratch/host-size.md) — the count and the two
|
|
offenders.
|
|
- [Research 004](../01-RESEARCH/004-lab-network/00-overview.md) — the topology and the lab's
|
|
own issuer.
|