jschoubben
b4607dfc03
Decided after measuring what renumbering actually costs: 96 references in code comments across two repositories, none of which would have failed to compile. They would have pointed at the wrong reasoning, which is worse than a broken link because nothing reports it. So a number identifies a record and never changes. It cannot also be a position -- a position moves when the set changes, and an identity that moves is not one. The reading order moves into an index generated from each record's `topic:`. Six topics, in the order somebody learns the system. The index is WRITTEN rather than only generated on demand, which reverses what this repository previously said. The reason it said otherwise is that a hand-written index drifts -- but a reader looking at the folder on a forge sees the folder, not a command, and the drift objection is answered by checking rather than by refusing to write one. That is §5's own rule: a rule states how it is checked. Two checks, both confirmed to bite. index.py fails when the written order no longer matches the records. records.py fails when a record has no topic or one nobody defined -- the quiet failure being a record that vanishes from the order rather than appearing in the wrong place.
125 lines
6.1 KiB
Markdown
125 lines
6.1 KiB
Markdown
---
|
|
topic: the tiers
|
|
status: accepted
|
|
date: 2026-08-28
|
|
deciders: jochen
|
|
reconstructed: false
|
|
---
|
|
|
|
# 7. Connectivity
|
|
|
|
*Consolidated 2026-08-28 from three records. Overlay, resolution, exposure, filtering and
|
|
certificates are one design.*
|
|
|
|
## Why it is control-plane work
|
|
|
|
Apply the test — *everything that needs to know about more than one node* — and not one of the
|
|
five can be answered by a machine on its own:
|
|
|
|
| | needs to know |
|
|
|---|---|
|
|
| **overlay** — who peers with whom | every node, and which can be dialled |
|
|
| **resolution** — which name is which node | every node |
|
|
| **exposure** — which public name reaches which container | which node is publicly reachable |
|
|
| **filtering** — which port is open, to whom | what is assigned here, and the overlay's shape |
|
|
| **certificates** — who may present which name | which name belongs to which node |
|
|
|
|
That is exactly what the current arrangement gets wrong, by computing all five on the node from a
|
|
direct database connection. Two modules do this, and they are the only two left holding a
|
|
credential to the control plane's database.
|
|
|
|
**The shape of the fix, once for all five:** the connectivity context computes the configuration;
|
|
it arrives over the link as `file` resources; the service reads files and knows nothing about the
|
|
mesh. **This costs no new host vocabulary.**
|
|
|
|
## A route is a grant
|
|
|
|
**Ingress is not substrate.** The control plane does not need a route to start — it listens
|
|
locally — and no node needs one to reach it, because the node dials out and has no listening
|
|
control surface. It grants itself a route afterwards, the way it grants itself a bucket.
|
|
|
|
The strongest objection deserves stating: the `api` is the one interface every surface speaks to,
|
|
so eventually it *does* want a public name. But **wanting one later is not needing one to
|
|
start**, and that distinction is the entire substrate test.
|
|
|
|
**A module that must be reachable declares it needs a route; the proxy provides one.** Ordinary
|
|
instantiation, with the direction mirrored — the consumer supplies a target and receives a name.
|
|
|
|
**Exposure is three facts at two scopes**, which is why it cannot live on the node:
|
|
|
|
| the fact | scope |
|
|
|---|---|
|
|
| the public name resolves to an address | **mesh** — which node is publicly reachable |
|
|
| a certificate valid for that name exists | **mesh** — issued once, used on one node |
|
|
| the proxy maps that name to that container | **node** |
|
|
|
|
**A node without a public address is proxied by one that has**, across the overlay. Most nodes sit
|
|
behind a connection with no forwarded port, so exposure cannot assume the workload's node is
|
|
reachable.
|
|
|
|
## Reachability is declared, not inferred
|
|
|
|
The overlay's peer graph is computed from whether a node can be dialled, and that was inferred
|
|
from a regular expression over the address. **The address is evidence of reachability; it is not
|
|
the fact**, and the gap has already cost:
|
|
|
|
| address | the regex says | actually |
|
|
|---|---|---|
|
|
| `100.64.0.0/10` — carrier-grade NAT | **public** | **not reachable.** An endpoint is written to an address nothing can reach |
|
|
| any IPv6 address | public | the test is v4 shapes only |
|
|
| a routable address behind a closed firewall | public | not reachable |
|
|
| a documentation range standing in for a public segment | private | reachable — this is the lab bug |
|
|
|
|
**A test environment having to choose its addresses to satisfy a regex is the regex telling us it
|
|
is not a fact.**
|
|
|
|
So: **an endpoint, or none** — declared. And **the hub is declared, never derived from an address
|
|
prefix**, because an election decided by the first four characters of an address fails silently,
|
|
cannot be queried, and makes a renumbering an outage.
|
|
|
|
**The address remains evidence and stops being the fact.** Where an observed endpoint disagrees
|
|
with a declared one, the disagreement is a **reportable condition**, not a silent correction.
|
|
|
|
**What does not change** is the lesson underneath: role does not imply reachability — a
|
|
home-hosted node is a server that cannot be dialled. This keeps that and stops encoding it as a
|
|
pattern match.
|
|
|
|
## A filter rule names its source
|
|
|
|
`scope: public` is declared in five manifests, is part of no rule type, and is **referenced by no
|
|
code**. So five manifests appear to restrict a port and restrict nothing — on the modules most
|
|
worth restricting.
|
|
|
|
**A rule names its source. `from:` is the only way to scope one, and a rule without one is open**
|
|
— which it must say plainly rather than appear to deny.
|
|
|
|
**`scope:` is removed rather than implemented**, because giving it meaning would leave two ways to
|
|
express one thing. And the general fix is that **an unknown key is refused**: the host's
|
|
declaration parser already works this way, and manifests are the layer where that discipline is
|
|
missing. `scope:` survived because nothing rejected it, and it spread by copying to five
|
|
manifests.
|
|
|
|
## Order, and what it costs
|
|
|
|
**The link runs on the underlay and never on the overlay.** The overlay is configured by the mesh,
|
|
so a link requiring it could never be established on a new node.
|
|
|
|
**The first declaration is the overlay and nothing else** — because a node's address and peers are
|
|
*assigned* so it cannot come earlier, and because it is the way back in. A node reachable over the
|
|
overlay can be fixed by hand if a later declaration breaks the machine; **a large first
|
|
declaration risks a node that is broken and unreachable at once.**
|
|
|
|
**Reachable is not the same as having a control surface.** Every node reaches every other over the
|
|
overlay — SSH, services, ordinary traffic — and every node consumes from the broker. What is
|
|
forbidden is a listening thing that accepts instructions and changes the machine.
|
|
|
|
## Consequences
|
|
|
|
- **The last two direct database connections leave the nodes**, and with them the database
|
|
credential every node carries.
|
|
- **The `/etc/hosts` floor goes**, along with the bootstrap circularity it patched.
|
|
- **Two certificate authorities stay separate on purpose**: a public one for public names, the
|
|
mesh's own for internal ones. A single-CA lab would hide any bug living in the split.
|
|
- **What happens when the hub is down**: nothing takes over. Non-co-located paths stop; co-located
|
|
peers and every assigned workload keep running.
|