ADR 0194: the mesh has one resolver, and every node asks it for the mesh's names
Every resolution fault found on 2026-10-03 was a per-node copy disagreeing with the truth: a hosts file read once, an operator's old line beside the mesh's, a node's resolver lent to a LAN. Every tunnel already converges on one node. Retires node-dns-resolver for a mesh-scoped mesh-resolver; nodes route only the mesh's suffix to it. Narrows 0121; amends connectivity §2 and the seats.
This commit is contained in:
@@ -9,6 +9,12 @@ extends: 0110-a-seat-is-a-module-assignment-from-a-closed-set.md
|
||||
|
||||
# 121. A system seat is named for its scope, and a module may define its own
|
||||
|
||||
> **Narrowed, not replaced — 2026-10-03.** *"`the-dns-port` → `node-dns-resolver`"* no longer holds:
|
||||
> the serving role moves to mesh scope as `mesh-resolver`, one per mesh, and `node-dns-resolver` is
|
||||
> retired ([ADR 0194](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)). The
|
||||
> distinction this record kept — serving and asking are two roles, two seats — stands, and
|
||||
> `node-resolver-config` is unchanged.
|
||||
|
||||
> **The mechanism changed — 2026-10-02, by [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md).** The naming rule stands. The build role this record made mesh-scoped — *the mesh's single build machine* — is node-scoped now: `node-build-agent`, one holder per machine, every holder taking from one work queue.
|
||||
|
||||
## Context
|
||||
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
---
|
||||
topic: the tiers
|
||||
status: accepted
|
||||
date: 2026-10-03
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
supersedes-in-part:
|
||||
- 0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md
|
||||
extends: 0191-the-meshs-resolver-holds-only-the-meshs-own-names.md
|
||||
---
|
||||
|
||||
# 194. The mesh has one resolver, and every node asks it for the mesh's names
|
||||
|
||||
## Context
|
||||
|
||||
**Every node runs its own resolver and holds its own copy of the mesh's names.** On the production
|
||||
mesh on 2026-10-03, each of the four nodes held `node-dns-resolver` with dnsmasq, fed on every push
|
||||
with a zones file (one wildcard per node) and a region of `/etc/hosts` (the machines), and pointed
|
||||
its own `/etc/resolv.conf` at itself. The controller computes the names once; four daemons then hold
|
||||
four copies, each read in its own way.
|
||||
|
||||
**Every resolution fault found that day was a copy disagreeing with the truth, not the truth being
|
||||
wrong:**
|
||||
|
||||
- **A copy read once.** dnsmasq reads `/etc/hosts` at start. After the controller stopped publishing
|
||||
public names ([ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md)), every node's
|
||||
hosts file was right and every resolver still answered the mail server's public name with a
|
||||
tunnel address, until each was restarted.
|
||||
- **A copy beside other copies.** On the workstation, a name resolved to two addresses in rotation:
|
||||
the mesh's region gave the tunnel address, and two lines the operator had written before the mesh
|
||||
existed — one in `/etc/hosts`, one in a file the resolver also reads — gave the LAN address. A
|
||||
comment beside one of them said to delete it once the mesh took over; nothing made that happen.
|
||||
- **A copy that became somebody else's resolver.** The home server's resolver also answers its LAN
|
||||
(a listen address added 2026-10-02), and the LAN's router hands that address out as the only DNS
|
||||
server. Every phone and television on the LAN resolved through a mesh node's private copy, which is
|
||||
how ADR 0191's outage reached them.
|
||||
|
||||
**And the overlay already has one centre.** Every node has exactly one tunnel peer — the anchor —
|
||||
and routes the whole private range through it. Two nodes on the same LAN reach each other through
|
||||
the anchor. So a name under `.internal` is only ever useful while the anchor is reachable: a resolver
|
||||
anywhere else adds a copy without adding an answer anybody can use.
|
||||
|
||||
**What a node asks is already a separate role.** The connectivity design split *serving* (answers
|
||||
the names) from *asking* (decides what the machine asks), because systemd-resolved cannot answer a
|
||||
wildcard and can only route the mesh's suffix to something that can
|
||||
([connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md)). ADR 0121 kept them as two seats,
|
||||
`node-dns-resolver` and `node-resolver-config`, both at node scope. No node runs systemd-resolved
|
||||
today; each writes `/etc/resolv.conf` as a plain file pointing at its own dnsmasq.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**1. Keep a resolver on every node, and make the copies more careful.** Restart on every file it
|
||||
reads, own every file it reads, refuse to listen on a LAN. Each is a fix for one way a copy goes
|
||||
stale, and the next way is not on the list yet. It keeps four answers to one question.
|
||||
|
||||
**2. One resolver for the mesh, and every node sends it every query.** The simplest asking side —
|
||||
`resolv.conf` names the mesh's resolver and nothing else. Rejected: public resolution then depends on
|
||||
the tunnel. A laptop whose tunnel is down could resolve nothing at all, and a public name would take
|
||||
a detour through the anchor for no reason ADR 0191 left standing.
|
||||
|
||||
**3. One resolver for the mesh's names; each node asks it for those only.** The mesh's resolver holds
|
||||
every node's internal domain. Each node's asking role routes the mesh's suffix to it and every other
|
||||
name to public resolvers. Chosen.
|
||||
|
||||
## Decision
|
||||
|
||||
**The mesh has one resolver.** It is a module holding a new mesh-scoped seat, **`mesh-resolver`**
|
||||
(capacity one). It holds each node's internal domain — `<node>.internal` and everything under it, at
|
||||
that node's private address ([ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md))
|
||||
— and listens on the private network only. It is placed on the node every tunnel converges on, so
|
||||
that it shares the overlay's single point rather than adding one. Which daemon fills the seat stays
|
||||
the module's business, as the connectivity design says.
|
||||
|
||||
**Every node asks it for the mesh's names and nothing else.** `node-resolver-config` routes the
|
||||
mesh's suffix to `mesh-resolver` and leaves every other name with public resolvers. Plain
|
||||
`resolv.conf` cannot route by domain, so the asking side is a stub that can — systemd-resolved, with
|
||||
the tunnel's link carrying the mesh resolver and the suffix as its routing domain. The stub holds no
|
||||
names; it decides only where a question goes.
|
||||
|
||||
**A container asks the mesh's resolver directly.** The container runtime cannot use a loopback stub
|
||||
and drops its routing domains, so the runtime's `dns` names `mesh-resolver`, which forwards public
|
||||
names for the containers that ask it. This is the one place a public name passes through the mesh,
|
||||
and it is stated rather than hidden.
|
||||
|
||||
**`node-dns-resolver` is retired**, and with it every per-node copy: the zones file, the mesh's region
|
||||
of `/etc/hosts` (the floor connectivity §2 already planned to remove), and the daemon on every node
|
||||
but the one holding `mesh-resolver`. This narrows ADR 0121's *"the-dns-port → node-dns-resolver"*:
|
||||
the serving role keeps its distinction from the asking role and moves to mesh scope, as ADR 0121 did
|
||||
for the private network.
|
||||
|
||||
**A LAN's resolver is not the mesh's.** No device that is not a member can reach a private address,
|
||||
so no member's resolver answers a LAN on the mesh's behalf. A router that hands out a node's address
|
||||
as a LAN's DNS server is pointed elsewhere before that node stops answering.
|
||||
|
||||
**The order is fixed, because every step before the last leaves a working resolver:**
|
||||
|
||||
1. `mesh-resolver` is assigned and answers on the private network.
|
||||
2. Each node's `node-resolver-config` switches to the routing stub, and the container runtime's `dns`
|
||||
to `mesh-resolver`.
|
||||
3. A LAN whose router points at a node's resolver is pointed at its router or a public resolver.
|
||||
4. `node-dns-resolver` is unassigned from every node, and the hosts region is withdrawn.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **One answer per name.** A name is wrong in one place or right everywhere; no node can hold a copy
|
||||
that disagrees, and no operator file on a node is read by the mesh's resolver.
|
||||
- **The anchor down means no `.internal` names** — which it already meant for `.internal` traffic,
|
||||
since every tunnel goes through it. Public resolution on every node is unaffected.
|
||||
- **A container's public resolution depends on the mesh's resolver.** Accepted, and named in the
|
||||
decision; a container that must resolve public names with the tunnel down is the case it costs.
|
||||
- **Every node gains systemd-resolved as its asking stub**, which today none runs. It is configured
|
||||
by the module holding `node-resolver-config`; the mesh still ships no resolver of its own.
|
||||
- **The runtime's `dns` changes once per node**, which the runtime reads only at start. With
|
||||
`live-restore` already on, that restart keeps every container running.
|
||||
- **A LAN loses a resolver it had borrowed.** The router change is an explicit step, done through
|
||||
the module that manages the router, before the node's resolver goes.
|
||||
|
||||
**How each is checked:**
|
||||
|
||||
- **One holder:** the seat has capacity one, so a second assignment is refused by the controller.
|
||||
- **Asking:** on each node, `resolvectl` shows the tunnel's link with `mesh-resolver` and the suffix as
|
||||
its routing domain; a name under `.internal` is answered by it, and a public name is answered
|
||||
without it (its query log shows no public name from a node).
|
||||
- **No copies:** no node but the holder listens on port 53, and no node's `/etc/hosts` carries a mesh
|
||||
region.
|
||||
- **A LAN:** the router's DHCP DNS option names no node's address.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md) — what the mesh's resolver
|
||||
holds; this record decides where it runs and how nodes reach it.
|
||||
- [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) — the two
|
||||
resolver seats, and the private network's move to mesh scope this mirrors.
|
||||
- [Connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md) — serving and asking as two roles;
|
||||
amended alongside this record.
|
||||
- [The seats](../03-DESIGN/01-to-be/26-the-seats.md) — the seat table, amended alongside.
|
||||
@@ -223,6 +223,7 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0148** — [The mesh's names are resolved, not copied into every container](0148-the-meshs-names-are-resolved-not-copied-into-containers.md)
|
||||
- **0151** — [A route's internal name is composed under the node that serves it](0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md)
|
||||
- **0191** — [The mesh's resolver holds only the mesh's own names; a public name resolves publicly](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md)
|
||||
- **0194** — [The mesh has one resolver, and every node asks it for the mesh's names](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)
|
||||
|
||||
### What runs on them, and how it gets there
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ code:
|
||||
- mesh-host internal/apply (the service that reflects a rule set)
|
||||
updated: 2026-10-03
|
||||
decisions:
|
||||
- 02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md
|
||||
- 02-DECISIONS/0191-the-meshs-resolver-holds-only-the-meshs-own-names.md
|
||||
- 02-DECISIONS/0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md
|
||||
- 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
|
||||
@@ -290,8 +291,9 @@ expensively enough to be worth restating:
|
||||
- **A node must not pin its own public name locally.** The duplicate record breaks resolution of
|
||||
that name for everything else that needs it.
|
||||
|
||||
**What the host receives:** the resolver's configuration, as files, listing every peer's internal
|
||||
name and overlay address.
|
||||
**What the host receives:** what to ask, not what to answer. The mesh has **one resolver**, holding
|
||||
every node's internal domain; a node routes the mesh's suffix to it and every other name to public
|
||||
resolvers ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)).
|
||||
|
||||
**What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh
|
||||
database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)
|
||||
@@ -349,6 +351,28 @@ not a list of containers.
|
||||
somebody starts by hand is not the mesh's to configure, and reaching into every container on a
|
||||
machine — declared or not — is what a nameserver in `resolv.conf` would be for.
|
||||
|
||||
### One resolver for the mesh
|
||||
|
||||
*2026-10-03.* **The mesh's names live in one place: the module holding `mesh-resolver`**, a mesh-scoped
|
||||
seat of capacity one, placed on the node every tunnel converges on. It holds one wildcard per node —
|
||||
`<node>.internal` and everything under it — and listens on the private network only. Every node's
|
||||
`node-resolver-config` routes the mesh's suffix to it and leaves every other name with public
|
||||
resolvers; plain `resolv.conf` cannot route by domain, so the asking side is a stub that can
|
||||
(systemd-resolved, the tunnel's link carrying the suffix as its routing domain). The container runtime
|
||||
cannot use a loopback stub, so its `dns` names `mesh-resolver`, which forwards public names for
|
||||
containers — the one place a public name passes through the mesh
|
||||
([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)).
|
||||
|
||||
**No node holds a copy.** The per-node resolver, its zones file and the mesh's region of `/etc/hosts`
|
||||
go: every resolution fault found on 2026-10-03 was a copy disagreeing with the truth — a hosts file
|
||||
read once at start, an operator's old line beside the mesh's, a node's resolver lent to a LAN. No
|
||||
member's resolver answers a LAN; a router pointing at one is moved first. *Checked by `resolvectl` on
|
||||
each node (the tunnel's link, `mesh-resolver`, the suffix as routing domain), by no node but the
|
||||
holder listening on port 53, and by the router's DHCP DNS option naming no node.*
|
||||
|
||||
*What follows describes the per-node resolver this replaces — how it was built and why the roles were
|
||||
split. The split stands; the serving role's scope is what moved.*
|
||||
|
||||
### The resolver, built
|
||||
|
||||
*2026-08-31.* **A service is reached at `<service>.<node>.internal`** — the first label is the
|
||||
@@ -988,6 +1012,9 @@ The list is worth having in one place, because it is most of the argument:
|
||||
|
||||
## Open
|
||||
|
||||
- **One resolver ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)).** Not built: every node still runs
|
||||
`node-dns-resolver`. The migration's four steps are in the record, in order.
|
||||
|
||||
- ~~**What happens when the hub is down.**~~ **Resolved** by
|
||||
[ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), together with `06`'s
|
||||
matching question — they were one question. Nothing takes over. WireGuard has no failover, the
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
layer: to-be
|
||||
status: implemented
|
||||
status: in-progress
|
||||
code:
|
||||
- mesh-controller internal/catalogue/seats.go
|
||||
- mesh-controller internal/catalogue/resolve.go
|
||||
@@ -10,8 +10,9 @@ code:
|
||||
- mesh-controller cmd/mesh-controller/source.go
|
||||
- mesh-controller internal/inventory/migrations/0032-a-source-may-live-on-a-seat.sql
|
||||
- mesh-catalog modules/gitea/module.json
|
||||
updated: 2026-10-01
|
||||
updated: 2026-10-03
|
||||
decisions:
|
||||
- 02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md
|
||||
- 02-DECISIONS/0161-what-deserves-a-seat.md
|
||||
- 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
|
||||
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
|
||||
@@ -126,7 +127,8 @@ convention, which later seats departed from.
|
||||
| `mesh-npm-package-registry` | `npm-package-registry` | mesh | `npm-package-registry` | the forge |
|
||||
| `mesh-git` | `git` | mesh | `git` | the forge |
|
||||
| `mesh-build-machine` | `the-build-machine` | node | — | a builder |
|
||||
| `mesh-dns-port` | `the-dns-port` | node | — | the local resolver |
|
||||
| `mesh-resolver` | — | mesh | — | the mesh's one resolver, holding every node's internal domain ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)) |
|
||||
| ~~`mesh-dns-port`~~ | `the-dns-port` | node | — | retired by [ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md): the local resolver became the mesh's one |
|
||||
| `mesh-intrusion-prevention` | `the-intrusion-prevention` | node | — | an intrusion-prevention service |
|
||||
| `mesh-packet-filter` | `the-packet-filter` | node | — | the packet filter |
|
||||
| `mesh-private-network` | `the-private-network` | node | — | the private network the mesh runs over |
|
||||
|
||||
Reference in New Issue
Block a user