Every resolution fault found on 2026-10-03 was a per-node copy disagreeing with the truth: a hosts file read once, an operator's old line beside the mesh's, a node's resolver lent to a LAN. Every tunnel already converges on one node. Retires node-dns-resolver for a mesh-scoped mesh-resolver; nodes route only the mesh's suffix to it. Narrows 0121; amends connectivity §2 and the seats.
8.4 KiB
topic, status, date, deciders, reconstructed, supersedes-in-part, extends
| topic | status | date | deciders | reconstructed | supersedes-in-part | extends | |
|---|---|---|---|---|---|---|---|
| the tiers | accepted | 2026-10-03 | jochen | false |
|
0191-the-meshs-resolver-holds-only-the-meshs-own-names.md |
194. The mesh has one resolver, and every node asks it for the mesh's names
Context
Every node runs its own resolver and holds its own copy of the mesh's names. On the production
mesh on 2026-10-03, each of the four nodes held node-dns-resolver with dnsmasq, fed on every push
with a zones file (one wildcard per node) and a region of /etc/hosts (the machines), and pointed
its own /etc/resolv.conf at itself. The controller computes the names once; four daemons then hold
four copies, each read in its own way.
Every resolution fault found that day was a copy disagreeing with the truth, not the truth being wrong:
- A copy read once. dnsmasq reads
/etc/hostsat start. After the controller stopped publishing public names (ADR 0191), every node's hosts file was right and every resolver still answered the mail server's public name with a tunnel address, until each was restarted. - A copy beside other copies. On the workstation, a name resolved to two addresses in rotation:
the mesh's region gave the tunnel address, and two lines the operator had written before the mesh
existed — one in
/etc/hosts, one in a file the resolver also reads — gave the LAN address. A comment beside one of them said to delete it once the mesh took over; nothing made that happen. - A copy that became somebody else's resolver. The home server's resolver also answers its LAN (a listen address added 2026-10-02), and the LAN's router hands that address out as the only DNS server. Every phone and television on the LAN resolved through a mesh node's private copy, which is how ADR 0191's outage reached them.
And the overlay already has one centre. Every node has exactly one tunnel peer — the anchor —
and routes the whole private range through it. Two nodes on the same LAN reach each other through
the anchor. So a name under .internal is only ever useful while the anchor is reachable: a resolver
anywhere else adds a copy without adding an answer anybody can use.
What a node asks is already a separate role. The connectivity design split serving (answers
the names) from asking (decides what the machine asks), because systemd-resolved cannot answer a
wildcard and can only route the mesh's suffix to something that can
(connectivity §2). ADR 0121 kept them as two seats,
node-dns-resolver and node-resolver-config, both at node scope. No node runs systemd-resolved
today; each writes /etc/resolv.conf as a plain file pointing at its own dnsmasq.
Considered Options
1. Keep a resolver on every node, and make the copies more careful. Restart on every file it reads, own every file it reads, refuse to listen on a LAN. Each is a fix for one way a copy goes stale, and the next way is not on the list yet. It keeps four answers to one question.
2. One resolver for the mesh, and every node sends it every query. The simplest asking side —
resolv.conf names the mesh's resolver and nothing else. Rejected: public resolution then depends on
the tunnel. A laptop whose tunnel is down could resolve nothing at all, and a public name would take
a detour through the anchor for no reason ADR 0191 left standing.
3. One resolver for the mesh's names; each node asks it for those only. The mesh's resolver holds every node's internal domain. Each node's asking role routes the mesh's suffix to it and every other name to public resolvers. Chosen.
Decision
The mesh has one resolver. It is a module holding a new mesh-scoped seat, mesh-resolver
(capacity one). It holds each node's internal domain — <node>.internal and everything under it, at
that node's private address (ADR 0191)
— and listens on the private network only. It is placed on the node every tunnel converges on, so
that it shares the overlay's single point rather than adding one. Which daemon fills the seat stays
the module's business, as the connectivity design says.
Every node asks it for the mesh's names and nothing else. node-resolver-config routes the
mesh's suffix to mesh-resolver and leaves every other name with public resolvers. Plain
resolv.conf cannot route by domain, so the asking side is a stub that can — systemd-resolved, with
the tunnel's link carrying the mesh resolver and the suffix as its routing domain. The stub holds no
names; it decides only where a question goes.
A container asks the mesh's resolver directly. The container runtime cannot use a loopback stub
and drops its routing domains, so the runtime's dns names mesh-resolver, which forwards public
names for the containers that ask it. This is the one place a public name passes through the mesh,
and it is stated rather than hidden.
node-dns-resolver is retired, and with it every per-node copy: the zones file, the mesh's region
of /etc/hosts (the floor connectivity §2 already planned to remove), and the daemon on every node
but the one holding mesh-resolver. This narrows ADR 0121's "the-dns-port → node-dns-resolver":
the serving role keeps its distinction from the asking role and moves to mesh scope, as ADR 0121 did
for the private network.
A LAN's resolver is not the mesh's. No device that is not a member can reach a private address, so no member's resolver answers a LAN on the mesh's behalf. A router that hands out a node's address as a LAN's DNS server is pointed elsewhere before that node stops answering.
The order is fixed, because every step before the last leaves a working resolver:
mesh-resolveris assigned and answers on the private network.- Each node's
node-resolver-configswitches to the routing stub, and the container runtime'sdnstomesh-resolver. - A LAN whose router points at a node's resolver is pointed at its router or a public resolver.
node-dns-resolveris unassigned from every node, and the hosts region is withdrawn.
Consequences
- One answer per name. A name is wrong in one place or right everywhere; no node can hold a copy that disagrees, and no operator file on a node is read by the mesh's resolver.
- The anchor down means no
.internalnames — which it already meant for.internaltraffic, since every tunnel goes through it. Public resolution on every node is unaffected. - A container's public resolution depends on the mesh's resolver. Accepted, and named in the decision; a container that must resolve public names with the tunnel down is the case it costs.
- Every node gains systemd-resolved as its asking stub, which today none runs. It is configured
by the module holding
node-resolver-config; the mesh still ships no resolver of its own. - The runtime's
dnschanges once per node, which the runtime reads only at start. Withlive-restorealready on, that restart keeps every container running. - A LAN loses a resolver it had borrowed. The router change is an explicit step, done through the module that manages the router, before the node's resolver goes.
How each is checked:
- One holder: the seat has capacity one, so a second assignment is refused by the controller.
- Asking: on each node,
resolvectlshows the tunnel's link withmesh-resolverand the suffix as its routing domain; a name under.internalis answered by it, and a public name is answered without it (its query log shows no public name from a node). - No copies: no node but the holder listens on port 53, and no node's
/etc/hostscarries a mesh region. - A LAN: the router's DHCP DNS option names no node's address.
References
- ADR 0191 — what the mesh's resolver holds; this record decides where it runs and how nodes reach it.
- ADR 0121 — the two resolver seats, and the private network's move to mesh scope this mirrors.
- Connectivity §2 — serving and asking as two roles; amended alongside this record.
- The seats — the seat table, amended alongside.