2026-10-03 19:51:04 +00:00
5 changed files with 191 additions and 5 deletions
@@ -9,6 +9,12 @@ extends: 0110-a-seat-is-a-module-assignment-from-a-closed-set.md

 # 121. A system seat is named for its scope, and a module may define its own

+> **Narrowed, not replaced — 2026-10-03.** *"`the-dns-port` → `node-dns-resolver`"* no longer holds:
+> the serving role moves to mesh scope as `mesh-resolver`, one per mesh, and `node-dns-resolver` is
+> retired ([ADR 0194](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)). The
+> distinction this record kept — serving and asking are two roles, two seats — stands, and
+> `node-resolver-config` is unchanged.
+
 > **The mechanism changed — 2026-10-02, by [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md).** The naming rule stands. The build role this record made mesh-scoped — *the mesh's single build machine* — is node-scoped now: `node-build-agent`, one holder per machine, every holder taking from one work queue.

 ## Context
@@ -0,0 +1,149 @@
+---
+topic: the tiers
+status: accepted
+date: 2026-10-03
+deciders: jochen
+reconstructed: false
+supersedes-in-part:
+  - 0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md
+extends: 0191-the-meshs-resolver-holds-only-the-meshs-own-names.md
+---
+
+# 194. The mesh has one resolver, and every node asks it for the mesh's names
+
+## Context
+
+**Every node runs its own resolver and holds its own copy of the mesh's names.** On the production
+mesh on 2026-10-03, each of the four nodes held `node-dns-resolver` with dnsmasq, fed on every push
+with a zones file (one wildcard per node) and a region of `/etc/hosts` (the machines), and pointed
+its own `/etc/resolv.conf` at itself. The controller computes the names once; four daemons then hold
+four copies, each read in its own way.
+
+**Every resolution fault found that day was a copy disagreeing with the truth, not the truth being
+wrong:**
+
+- **A copy read once.** dnsmasq reads `/etc/hosts` at start. After the controller stopped publishing
+  public names ([ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md)), every node's
+  hosts file was right and every resolver still answered the mail server's public name with a
+  tunnel address, until each was restarted.
+- **A copy beside other copies.** On the workstation, a name resolved to two addresses in rotation:
+  the mesh's region gave the tunnel address, and two lines the operator had written before the mesh
+  existed — one in `/etc/hosts`, one in a file the resolver also reads — gave the LAN address. A
+  comment beside one of them said to delete it once the mesh took over; nothing made that happen.
+- **A copy that became somebody else's resolver.** The home server's resolver also answers its LAN
+  (a listen address added 2026-10-02), and the LAN's router hands that address out as the only DNS
+  server. Every phone and television on the LAN resolved through a mesh node's private copy, which is
+  how ADR 0191's outage reached them.
+
+**And the overlay already has one centre.** Every node has exactly one tunnel peer — the anchor —
+and routes the whole private range through it. Two nodes on the same LAN reach each other through
+the anchor. So a name under `.internal` is only ever useful while the anchor is reachable: a resolver
+anywhere else adds a copy without adding an answer anybody can use.
+
+**What a node asks is already a separate role.** The connectivity design split *serving* (answers
+the names) from *asking* (decides what the machine asks), because systemd-resolved cannot answer a
+wildcard and can only route the mesh's suffix to something that can
+([connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md)). ADR 0121 kept them as two seats,
+`node-dns-resolver` and `node-resolver-config`, both at node scope. No node runs systemd-resolved
+today; each writes `/etc/resolv.conf` as a plain file pointing at its own dnsmasq.
+
+## Considered Options
+
+**1. Keep a resolver on every node, and make the copies more careful.** Restart on every file it
+reads, own every file it reads, refuse to listen on a LAN. Each is a fix for one way a copy goes
+stale, and the next way is not on the list yet. It keeps four answers to one question.
+
+**2. One resolver for the mesh, and every node sends it every query.** The simplest asking side —
+`resolv.conf` names the mesh's resolver and nothing else. Rejected: public resolution then depends on
+the tunnel. A laptop whose tunnel is down could resolve nothing at all, and a public name would take
+a detour through the anchor for no reason ADR 0191 left standing.
+
+**3. One resolver for the mesh's names; each node asks it for those only.** The mesh's resolver holds
+every node's internal domain. Each node's asking role routes the mesh's suffix to it and every other
+name to public resolvers. Chosen.
+
+## Decision
+
+**The mesh has one resolver.** It is a module holding a new mesh-scoped seat, **`mesh-resolver`**
+(capacity one). It holds each node's internal domain — `<node>.internal` and everything under it, at
+that node's private address ([ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md))
+— and listens on the private network only. It is placed on the node every tunnel converges on, so
+that it shares the overlay's single point rather than adding one. Which daemon fills the seat stays
+the module's business, as the connectivity design says.
+
+**Every node asks it for the mesh's names and nothing else.** `node-resolver-config` routes the
+mesh's suffix to `mesh-resolver` and leaves every other name with public resolvers.
+
+**Why a stub on every node.** `resolv.conf` cannot route by domain: the C library asks the servers it
+lists in order, for every name, and moves to the next only when one does not answer — an NXDOMAIN
+from the first is final. Listing `mesh-resolver` first sends every public name through the tunnel
+(option 2); listing a public resolver first means `.internal` is never asked of the mesh. Something on
+the node has to look at the name before choosing a server, and that is a stub resolver. Keeping
+dnsmasq for it would keep a daemon that reads hosts files and can be told to answer a LAN — the two
+ways copies went wrong. systemd-resolved holds no names of its own, routes by domain natively (a
+routing domain `~<suffix>` on the server that answers it), and is part of systemd, already installed
+on every node and enabled on none.
+
+**So the asking side is a `systemd-resolved` module**, claiming `node-resolver-config` — the same claim
+as the `resolv-conf` module it replaces, so the mesh refuses both on one node. It enables the service,
+writes its configuration (the mesh resolver for the suffix, public resolvers for everything else), and
+writes `/etc/resolv.conf` as a file naming the stub — a file the module owns, not a link to one.
+
+**A container asks the mesh's resolver directly.** The container runtime cannot use a loopback stub
+and drops its routing domains, so the runtime's `dns` names `mesh-resolver`, which forwards public
+names for the containers that ask it. This is the one place a public name passes through the mesh,
+and it is stated rather than hidden.
+
+**`node-dns-resolver` is retired**, and with it every per-node copy: the zones file, the mesh's region
+of `/etc/hosts` (the floor connectivity §2 already planned to remove), and the daemon on every node
+but the one holding `mesh-resolver`. This narrows ADR 0121's *"the-dns-port → node-dns-resolver"*:
+the serving role keeps its distinction from the asking role and moves to mesh scope, as ADR 0121 did
+for the private network.
+
+**A LAN's resolver is not the mesh's.** No device that is not a member can reach a private address,
+so no member's resolver answers a LAN on the mesh's behalf. A router that hands out a node's address
+as a LAN's DNS server is pointed elsewhere before that node stops answering.
+
+**The order is fixed, because every step before the last leaves a working resolver:**
+
+1. `mesh-resolver` is assigned and answers on the private network.
+2. Each node's `node-resolver-config` moves from `resolv-conf` to `systemd-resolved`, and the container
+   runtime's `dns` to `mesh-resolver`.
+3. A LAN whose router points at a node's resolver is pointed at its router or a public resolver.
+4. `node-dns-resolver` is unassigned from every node, and the hosts region is withdrawn.
+
+## Consequences
+
+- **One answer per name.** A name is wrong in one place or right everywhere; no node can hold a copy
+  that disagrees, and no operator file on a node is read by the mesh's resolver.
+- **The anchor down means no `.internal` names** — which it already meant for `.internal` traffic,
+  since every tunnel goes through it. Public resolution on every node is unaffected.
+- **A container's public resolution depends on the mesh's resolver.** Accepted, and named in the
+  decision; a container that must resolve public names with the tunnel down is the case it costs.
+- **Every node runs systemd-resolved**, through the `systemd-resolved` module. It is installed
+  everywhere already and enabled nowhere; the mesh still ships no resolver of its own.
+- **The runtime's `dns` changes once per node**, which the runtime reads only at start. With
+  `live-restore` already on, that restart keeps every container running.
+- **A LAN loses a resolver it had borrowed.** The router change is an explicit step, done through
+  the module that manages the router, before the node's resolver goes.
+
+**How each is checked:**
+
+- **One holder:** the seat has capacity one, so a second assignment is refused by the controller.
+- **Asking:** on each node, `resolvectl` shows the tunnel's link with `mesh-resolver` and the suffix as
+  its routing domain; a name under `.internal` is answered by it, and a public name is answered
+  without it (its query log shows no public name from a node).
+- **No copies:** no node but the holder answers DNS on a private or LAN address — every other node's
+  port 53 is systemd-resolved's loopback stub and nothing else — and no node's `/etc/hosts` carries a
+  mesh region.
+- **A LAN:** the router's DHCP DNS option names no node's address.
+
+## References
+
+- [ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md) — what the mesh's resolver
+  holds; this record decides where it runs and how nodes reach it.
+- [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) — the two
+  resolver seats, and the private network's move to mesh scope this mirrors.
+- [Connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md) — serving and asking as two roles;
+  amended alongside this record.
+- [The seats](../03-DESIGN/01-to-be/26-the-seats.md) — the seat table, amended alongside.
@@ -223,6 +223,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0148** — [The mesh's names are resolved, not copied into every container](0148-the-meshs-names-are-resolved-not-copied-into-containers.md)
 - **0151** — [A route's internal name is composed under the node that serves it](0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md)
 - **0191** — [The mesh's resolver holds only the mesh's own names; a public name resolves publicly](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md)
+- **0194** — [The mesh has one resolver, and every node asks it for the mesh's names](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)

 ### What runs on them, and how it gets there

@@ -10,6 +10,7 @@ code:
  - mesh-host internal/apply (the service that reflects a rule set)
 updated: 2026-10-03
 decisions:
+  - 02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md
  - 02-DECISIONS/0191-the-meshs-resolver-holds-only-the-meshs-own-names.md
  - 02-DECISIONS/0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md
  - 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md
@@ -290,8 +291,9 @@ expensively enough to be worth restating:
 - **A node must not pin its own public name locally.** The duplicate record breaks resolution of
  that name for everything else that needs it.

-**What the host receives:** the resolver's configuration, as files, listing every peer's internal
-name and overlay address.
+**What the host receives:** what to ask, not what to answer. The mesh has **one resolver**, holding
+every node's internal domain; a node routes the mesh's suffix to it and every other name to public
+resolvers ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)).

 **What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh
 database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)
@@ -349,6 +351,29 @@ not a list of containers.
 somebody starts by hand is not the mesh's to configure, and reaching into every container on a
 machine — declared or not — is what a nameserver in `resolv.conf` would be for.

+### One resolver for the mesh
+
+*2026-10-03.* **The mesh's names live in one place: the module holding `mesh-resolver`**, a mesh-scoped
+seat of capacity one, placed on the node every tunnel converges on. It holds one wildcard per node —
+`<node>.internal` and everything under it — and listens on the private network only. Every node's
+`node-resolver-config` routes the mesh's suffix to it and leaves every other name with public
+resolvers; plain `resolv.conf` cannot route by domain, so the asking side is a stub that can — a
+`systemd-resolved` module claiming `node-resolver-config` in place of `resolv-conf`, routing the
+suffix to `mesh-resolver`. The container runtime cannot use a loopback stub, so its `dns` names
+`mesh-resolver`, which forwards public names for containers — the one place a public name passes
+through the mesh
+([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)).
+
+**No node holds a copy.** The per-node resolver, its zones file and the mesh's region of `/etc/hosts`
+go: every resolution fault found on 2026-10-03 was a copy disagreeing with the truth — a hosts file
+read once at start, an operator's old line beside the mesh's, a node's resolver lent to a LAN. No
+member's resolver answers a LAN; a router pointing at one is moved first. *Checked by `resolvectl` on
+each node (the tunnel's link, `mesh-resolver`, the suffix as routing domain), by no node but the
+holder answering DNS on a private or LAN address, and by the router's DHCP DNS option naming no node.*
+
+*What follows describes the per-node resolver this replaces — how it was built and why the roles were
+split. The split stands; the serving role's scope is what moved.*
+
 ### The resolver, built

 *2026-08-31.* **A service is reached at `<service>.<node>.internal`** — the first label is the
@@ -988,6 +1013,9 @@ The list is worth having in one place, because it is most of the argument:

 ## Open

+- **One resolver ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)).** Not built: every node still runs
+  `node-dns-resolver`. The migration's four steps are in the record, in order.
+
 - ~~**What happens when the hub is down.**~~ **Resolved** by
  [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), together with `06`'s
  matching question — they were one question. Nothing takes over. WireGuard has no failover, the
@@ -1,6 +1,6 @@
 ---
 layer: to-be
-status: implemented
+status: in-progress
 code:
  - mesh-controller internal/catalogue/seats.go
  - mesh-controller internal/catalogue/resolve.go
@@ -10,8 +10,9 @@ code:
  - mesh-controller cmd/mesh-controller/source.go
  - mesh-controller internal/inventory/migrations/0032-a-source-may-live-on-a-seat.sql
  - mesh-catalog modules/gitea/module.json
-updated: 2026-10-01
+updated: 2026-10-03
 decisions:
+  - 02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md
  - 02-DECISIONS/0161-what-deserves-a-seat.md
  - 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md
  - 02-DECISIONS/0126-a-module-declares-its-own-seats.md
@@ -126,7 +127,8 @@ convention, which later seats departed from.
 | `mesh-npm-package-registry` | `npm-package-registry` | mesh | `npm-package-registry` | the forge |
 | `mesh-git` | `git` | mesh | `git` | the forge |
 | `mesh-build-machine` | `the-build-machine` | node | — | a builder |
-| `mesh-dns-port` | `the-dns-port` | node | — | the local resolver |
+| `mesh-resolver` | — | mesh | — | the mesh's one resolver, holding every node's internal domain ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)) |
+| ~~`mesh-dns-port`~~ | `the-dns-port` | node | — | retired by [ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md): the local resolver became the mesh's one |
 | `mesh-intrusion-prevention` | `the-intrusion-prevention` | node | — | an intrusion-prevention service |
 | `mesh-packet-filter` | `the-packet-filter` | node | — | the packet filter |
 | `mesh-private-network` | `the-private-network` | node | — | the private network the mesh runs over |