diff --git a/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md b/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md index 3223921..9f32bfa 100644 --- a/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md +++ b/02-DECISIONS/0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md @@ -9,6 +9,12 @@ extends: 0110-a-seat-is-a-module-assignment-from-a-closed-set.md # 121. A system seat is named for its scope, and a module may define its own +> **Narrowed, not replaced — 2026-10-03.** *"`the-dns-port` → `node-dns-resolver`"* no longer holds: +> the serving role moves to mesh scope as `mesh-resolver`, one per mesh, and `node-dns-resolver` is +> retired ([ADR 0194](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)). The +> distinction this record kept — serving and asking are two roles, two seats — stands, and +> `node-resolver-config` is unchanged. + > **The mechanism changed — 2026-10-02, by [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md).** The naming rule stands. The build role this record made mesh-scoped — *the mesh's single build machine* — is node-scoped now: `node-build-agent`, one holder per machine, every holder taking from one work queue. ## Context diff --git a/02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md b/02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md new file mode 100644 index 0000000..08c5935 --- /dev/null +++ b/02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md @@ -0,0 +1,149 @@ +--- +topic: the tiers +status: accepted +date: 2026-10-03 +deciders: jochen +reconstructed: false +supersedes-in-part: + - 0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md +extends: 0191-the-meshs-resolver-holds-only-the-meshs-own-names.md +--- + +# 194. The mesh has one resolver, and every node asks it for the mesh's names + +## Context + +**Every node runs its own resolver and holds its own copy of the mesh's names.** On the production +mesh on 2026-10-03, each of the four nodes held `node-dns-resolver` with dnsmasq, fed on every push +with a zones file (one wildcard per node) and a region of `/etc/hosts` (the machines), and pointed +its own `/etc/resolv.conf` at itself. The controller computes the names once; four daemons then hold +four copies, each read in its own way. + +**Every resolution fault found that day was a copy disagreeing with the truth, not the truth being +wrong:** + +- **A copy read once.** dnsmasq reads `/etc/hosts` at start. After the controller stopped publishing + public names ([ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md)), every node's + hosts file was right and every resolver still answered the mail server's public name with a + tunnel address, until each was restarted. +- **A copy beside other copies.** On the workstation, a name resolved to two addresses in rotation: + the mesh's region gave the tunnel address, and two lines the operator had written before the mesh + existed — one in `/etc/hosts`, one in a file the resolver also reads — gave the LAN address. A + comment beside one of them said to delete it once the mesh took over; nothing made that happen. +- **A copy that became somebody else's resolver.** The home server's resolver also answers its LAN + (a listen address added 2026-10-02), and the LAN's router hands that address out as the only DNS + server. Every phone and television on the LAN resolved through a mesh node's private copy, which is + how ADR 0191's outage reached them. + +**And the overlay already has one centre.** Every node has exactly one tunnel peer — the anchor — +and routes the whole private range through it. Two nodes on the same LAN reach each other through +the anchor. So a name under `.internal` is only ever useful while the anchor is reachable: a resolver +anywhere else adds a copy without adding an answer anybody can use. + +**What a node asks is already a separate role.** The connectivity design split *serving* (answers +the names) from *asking* (decides what the machine asks), because systemd-resolved cannot answer a +wildcard and can only route the mesh's suffix to something that can +([connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md)). ADR 0121 kept them as two seats, +`node-dns-resolver` and `node-resolver-config`, both at node scope. No node runs systemd-resolved +today; each writes `/etc/resolv.conf` as a plain file pointing at its own dnsmasq. + +## Considered Options + +**1. Keep a resolver on every node, and make the copies more careful.** Restart on every file it +reads, own every file it reads, refuse to listen on a LAN. Each is a fix for one way a copy goes +stale, and the next way is not on the list yet. It keeps four answers to one question. + +**2. One resolver for the mesh, and every node sends it every query.** The simplest asking side — +`resolv.conf` names the mesh's resolver and nothing else. Rejected: public resolution then depends on +the tunnel. A laptop whose tunnel is down could resolve nothing at all, and a public name would take +a detour through the anchor for no reason ADR 0191 left standing. + +**3. One resolver for the mesh's names; each node asks it for those only.** The mesh's resolver holds +every node's internal domain. Each node's asking role routes the mesh's suffix to it and every other +name to public resolvers. Chosen. + +## Decision + +**The mesh has one resolver.** It is a module holding a new mesh-scoped seat, **`mesh-resolver`** +(capacity one). It holds each node's internal domain — `.internal` and everything under it, at +that node's private address ([ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md)) +— and listens on the private network only. It is placed on the node every tunnel converges on, so +that it shares the overlay's single point rather than adding one. Which daemon fills the seat stays +the module's business, as the connectivity design says. + +**Every node asks it for the mesh's names and nothing else.** `node-resolver-config` routes the +mesh's suffix to `mesh-resolver` and leaves every other name with public resolvers. + +**Why a stub on every node.** `resolv.conf` cannot route by domain: the C library asks the servers it +lists in order, for every name, and moves to the next only when one does not answer — an NXDOMAIN +from the first is final. Listing `mesh-resolver` first sends every public name through the tunnel +(option 2); listing a public resolver first means `.internal` is never asked of the mesh. Something on +the node has to look at the name before choosing a server, and that is a stub resolver. Keeping +dnsmasq for it would keep a daemon that reads hosts files and can be told to answer a LAN — the two +ways copies went wrong. systemd-resolved holds no names of its own, routes by domain natively (a +routing domain `~` on the server that answers it), and is part of systemd, already installed +on every node and enabled on none. + +**So the asking side is a `systemd-resolved` module**, claiming `node-resolver-config` — the same claim +as the `resolv-conf` module it replaces, so the mesh refuses both on one node. It enables the service, +writes its configuration (the mesh resolver for the suffix, public resolvers for everything else), and +writes `/etc/resolv.conf` as a file naming the stub — a file the module owns, not a link to one. + +**A container asks the mesh's resolver directly.** The container runtime cannot use a loopback stub +and drops its routing domains, so the runtime's `dns` names `mesh-resolver`, which forwards public +names for the containers that ask it. This is the one place a public name passes through the mesh, +and it is stated rather than hidden. + +**`node-dns-resolver` is retired**, and with it every per-node copy: the zones file, the mesh's region +of `/etc/hosts` (the floor connectivity §2 already planned to remove), and the daemon on every node +but the one holding `mesh-resolver`. This narrows ADR 0121's *"the-dns-port → node-dns-resolver"*: +the serving role keeps its distinction from the asking role and moves to mesh scope, as ADR 0121 did +for the private network. + +**A LAN's resolver is not the mesh's.** No device that is not a member can reach a private address, +so no member's resolver answers a LAN on the mesh's behalf. A router that hands out a node's address +as a LAN's DNS server is pointed elsewhere before that node stops answering. + +**The order is fixed, because every step before the last leaves a working resolver:** + +1. `mesh-resolver` is assigned and answers on the private network. +2. Each node's `node-resolver-config` moves from `resolv-conf` to `systemd-resolved`, and the container + runtime's `dns` to `mesh-resolver`. +3. A LAN whose router points at a node's resolver is pointed at its router or a public resolver. +4. `node-dns-resolver` is unassigned from every node, and the hosts region is withdrawn. + +## Consequences + +- **One answer per name.** A name is wrong in one place or right everywhere; no node can hold a copy + that disagrees, and no operator file on a node is read by the mesh's resolver. +- **The anchor down means no `.internal` names** — which it already meant for `.internal` traffic, + since every tunnel goes through it. Public resolution on every node is unaffected. +- **A container's public resolution depends on the mesh's resolver.** Accepted, and named in the + decision; a container that must resolve public names with the tunnel down is the case it costs. +- **Every node runs systemd-resolved**, through the `systemd-resolved` module. It is installed + everywhere already and enabled nowhere; the mesh still ships no resolver of its own. +- **The runtime's `dns` changes once per node**, which the runtime reads only at start. With + `live-restore` already on, that restart keeps every container running. +- **A LAN loses a resolver it had borrowed.** The router change is an explicit step, done through + the module that manages the router, before the node's resolver goes. + +**How each is checked:** + +- **One holder:** the seat has capacity one, so a second assignment is refused by the controller. +- **Asking:** on each node, `resolvectl` shows the tunnel's link with `mesh-resolver` and the suffix as + its routing domain; a name under `.internal` is answered by it, and a public name is answered + without it (its query log shows no public name from a node). +- **No copies:** no node but the holder answers DNS on a private or LAN address — every other node's + port 53 is systemd-resolved's loopback stub and nothing else — and no node's `/etc/hosts` carries a + mesh region. +- **A LAN:** the router's DHCP DNS option names no node's address. + +## References + +- [ADR 0191](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md) — what the mesh's resolver + holds; this record decides where it runs and how nodes reach it. +- [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md) — the two + resolver seats, and the private network's move to mesh scope this mirrors. +- [Connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md) — serving and asking as two roles; + amended alongside this record. +- [The seats](../03-DESIGN/01-to-be/26-the-seats.md) — the seat table, amended alongside. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 1b643d4..af63814 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -223,6 +223,7 @@ python3 00-META/checks/index.py fail if stale - **0148** — [The mesh's names are resolved, not copied into every container](0148-the-meshs-names-are-resolved-not-copied-into-containers.md) - **0151** — [A route's internal name is composed under the node that serves it](0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md) - **0191** — [The mesh's resolver holds only the mesh's own names; a public name resolves publicly](0191-the-meshs-resolver-holds-only-the-meshs-own-names.md) +- **0194** — [The mesh has one resolver, and every node asks it for the mesh's names](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md) ### What runs on them, and how it gets there diff --git a/03-DESIGN/01-to-be/08-connectivity.md b/03-DESIGN/01-to-be/08-connectivity.md index 9b0ae5d..8249359 100644 --- a/03-DESIGN/01-to-be/08-connectivity.md +++ b/03-DESIGN/01-to-be/08-connectivity.md @@ -10,6 +10,7 @@ code: - mesh-host internal/apply (the service that reflects a rule set) updated: 2026-10-03 decisions: + - 02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md - 02-DECISIONS/0191-the-meshs-resolver-holds-only-the-meshs-own-names.md - 02-DECISIONS/0180-the-found-front-end-is-uninstalled-once-a-machine-is-converged.md - 02-DECISIONS/0170-the-firewall-seat-serves-its-verbs.md @@ -290,8 +291,9 @@ expensively enough to be worth restating: - **A node must not pin its own public name locally.** The duplicate record breaks resolution of that name for everything else that needs it. -**What the host receives:** the resolver's configuration, as files, listing every peer's internal -name and overlay address. +**What the host receives:** what to ask, not what to answer. The mesh has **one resolver**, holding +every node's internal domain; a node routes the mesh's suffix to it and every other name to public +resolvers ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)). **What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) @@ -349,6 +351,29 @@ not a list of containers. somebody starts by hand is not the mesh's to configure, and reaching into every container on a machine — declared or not — is what a nameserver in `resolv.conf` would be for. +### One resolver for the mesh + +*2026-10-03.* **The mesh's names live in one place: the module holding `mesh-resolver`**, a mesh-scoped +seat of capacity one, placed on the node every tunnel converges on. It holds one wildcard per node — +`.internal` and everything under it — and listens on the private network only. Every node's +`node-resolver-config` routes the mesh's suffix to it and leaves every other name with public +resolvers; plain `resolv.conf` cannot route by domain, so the asking side is a stub that can — a +`systemd-resolved` module claiming `node-resolver-config` in place of `resolv-conf`, routing the +suffix to `mesh-resolver`. The container runtime cannot use a loopback stub, so its `dns` names +`mesh-resolver`, which forwards public names for containers — the one place a public name passes +through the mesh +([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)). + +**No node holds a copy.** The per-node resolver, its zones file and the mesh's region of `/etc/hosts` +go: every resolution fault found on 2026-10-03 was a copy disagreeing with the truth — a hosts file +read once at start, an operator's old line beside the mesh's, a node's resolver lent to a LAN. No +member's resolver answers a LAN; a router pointing at one is moved first. *Checked by `resolvectl` on +each node (the tunnel's link, `mesh-resolver`, the suffix as routing domain), by no node but the +holder answering DNS on a private or LAN address, and by the router's DHCP DNS option naming no node.* + +*What follows describes the per-node resolver this replaces — how it was built and why the roles were +split. The split stands; the serving role's scope is what moved.* + ### The resolver, built *2026-08-31.* **A service is reached at `..internal`** — the first label is the @@ -988,6 +1013,9 @@ The list is worth having in one place, because it is most of the argument: ## Open +- **One resolver ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)).** Not built: every node still runs + `node-dns-resolver`. The migration's four steps are in the record, in order. + - ~~**What happens when the hub is down.**~~ **Resolved** by [ADR 0006](../../02-DECISIONS/0006-the-substrate-and-the-control-plane.md), together with `06`'s matching question — they were one question. Nothing takes over. WireGuard has no failover, the diff --git a/03-DESIGN/01-to-be/26-the-seats.md b/03-DESIGN/01-to-be/26-the-seats.md index 37804e7..648e48f 100644 --- a/03-DESIGN/01-to-be/26-the-seats.md +++ b/03-DESIGN/01-to-be/26-the-seats.md @@ -1,6 +1,6 @@ --- layer: to-be -status: implemented +status: in-progress code: - mesh-controller internal/catalogue/seats.go - mesh-controller internal/catalogue/resolve.go @@ -10,8 +10,9 @@ code: - mesh-controller cmd/mesh-controller/source.go - mesh-controller internal/inventory/migrations/0032-a-source-may-live-on-a-seat.sql - mesh-catalog modules/gitea/module.json -updated: 2026-10-01 +updated: 2026-10-03 decisions: + - 02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md - 02-DECISIONS/0161-what-deserves-a-seat.md - 02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md - 02-DECISIONS/0126-a-module-declares-its-own-seats.md @@ -126,7 +127,8 @@ convention, which later seats departed from. | `mesh-npm-package-registry` | `npm-package-registry` | mesh | `npm-package-registry` | the forge | | `mesh-git` | `git` | mesh | `git` | the forge | | `mesh-build-machine` | `the-build-machine` | node | — | a builder | -| `mesh-dns-port` | `the-dns-port` | node | — | the local resolver | +| `mesh-resolver` | — | mesh | — | the mesh's one resolver, holding every node's internal domain ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)) | +| ~~`mesh-dns-port`~~ | `the-dns-port` | node | — | retired by [ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md): the local resolver became the mesh's one | | `mesh-intrusion-prevention` | `the-intrusion-prevention` | node | — | an intrusion-prevention service | | `mesh-packet-filter` | `the-packet-filter` | node | — | the packet filter | | `mesh-private-network` | `the-private-network` | node | — | the private network the mesh runs over |