Merge pull request 'ADR 0223: the mesh has two resolvers, and a machine lists only them' (#115) from decision/0223-the-mesh-has-two-resolvers into main

2026-10-05 20:48:09 +00:00
parent 6e59281806 4f1300fd48
commit 178994ee0d
10 changed files with 285 additions and 24 deletions
@@ -11,6 +11,11 @@ extends: 0191-the-meshs-resolver-holds-only-the-meshs-own-names.md

 # 194. The mesh has one resolver, and every node asks it for the mesh's names

+> **The mechanism changed — 2026-10-05, by [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md).** `mesh-resolver` (named `mesh-dns-resolver` in the
+> set) is no longer of capacity one: it is a replicated seat, held on the anchor and on the home
+> server, each holding every node's internal domain from the same roster. One resolving module, the
+> mesh's names held only by its holders, and the retirement of every per-node copy stand.
+
 > **Narrowed, not replaced — 2026-10-03.** How a node asks is decided again by
 > [ADR 0196](0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md): every
 > node and container asks `mesh-resolver` first and a public resolver only when it is silent. There is
@@ -10,6 +10,13 @@ supersedes-in-part:

 # 196. A node asks the mesh's resolver first, and a public one only when it is silent

+> **Narrowed, not replaced — 2026-10-05, by [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md).** The public resolver listed second goes: musl asks
+> every listed server at once and takes the first reply, so a public "no such name" for a mesh name
+> won it, and every Alpine build on the home server failed. Every machine now lists the mesh's
+> resolvers — two holders of `mesh-dns-resolver`, its own first on a holder — and nothing else. Every
+> node and container asking the mesh's resolver for every name, with no stub and no runtime `dns`,
+> stands. The "fallback" consequence and check below describe what this record decided, not what runs.
+
 ## Context

 **[ADR 0194](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md) gave the
@@ -9,6 +9,10 @@ extends: 0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-nam

 # 199. A module that answers names declares its zone, and a node's hosts file is one module's

+> **Decided to change — 2026-10-05, by [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md), not yet built.** The `hosts` module is renamed `hostname`
+> and its seat `node-hostname`, and it owns `/etc/hostname` as well as `/etc/hosts`; the operator's
+> lines are kept as this record decides. Until that is built, everything below stands as decided.
+
 ## Context

 **[ADR 0194](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md) and
@@ -9,6 +9,10 @@ extends: 02-DECISIONS/0207-a-module-depends-on-the-node-seats-that-apply-its-res

 # 220. What a machine asks needs its uplink held, and the retired resolver pieces go

+> **Decided to go — 2026-10-05, by [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md), not yet built.** `/etc/resolv.conf` becomes the `node-uplink`
+> holder's file, and `resolv-conf`, `node-resolver-config` and this record's dependency of it on
+> `node-uplink` retire with it. Until that is built, everything below stands as decided.
+
 ## Context

 **Three things about a machine's resolver were left half done when the mesh moved to one resolver.**
@@ -0,0 +1,193 @@
+---
+topic: the tiers
+status: accepted
+date: 2026-10-05
+deciders: jochen
+reconstructed: false
+supersedes-in-part:
+  - 0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md
+extends: 02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md
+---
+
+# 223. The mesh has two resolvers, and a machine lists only them
+
+## Context
+
+**[ADR 0196](0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md) wrote
+every machine's `/etc/resolv.conf` as the mesh's resolver first and a public resolver second.** Its
+reasoning is the C library's: servers are asked in order, the next only when one does not answer, and
+an answer from the first — "no such name" included — is final. That holds for glibc. It does not hold
+for musl, the C library of every Alpine image: musl sends the query to every listed server at once
+and takes the first reply.
+
+**On the production mesh on 2026-10-05 that made the anchor's own name unresolvable from the home
+server's builds.** The build agent on the home server runs its steps in Alpine containers on the host
+network, so they read the machine's `resolv.conf` as it is. Asked for `<anchor>.internal`, the public
+resolver — which has no such name and is the nearer of the two — answered NXDOMAIN first, and musl
+took it. Reproduced six times out of six inside the build agent's own container; every build on that
+machine failed fetching from the anchor. The same lookup from glibc on the same machine answered every
+time. [Issue 262](../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md)
+was the same library failing on a different answer (NXDOMAIN for a missing IPv6 record); fixing that
+did not touch this one, because here the wrong answer comes from a server that should never have been
+asked.
+
+**The public line was there for one case: the mesh's resolver unreachable.** ADR 0196 kept public
+names resolving with the anchor down, the tunnel down, or a laptop behind a captive portal. That case
+is real and rare; the musl case is every lookup of a mesh name from every Alpine container on any
+machine that is not the anchor.
+
+**A seat's work can already be shared by several holders.** [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md)
+made the build role node-scoped with every holder pulling from one queue. A mesh-scoped seat has
+always had exactly one holder: by derivation, or on record since
+[ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md), where a handover replaces the
+holder in one write and every other eligible assignment stands beside it, silent.
+
+## Considered Options
+
+1. **The mesh's resolver alone.** Every mesh name and every public name answered consistently, by one
+   server. Rejected alone: with the anchor down, no machine resolves anything — the reason ADR 0194
+   gave against it, and still true.
+2. **A local forwarder on every machine** — a small resolver on loopback that asks the mesh's resolver
+   for the mesh's suffix and public resolvers for the rest, with `resolv.conf` naming only it. Rejected:
+   it is ADR 0194's per-node stub again, which ADR 0196 removed — a daemon and a module on every node,
+   and a container cannot use a loopback resolver, so containers would need a second configuration.
+   Every resolution fault found on 2026-10-03 was a per-node copy disagreeing with the truth.
+3. **Split by kind of machine** — servers list the mesh's resolver alone, laptops keep a public
+   fallback. Rejected: a laptop runs Alpine containers too, and a rule that differs by machine is a
+   rule nobody can state about the mesh.
+4. **Two mesh resolvers, and nothing else listed.** The same module, the same roster and the same zones
+   on two machines, and every machine lists both. Whichever answers first gives the same answer, so
+   musl's race is harmless; a public name still resolves through either. Chosen.
+
+## Decision
+
+**1. Now: the mesh has two resolvers.** The seat `mesh-dns-resolver` may be held on more than one
+machine — the anchor and the home server — each holder answering the same mesh names: the same
+module, the same machine list, the same zones, rendered by the controller into each.
+
+- **A seat may be replicated.** It is an attribute of the seat in the mesh's definition, compiled with
+  the set and never stored, as what a seat needs ([ADR 0220](0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md))
+  and what it receives are. `mesh-dns-resolver` is the only replicated seat. Making another one is a
+  decision, recorded. In the glossary's words it is the mesh's first **bench** — a seat whose
+  holders coexist — and *replicated* says what kind: every holder answers the same thing.
+- **Every holder is on record, and each is added by an act**: `seat mesh-dns-resolver --add
+  <node>/<module>` records one more holder beside those on record. Assigning the module is not enough:
+  an assignment not on record stands beside the holders, eligible and silent, exactly as ADR 0131 says,
+  and two claimants with nothing on record are refused as for any mesh seat. `--to` still hands the
+  seat over, leaving exactly one holder. `--add` on a seat held once is refused, naming `--to`.
+- **One per machine still**: two modules on one machine claiming it are refused. And a seat held once
+  stays held once: a second claimant on another machine is refused while nothing is on record, and a
+  store recording two holders of such a seat is refused, naming the seat.
+- **Every machine's `/etc/resolv.conf` lists every holder's private address, the holder on the machine
+  itself first if it is one, then the rest in name order, and no public resolver.** The controller
+  gives a module's roster template the holders of each replicated seat, ordered so; the module holding
+  `node-resolver-config` writes the file from it. On a holder, its own private address is first: the
+  resolver listens there and on loopback, and the private address is the one a container on that
+  machine can reach.
+- **The requirement stays.** What writes the file still requires `wildcard-resolution`, so a machine is
+  refused when nothing in the mesh resolves, rather than given a file listing nothing. A holder answers
+  its own requirement; any other machine is bound to the first holder by name. Nothing reads that
+  binding's address any more — the file lists every holder — and the binding is kept for the refusal
+  and the order of delivery.
+
+**2. Next, decided and not yet built: `/etc/resolv.conf` belongs to the uplink's holder.** The program
+that manages the machine's network already has to be told to keep off the file
+([ADR 0117](0117-a-machines-uplink-is-a-seat.md)); instead, the `node-uplink` holder writes it, given
+the resolvers by the mesh: NetworkManager through its global DNS configuration, dhcpcd through static
+nameservers, and systemd-networkd's module declaring the file itself. `resolv-conf` and the seat
+`node-resolver-config` then retire, and ADR 0220's dependency of `node-resolver-config` on `node-uplink`
+goes with them. One owner for the file, and it is the program that would otherwise rewrite it.
+
+**3. Next, decided and not yet built: a machine's names are one seat's.** `/etc/hosts` and
+`/etc/hostname` belong to one seat for the machine's identity. The `hosts` module, holding
+`node-hosts-file` ([ADR 0199](0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md)),
+is renamed **`hostname`** and holds the seat renamed **`node-hostname`**; it writes `/etc/hostname`
+and the machine's own `127.0.1.1` line, and keeps every operator's line as ADR 0199 does today.
+
+- **Why one seat for both files**: the mesh's only content in `/etc/hosts` is the machine's own name,
+  and nothing owns `/etc/hostname` today. Two files saying one fact belong to one owner.
+- **Why `node-hostname`**: a system seat is named `node-` for its scope and then for its role
+  ([ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md)), and the role
+  is the machine's name. `node-identity` was considered and rejected: a machine's identity in the mesh
+  already means its key and certificate. `node-host` was rejected for the reason the module name
+  `host` was: "the host" is what the [node-engine](../00-META/glossary.md) was called until now, and
+  its repository and binary still carry `mesh-host`.
+- **Why a module and not part of the node-engine**: the node-engine applies every module's resources
+  and owns no file's content; every file it writes belongs to the module that declared it. A file's
+  content belongs to a seat's holder, and a seat's holder stays replaceable — another module can hold
+  `node-hostname` on a machine that names itself another way, and the node-engine does not change.
+- The rename goes through the seat set's rename (an alias keeps `node-hosts-file` resolving,
+  [ADR 0122](0122-a-seat-is-data-a-rename-is-a-database-update.md)), so nothing claiming the old name
+  breaks in between.
+
+## Consequences
+
+- **musl and glibc agree.** Every listed server answers a mesh name the same way, so the race musl
+  runs has one possible outcome; a public name resolves through either holder's upstreams.
+- **Accepted cost: with no mesh resolver reachable, a machine has no DNS at all** until it can reach
+  one. The anchor is the hub of the private network, so with it down the private network is down too,
+  and the second resolver is reachable only on its own machine and from machines on its own LAN. The
+  mesh assumes the anchor is up about 99.9% of the time. The same holds for a laptop behind a captive
+  portal that keeps the tunnel down: it resolves nothing, the portal's own name included, until the
+  tunnel is up.
+- **The resolver's options change from one attempt to two**, one second each. With no public resolver
+  to fall back to, a single dropped datagram would otherwise fail a lookup on a machine that reaches
+  only one resolver.
+- **Holdings are keyed by seat and assignment.** The store's holding table takes a numbered migration;
+  every existing row is one per seat and satisfies the new key. A handover replaces every holder in one
+  transaction.
+- **Unassigning a holder takes its own row only**: the other resolver keeps holding. Removing the
+  second resolver is unassigning it, then `push --behind`.
+- **The rollout order matters.** The controller that knows replicated seats and renders the holders
+  rolls out first; the catalogue's `resolv-conf`, which reads the holders, second — a controller without
+  them cannot render it; and only then is the second holder added, because an older `resolv-conf`
+  reading its one binding would name the first holder by name order, which may be the new one, and
+  still list the public resolver.
+- **A container keeps the resolvers it started with.** A container on the default bridge copies its
+  machine's `resolv.conf` when it starts; one on a user-defined network is answered by the runtime's
+  embedded resolver, which forwards to the servers it read at start. Either keeps the public resolver
+  until restarted.
+- **Parts 2 and 3 change who writes two files on every machine**, each a handover between modules on
+  the same path; they wait for their own build handoff.
+
+## How it is checked
+
+| Rule | Checked by |
+|---|---|
+| `mesh-dns-resolver` is replicated, and no other seat is; it survives loading the set from the store | mesh-controller unit test over the compiled set and over a store row without the attribute |
+| Two holders on record compose on both, with no refusal, and each lists itself first | mesh-controller resolution and composition test with the catalogue's `dnsmasq` and `resolv-conf` |
+| A machine holding no resolver lists both holders, by name, and no public resolver | the same test on a third machine, asserting no public address appears |
+| A holder answers its own requirement although another holder sorts first | a resolution test on the second holder (issue 258's case kept) |
+| A seat held once still refuses a second claimant, and refuses two holders on record | resolution tests on `mesh-store` |
+| Two claimants of the replicated seat with nothing on record are refused, naming the handover | a resolution test |
+| Every consumer is bound to the same holder whatever order the mesh was resolved in | a unit test on the holder among providers |
+| The store keeps several holders of one seat, once each; a handover leaves one; unassigning takes only its own row | a store test against a live database, through the migration |
+| `seat --add` is refused for a seat held once | the controller's `seat` command |
+| The resolver's machine list has one host record per machine (issue 262's missing check) | a mesh-controller composition test counting host records |
+| `resolv-conf` lists only the seat's holders, with two short attempts | a mesh-controller test reading the catalogue's `resolv-conf` |
+| Live: every machine's `/etc/resolv.conf` lists both holders' private addresses, its own first on a holder, and nothing else; an Alpine container on the host network of the home server resolves `<anchor>.internal` every time | after rollout, read through each machine's tools, and `getent hosts` in an Alpine container repeated ten times |
+| Parts 2 and 3 | not yet built; their checks are written with their handoff |
+
+## References
+
+- [ADR 0194](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md) — the
+  mesh's resolver; now held on two machines, each holding every node's internal domain.
+- [ADR 0196](0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md) —
+  replaced in part: its public resolver second goes. Every node and container asking the mesh's
+  resolvers for every name, with no stub and no runtime `dns`, stands.
+- [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md) —
+  several holders sharing a role, for a node seat.
+- [ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) — holders on record, handover,
+  eligible and silent.
+- [ADR 0220](0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md) — the
+  uplink's dependency, which part 2 retires.
+- [ADR 0117](0117-a-machines-uplink-is-a-seat.md), [ADR 0199](0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md),
+  [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md),
+  [ADR 0122](0122-a-seat-is-data-a-rename-is-a-database-update.md).
+- [Issue 258](../04-ISSUES/258-every-machine-bound-the-resolver-to-itself/00-report.md),
+  [issue 262](../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md).
+- [Connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md) and [the seats](../03-DESIGN/01-to-be/26-the-seats.md),
+  amended alongside.
+- mesh-controller `internal/catalogue/seats.go`, `resolve.go`, `roster.go`,
+  `cmd/mesh-controller/seats.go`, `holdings.go`, and the store migration keying a holding by seat and
+  assignment; mesh-catalog `modules/resolv-conf`, `modules/dnsmasq`.
@@ -235,6 +235,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0194** — [The mesh has one resolver, and every node asks it for the mesh's names](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)
 - **0196** — [A node asks the mesh's resolver first, and a public one only when it is silent](0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md)
 - **0199** — [A module that answers names declares its zone, and a node's hosts file is one module's](0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md)
+- **0223** — [The mesh has two resolvers, and a machine lists only them](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)

 ### What runs on them, and how it gets there