hq/02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md

topic, status, date, deciders, reconstructed, supersedes-in-part, extends

topic	status	date	deciders	reconstructed	supersedes-in-part	extends
the tiers	accepted	2026-10-05	jochen	false	0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md	02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md

223. The mesh has two resolvers, and a machine lists only them

Context

ADR 0196 wrote every machine's /etc/resolv.conf as the mesh's resolver first and a public resolver second. Its reasoning is the C library's: servers are asked in order, the next only when one does not answer, and an answer from the first — "no such name" included — is final. That holds for glibc. It does not hold for musl, the C library of every Alpine image: musl sends the query to every listed server at once and takes the first reply.

On the production mesh on 2026-10-05 that made the anchor's own name unresolvable from the home server's builds. The build agent on the home server runs its steps in Alpine containers on the host network, so they read the machine's resolv.conf as it is. Asked for <anchor>.internal, the public resolver — which has no such name and is the nearer of the two — answered NXDOMAIN first, and musl took it. Reproduced six times out of six inside the build agent's own container; every build on that machine failed fetching from the anchor. The same lookup from glibc on the same machine answered every time. Issue 262 was the same library failing on a different answer (NXDOMAIN for a missing IPv6 record); fixing that did not touch this one, because here the wrong answer comes from a server that should never have been asked.

The public line was there for one case: the mesh's resolver unreachable. ADR 0196 kept public names resolving with the anchor down, the tunnel down, or a laptop behind a captive portal. That case is real and rare; the musl case is every lookup of a mesh name from every Alpine container on any machine that is not the anchor.

A seat's work can already be shared by several holders. ADR 0190 made the build role node-scoped with every holder pulling from one queue. A mesh-scoped seat has always had exactly one holder: by derivation, or on record since ADR 0131, where a handover replaces the holder in one write and every other eligible assignment stands beside it, silent.

Considered Options

1. The mesh's resolver alone. Every mesh name and every public name answered consistently, by one server. Rejected alone: with the anchor down, no machine resolves anything — the reason ADR 0194 gave against it, and still true.
2. A local forwarder on every machine — a small resolver on loopback that asks the mesh's resolver for the mesh's suffix and public resolvers for the rest, with resolv.conf naming only it. Rejected: it is ADR 0194's per-node stub again, which ADR 0196 removed — a daemon and a module on every node, and a container cannot use a loopback resolver, so containers would need a second configuration. Every resolution fault found on 2026-10-03 was a per-node copy disagreeing with the truth.
3. Split by kind of machine — servers list the mesh's resolver alone, laptops keep a public fallback. Rejected: a laptop runs Alpine containers too, and a rule that differs by machine is a rule nobody can state about the mesh.
4. Two mesh resolvers, and nothing else listed. The same module, the same roster and the same zones on two machines, and every machine lists both. Whichever answers first gives the same answer, so musl's race is harmless; a public name still resolves through either. Chosen.

Decision

1. Now: the mesh has two resolvers. The seat mesh-dns-resolver may be held on more than one machine — the anchor and the home server — each holder answering the same mesh names: the same module, the same machine list, the same zones, rendered by the controller into each.

• A seat may be replicated. It is an attribute of the seat in the mesh's definition, compiled with the set and never stored, as what a seat needs (ADR 0220) and what it receives are. mesh-dns-resolver is the only replicated seat. Making another one is a decision, recorded. In the glossary's words it is the mesh's first bench — a seat whose holders coexist — and replicated says what kind: every holder answers the same thing.
• Every holder is on record, and each is added by an act: seat mesh-dns-resolver --add <node>/<module> records one more holder beside those on record. Assigning the module is not enough: an assignment not on record stands beside the holders, eligible and silent, exactly as ADR 0131 says, and two claimants with nothing on record are refused as for any mesh seat. --to still hands the seat over, leaving exactly one holder. --add on a seat held once is refused, naming --to.
• One per machine still: two modules on one machine claiming it are refused. And a seat held once stays held once: a second claimant on another machine is refused while nothing is on record, and a store recording two holders of such a seat is refused, naming the seat.
• Every machine's /etc/resolv.conf lists every holder's private address, the holder on the machine itself first if it is one, then the rest in name order, and no public resolver. The controller gives a module's roster template the holders of each replicated seat, ordered so; the module holding node-resolver-config writes the file from it. On a holder, its own private address is first: the resolver listens there and on loopback, and the private address is the one a container on that machine can reach.
• The requirement stays. What writes the file still requires wildcard-resolution, so a machine is refused when nothing in the mesh resolves, rather than given a file listing nothing. A holder answers its own requirement; any other machine is bound to the first holder by name. Nothing reads that binding's address any more — the file lists every holder — and the binding is kept for the refusal and the order of delivery.

2. Next, decided and not yet built: /etc/resolv.conf belongs to the uplink's holder. The program that manages the machine's network already has to be told to keep off the file (ADR 0117); instead, the node-uplink holder writes it, given the resolvers by the mesh: NetworkManager through its global DNS configuration, dhcpcd through static nameservers, and systemd-networkd's module declaring the file itself. resolv-conf and the seat node-resolver-config then retire, and ADR 0220's dependency of node-resolver-config on node-uplink goes with them. One owner for the file, and it is the program that would otherwise rewrite it.

Progressive insight — 2026-10-05. Built, the managers' own mechanisms turned out not to write the mesh's file: NetworkManager's global DNS configuration and dhcpcd's resolv.conf hook each write /etc/resolv.conf in their own form — their own header, their own options line — and dhcpcd reads its configuration only at its next start, so a change of resolvers would wait for one. The record said NetworkManager and dhcpcd would be given the resolvers "through its global DNS configuration" and "through static nameservers"; instead every one of the three modules declares the file itself, from one template, and keeps its manager off it as before (dns=none, nohook resolv.conf, and nothing for systemd-networkd). The decision — the uplink's holder owns the file, and resolv-conf, node-resolver-config and ADR 0220's dependency retire — stands. How parts 2 and 3 are checked is in connectivity §2.

3. Next, decided and not yet built: a machine's names are one seat's. /etc/hosts and /etc/hostname belong to one seat for the machine's identity. The hosts module, holding node-hosts-file (ADR 0199), is renamed hostname and holds the seat renamed node-hostname; it writes /etc/hostname and the machine's own 127.0.1.1 line, and keeps every operator's line as ADR 0199 does today.

• Why one seat for both files: the mesh's only content in /etc/hosts is the machine's own name, and nothing owns /etc/hostname today. Two files saying one fact belong to one owner.
• Why node-hostname: a system seat is named node- for its scope and then for its role (ADR 0121), and the role is the machine's name. node-identity was considered and rejected: a machine's identity in the mesh already means its key and certificate. node-host was rejected for the reason the module name host was: "the host" is what the node-engine was called until now, and its repository and binary still carry mesh-host.
• Why a module and not part of the node-engine: the node-engine applies every module's resources and owns no file's content; every file it writes belongs to the module that declared it. A file's content belongs to a seat's holder, and a seat's holder stays replaceable — another module can hold node-hostname on a machine that names itself another way, and the node-engine does not change.
• The rename goes through the seat set's rename (an alias keeps node-hosts-file resolving, ADR 0122), so nothing claiming the old name breaks in between.

Consequences

• musl and glibc agree. Every listed server answers a mesh name the same way, so the race musl runs has one possible outcome; a public name resolves through either holder's upstreams.
• Accepted cost: with no mesh resolver reachable, a machine has no DNS at all until it can reach one. The anchor is the hub of the private network, so with it down the private network is down too, and the second resolver is reachable only on its own machine and from machines on its own LAN. The mesh assumes the anchor is up about 99.9% of the time. The same holds for a laptop behind a captive portal that keeps the tunnel down: it resolves nothing, the portal's own name included, until the tunnel is up.
• The resolver's options change from one attempt to two, one second each. With no public resolver to fall back to, a single dropped datagram would otherwise fail a lookup on a machine that reaches only one resolver.
• Holdings are keyed by seat and assignment. The store's holding table takes a numbered migration; every existing row is one per seat and satisfies the new key. A handover replaces every holder in one transaction.
• Unassigning a holder takes its own row only: the other resolver keeps holding. Removing the second resolver is unassigning it, then push --behind.
• The rollout order matters. The controller that knows replicated seats and renders the holders rolls out first; the catalogue's resolv-conf, which reads the holders, second — a controller without them cannot render it; and only then is the second holder added, because an older resolv-conf reading its one binding would name the first holder by name order, which may be the new one, and still list the public resolver.
• A container keeps the resolvers it started with. A container on the default bridge copies its machine's resolv.conf when it starts; one on a user-defined network is answered by the runtime's embedded resolver, which forwards to the servers it read at start. Either keeps the public resolver until restarted.
• Parts 2 and 3 change who writes two files on every machine, each a handover between modules on the same path; they wait for their own build handoff.

How it is checked

Rule	Checked by
mesh-dns-resolver is replicated, and no other seat is; it survives loading the set from the store	mesh-controller unit test over the compiled set and over a store row without the attribute
Two holders on record compose on both, with no refusal, and each lists itself first	mesh-controller resolution and composition test with the catalogue's dnsmasq and resolv-conf
A machine holding no resolver lists both holders, by name, and no public resolver	the same test on a third machine, asserting no public address appears
A holder answers its own requirement although another holder sorts first	a resolution test on the second holder (issue 258's case kept)
A seat held once still refuses a second claimant, and refuses two holders on record	resolution tests on mesh-store
Two claimants of the replicated seat with nothing on record are refused, naming the handover	a resolution test
Every consumer is bound to the same holder whatever order the mesh was resolved in	a unit test on the holder among providers
The store keeps several holders of one seat, once each; a handover leaves one; unassigning takes only its own row	a store test against a live database, through the migration
seat --add is refused for a seat held once	the controller's seat command
The resolver's machine list has one host record per machine (issue 262's missing check)	a mesh-controller composition test counting host records
resolv-conf lists only the seat's holders, with two short attempts	a mesh-controller test reading the catalogue's resolv-conf
Live: every machine's /etc/resolv.conf lists both holders' private addresses, its own first on a holder, and nothing else; an Alpine container on the host network of the home server resolves <anchor>.internal every time	after rollout, read through each machine's tools, and getent hosts in an Alpine container repeated ten times
Parts 2 and 3	not yet built; their checks are written with their handoff

References

• ADR 0194 — the mesh's resolver; now held on two machines, each holding every node's internal domain.
• ADR 0196 — replaced in part: its public resolver second goes. Every node and container asking the mesh's resolvers for every name, with no stub and no runtime dns, stands.
• ADR 0190 — several holders sharing a role, for a node seat.
• ADR 0131 — holders on record, handover, eligible and silent.
• ADR 0220 — the uplink's dependency, which part 2 retires.
• ADR 0117, ADR 0199, ADR 0121, ADR 0122.
• Issue 258, issue 262.
• Connectivity §2 and the seats, amended alongside.
• mesh-controller internal/catalogue/seats.go, resolve.go, roster.go, cmd/mesh-controller/seats.go, holdings.go, and the store migration keying a holding by seat and assignment; mesh-catalog modules/resolv-conf, modules/dnsmasq.