Merge pull request 'ADR 0223: the mesh has two resolvers, and a machine lists only them' (#115) from decision/0223-the-mesh-has-two-resolvers into main

2026-10-05 20:48:09 +00:00
parent 6e59281806 4f1300fd48
commit 178994ee0d
10 changed files with 285 additions and 24 deletions
@@ -80,7 +80,8 @@ another — and a mesh you cannot name precisely is a mesh two people describe d
  The set, with who holds each seat, is the overview of what a mesh has
  ([26 — The seats](../03-DESIGN/01-to-be/26-the-seats.md)). A seat has a **capacity**: a
  capacity-1 seat is exclusive (one holder); a higher-capacity seat is a **bench** (several holders
-  coexist).
+  coexist). The first bench is `mesh-dns-resolver`, a *replicated* mesh seat: one holder per machine,
+  each on record and each answering the same names ([ADR 0223](../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)).
 - **claim** — a module taking a spot on a seat. `claims: [{name, scope}]` in a manifest. A
  mesh-scoped exclusive claim is how the mesh says "there is one of me". A foundation seat is
  named after the server it guards: the `mesh-controller`, `postgres` and `lavinmq` modules claim
@@ -11,6 +11,11 @@ extends: 0191-the-meshs-resolver-holds-only-the-meshs-own-names.md

 # 194. The mesh has one resolver, and every node asks it for the mesh's names

+> **The mechanism changed — 2026-10-05, by [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md).** `mesh-resolver` (named `mesh-dns-resolver` in the
+> set) is no longer of capacity one: it is a replicated seat, held on the anchor and on the home
+> server, each holding every node's internal domain from the same roster. One resolving module, the
+> mesh's names held only by its holders, and the retirement of every per-node copy stand.
+
 > **Narrowed, not replaced — 2026-10-03.** How a node asks is decided again by
 > [ADR 0196](0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md): every
 > node and container asks `mesh-resolver` first and a public resolver only when it is silent. There is
@@ -10,6 +10,13 @@ supersedes-in-part:

 # 196. A node asks the mesh's resolver first, and a public one only when it is silent

+> **Narrowed, not replaced — 2026-10-05, by [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md).** The public resolver listed second goes: musl asks
+> every listed server at once and takes the first reply, so a public "no such name" for a mesh name
+> won it, and every Alpine build on the home server failed. Every machine now lists the mesh's
+> resolvers — two holders of `mesh-dns-resolver`, its own first on a holder — and nothing else. Every
+> node and container asking the mesh's resolver for every name, with no stub and no runtime `dns`,
+> stands. The "fallback" consequence and check below describe what this record decided, not what runs.
+
 ## Context

 **[ADR 0194](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md) gave the
@@ -9,6 +9,10 @@ extends: 0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-nam

 # 199. A module that answers names declares its zone, and a node's hosts file is one module's

+> **Decided to change — 2026-10-05, by [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md), not yet built.** The `hosts` module is renamed `hostname`
+> and its seat `node-hostname`, and it owns `/etc/hostname` as well as `/etc/hosts`; the operator's
+> lines are kept as this record decides. Until that is built, everything below stands as decided.
+
 ## Context

 **[ADR 0194](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md) and
@@ -9,6 +9,10 @@ extends: 02-DECISIONS/0207-a-module-depends-on-the-node-seats-that-apply-its-res

 # 220. What a machine asks needs its uplink held, and the retired resolver pieces go

+> **Decided to go — 2026-10-05, by [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md), not yet built.** `/etc/resolv.conf` becomes the `node-uplink`
+> holder's file, and `resolv-conf`, `node-resolver-config` and this record's dependency of it on
+> `node-uplink` retire with it. Until that is built, everything below stands as decided.
+
 ## Context

 **Three things about a machine's resolver were left half done when the mesh moved to one resolver.**
@@ -0,0 +1,193 @@
+---
+topic: the tiers
+status: accepted
+date: 2026-10-05
+deciders: jochen
+reconstructed: false
+supersedes-in-part:
+  - 0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md
+extends: 02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md
+---
+
+# 223. The mesh has two resolvers, and a machine lists only them
+
+## Context
+
+**[ADR 0196](0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md) wrote
+every machine's `/etc/resolv.conf` as the mesh's resolver first and a public resolver second.** Its
+reasoning is the C library's: servers are asked in order, the next only when one does not answer, and
+an answer from the first — "no such name" included — is final. That holds for glibc. It does not hold
+for musl, the C library of every Alpine image: musl sends the query to every listed server at once
+and takes the first reply.
+
+**On the production mesh on 2026-10-05 that made the anchor's own name unresolvable from the home
+server's builds.** The build agent on the home server runs its steps in Alpine containers on the host
+network, so they read the machine's `resolv.conf` as it is. Asked for `<anchor>.internal`, the public
+resolver — which has no such name and is the nearer of the two — answered NXDOMAIN first, and musl
+took it. Reproduced six times out of six inside the build agent's own container; every build on that
+machine failed fetching from the anchor. The same lookup from glibc on the same machine answered every
+time. [Issue 262](../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md)
+was the same library failing on a different answer (NXDOMAIN for a missing IPv6 record); fixing that
+did not touch this one, because here the wrong answer comes from a server that should never have been
+asked.
+
+**The public line was there for one case: the mesh's resolver unreachable.** ADR 0196 kept public
+names resolving with the anchor down, the tunnel down, or a laptop behind a captive portal. That case
+is real and rare; the musl case is every lookup of a mesh name from every Alpine container on any
+machine that is not the anchor.
+
+**A seat's work can already be shared by several holders.** [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md)
+made the build role node-scoped with every holder pulling from one queue. A mesh-scoped seat has
+always had exactly one holder: by derivation, or on record since
+[ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md), where a handover replaces the
+holder in one write and every other eligible assignment stands beside it, silent.
+
+## Considered Options
+
+1. **The mesh's resolver alone.** Every mesh name and every public name answered consistently, by one
+   server. Rejected alone: with the anchor down, no machine resolves anything — the reason ADR 0194
+   gave against it, and still true.
+2. **A local forwarder on every machine** — a small resolver on loopback that asks the mesh's resolver
+   for the mesh's suffix and public resolvers for the rest, with `resolv.conf` naming only it. Rejected:
+   it is ADR 0194's per-node stub again, which ADR 0196 removed — a daemon and a module on every node,
+   and a container cannot use a loopback resolver, so containers would need a second configuration.
+   Every resolution fault found on 2026-10-03 was a per-node copy disagreeing with the truth.
+3. **Split by kind of machine** — servers list the mesh's resolver alone, laptops keep a public
+   fallback. Rejected: a laptop runs Alpine containers too, and a rule that differs by machine is a
+   rule nobody can state about the mesh.
+4. **Two mesh resolvers, and nothing else listed.** The same module, the same roster and the same zones
+   on two machines, and every machine lists both. Whichever answers first gives the same answer, so
+   musl's race is harmless; a public name still resolves through either. Chosen.
+
+## Decision
+
+**1. Now: the mesh has two resolvers.** The seat `mesh-dns-resolver` may be held on more than one
+machine — the anchor and the home server — each holder answering the same mesh names: the same
+module, the same machine list, the same zones, rendered by the controller into each.
+
+- **A seat may be replicated.** It is an attribute of the seat in the mesh's definition, compiled with
+  the set and never stored, as what a seat needs ([ADR 0220](0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md))
+  and what it receives are. `mesh-dns-resolver` is the only replicated seat. Making another one is a
+  decision, recorded. In the glossary's words it is the mesh's first **bench** — a seat whose
+  holders coexist — and *replicated* says what kind: every holder answers the same thing.
+- **Every holder is on record, and each is added by an act**: `seat mesh-dns-resolver --add
+  <node>/<module>` records one more holder beside those on record. Assigning the module is not enough:
+  an assignment not on record stands beside the holders, eligible and silent, exactly as ADR 0131 says,
+  and two claimants with nothing on record are refused as for any mesh seat. `--to` still hands the
+  seat over, leaving exactly one holder. `--add` on a seat held once is refused, naming `--to`.
+- **One per machine still**: two modules on one machine claiming it are refused. And a seat held once
+  stays held once: a second claimant on another machine is refused while nothing is on record, and a
+  store recording two holders of such a seat is refused, naming the seat.
+- **Every machine's `/etc/resolv.conf` lists every holder's private address, the holder on the machine
+  itself first if it is one, then the rest in name order, and no public resolver.** The controller
+  gives a module's roster template the holders of each replicated seat, ordered so; the module holding
+  `node-resolver-config` writes the file from it. On a holder, its own private address is first: the
+  resolver listens there and on loopback, and the private address is the one a container on that
+  machine can reach.
+- **The requirement stays.** What writes the file still requires `wildcard-resolution`, so a machine is
+  refused when nothing in the mesh resolves, rather than given a file listing nothing. A holder answers
+  its own requirement; any other machine is bound to the first holder by name. Nothing reads that
+  binding's address any more — the file lists every holder — and the binding is kept for the refusal
+  and the order of delivery.
+
+**2. Next, decided and not yet built: `/etc/resolv.conf` belongs to the uplink's holder.** The program
+that manages the machine's network already has to be told to keep off the file
+([ADR 0117](0117-a-machines-uplink-is-a-seat.md)); instead, the `node-uplink` holder writes it, given
+the resolvers by the mesh: NetworkManager through its global DNS configuration, dhcpcd through static
+nameservers, and systemd-networkd's module declaring the file itself. `resolv-conf` and the seat
+`node-resolver-config` then retire, and ADR 0220's dependency of `node-resolver-config` on `node-uplink`
+goes with them. One owner for the file, and it is the program that would otherwise rewrite it.
+
+**3. Next, decided and not yet built: a machine's names are one seat's.** `/etc/hosts` and
+`/etc/hostname` belong to one seat for the machine's identity. The `hosts` module, holding
+`node-hosts-file` ([ADR 0199](0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md)),
+is renamed **`hostname`** and holds the seat renamed **`node-hostname`**; it writes `/etc/hostname`
+and the machine's own `127.0.1.1` line, and keeps every operator's line as ADR 0199 does today.
+
+- **Why one seat for both files**: the mesh's only content in `/etc/hosts` is the machine's own name,
+  and nothing owns `/etc/hostname` today. Two files saying one fact belong to one owner.
+- **Why `node-hostname`**: a system seat is named `node-` for its scope and then for its role
+  ([ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md)), and the role
+  is the machine's name. `node-identity` was considered and rejected: a machine's identity in the mesh
+  already means its key and certificate. `node-host` was rejected for the reason the module name
+  `host` was: "the host" is what the [node-engine](../00-META/glossary.md) was called until now, and
+  its repository and binary still carry `mesh-host`.
+- **Why a module and not part of the node-engine**: the node-engine applies every module's resources
+  and owns no file's content; every file it writes belongs to the module that declared it. A file's
+  content belongs to a seat's holder, and a seat's holder stays replaceable — another module can hold
+  `node-hostname` on a machine that names itself another way, and the node-engine does not change.
+- The rename goes through the seat set's rename (an alias keeps `node-hosts-file` resolving,
+  [ADR 0122](0122-a-seat-is-data-a-rename-is-a-database-update.md)), so nothing claiming the old name
+  breaks in between.
+
+## Consequences
+
+- **musl and glibc agree.** Every listed server answers a mesh name the same way, so the race musl
+  runs has one possible outcome; a public name resolves through either holder's upstreams.
+- **Accepted cost: with no mesh resolver reachable, a machine has no DNS at all** until it can reach
+  one. The anchor is the hub of the private network, so with it down the private network is down too,
+  and the second resolver is reachable only on its own machine and from machines on its own LAN. The
+  mesh assumes the anchor is up about 99.9% of the time. The same holds for a laptop behind a captive
+  portal that keeps the tunnel down: it resolves nothing, the portal's own name included, until the
+  tunnel is up.
+- **The resolver's options change from one attempt to two**, one second each. With no public resolver
+  to fall back to, a single dropped datagram would otherwise fail a lookup on a machine that reaches
+  only one resolver.
+- **Holdings are keyed by seat and assignment.** The store's holding table takes a numbered migration;
+  every existing row is one per seat and satisfies the new key. A handover replaces every holder in one
+  transaction.
+- **Unassigning a holder takes its own row only**: the other resolver keeps holding. Removing the
+  second resolver is unassigning it, then `push --behind`.
+- **The rollout order matters.** The controller that knows replicated seats and renders the holders
+  rolls out first; the catalogue's `resolv-conf`, which reads the holders, second — a controller without
+  them cannot render it; and only then is the second holder added, because an older `resolv-conf`
+  reading its one binding would name the first holder by name order, which may be the new one, and
+  still list the public resolver.
+- **A container keeps the resolvers it started with.** A container on the default bridge copies its
+  machine's `resolv.conf` when it starts; one on a user-defined network is answered by the runtime's
+  embedded resolver, which forwards to the servers it read at start. Either keeps the public resolver
+  until restarted.
+- **Parts 2 and 3 change who writes two files on every machine**, each a handover between modules on
+  the same path; they wait for their own build handoff.
+
+## How it is checked
+
+| Rule | Checked by |
+|---|---|
+| `mesh-dns-resolver` is replicated, and no other seat is; it survives loading the set from the store | mesh-controller unit test over the compiled set and over a store row without the attribute |
+| Two holders on record compose on both, with no refusal, and each lists itself first | mesh-controller resolution and composition test with the catalogue's `dnsmasq` and `resolv-conf` |
+| A machine holding no resolver lists both holders, by name, and no public resolver | the same test on a third machine, asserting no public address appears |
+| A holder answers its own requirement although another holder sorts first | a resolution test on the second holder (issue 258's case kept) |
+| A seat held once still refuses a second claimant, and refuses two holders on record | resolution tests on `mesh-store` |
+| Two claimants of the replicated seat with nothing on record are refused, naming the handover | a resolution test |
+| Every consumer is bound to the same holder whatever order the mesh was resolved in | a unit test on the holder among providers |
+| The store keeps several holders of one seat, once each; a handover leaves one; unassigning takes only its own row | a store test against a live database, through the migration |
+| `seat --add` is refused for a seat held once | the controller's `seat` command |
+| The resolver's machine list has one host record per machine (issue 262's missing check) | a mesh-controller composition test counting host records |
+| `resolv-conf` lists only the seat's holders, with two short attempts | a mesh-controller test reading the catalogue's `resolv-conf` |
+| Live: every machine's `/etc/resolv.conf` lists both holders' private addresses, its own first on a holder, and nothing else; an Alpine container on the host network of the home server resolves `<anchor>.internal` every time | after rollout, read through each machine's tools, and `getent hosts` in an Alpine container repeated ten times |
+| Parts 2 and 3 | not yet built; their checks are written with their handoff |
+
+## References
+
+- [ADR 0194](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md) — the
+  mesh's resolver; now held on two machines, each holding every node's internal domain.
+- [ADR 0196](0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md) —
+  replaced in part: its public resolver second goes. Every node and container asking the mesh's
+  resolvers for every name, with no stub and no runtime `dns`, stands.
+- [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md) —
+  several holders sharing a role, for a node seat.
+- [ADR 0131](0131-everything-on-the-mesh-speaks-to-the-broker-seat.md) — holders on record, handover,
+  eligible and silent.
+- [ADR 0220](0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md) — the
+  uplink's dependency, which part 2 retires.
+- [ADR 0117](0117-a-machines-uplink-is-a-seat.md), [ADR 0199](0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md),
+  [ADR 0121](0121-a-system-seat-is-named-for-its-scope-and-modules-define-their-own.md),
+  [ADR 0122](0122-a-seat-is-data-a-rename-is-a-database-update.md).
+- [Issue 258](../04-ISSUES/258-every-machine-bound-the-resolver-to-itself/00-report.md),
+  [issue 262](../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md).
+- [Connectivity §2](../03-DESIGN/01-to-be/08-connectivity.md) and [the seats](../03-DESIGN/01-to-be/26-the-seats.md),
+  amended alongside.
+- mesh-controller `internal/catalogue/seats.go`, `resolve.go`, `roster.go`,
+  `cmd/mesh-controller/seats.go`, `holdings.go`, and the store migration keying a holding by seat and
+  assignment; mesh-catalog `modules/resolv-conf`, `modules/dnsmasq`.
@@ -235,6 +235,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0194** — [The mesh has one resolver, and every node asks it for the mesh's names](0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)
 - **0196** — [A node asks the mesh's resolver first, and a public one only when it is silent](0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md)
 - **0199** — [A module that answers names declares its zone, and a node's hosts file is one module's](0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md)
+- **0223** — [The mesh has two resolvers, and a machine lists only them](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)

 ### What runs on them, and how it gets there

@@ -9,13 +9,16 @@ code:
  - mesh-controller internal/catalogue/zones.go (the zones a module answers, ADR 0199)
  - mesh-controller internal/catalogue/seats.go (mesh-dns-resolver, node-hosts-file, node-resolver-config needing node-uplink)
  - mesh-controller internal/catalogue/seat_dependencies.go (a seat's need checked at assignment, ADR 0220)
-  - mesh-catalog modules/dnsmasq (the mesh's one resolver)
+  - mesh-controller cmd/mesh-controller/holdings.go (the holders of a replicated seat, ADR 0223)
+  - mesh-controller internal/catalogue/roster.go (each replicated seat's holders, for a template)
+  - mesh-catalog modules/dnsmasq (the mesh's resolvers)
  - mesh-catalog modules/resolv-conf (what a node asks)
  - mesh-catalog modules/hosts (a node's /etc/hosts)
  - mesh-host internal/identity/serving.go
  - mesh-host internal/apply (the service that reflects a rule set)
 updated: 2026-10-05
 decisions:
+  - 02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md
  - 02-DECISIONS/0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md
  - 02-DECISIONS/0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md
  - 02-DECISIONS/0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md
@@ -300,10 +303,10 @@ expensively enough to be worth restating:
 - **A node must not pin its own public name locally.** The duplicate record breaks resolution of
  that name for everything else that needs it.

-**What the host receives:** what to ask, not what to answer. The mesh has **one resolver**, holding
-every node's internal domain; a node asks it first and a public resolver only when it is silent
+**What the host receives:** what to ask, not what to answer. The mesh has **two resolvers**, each
+holding every node's internal domain; a node lists both and nothing else
 ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md),
-[ADR 0196](../../02-DECISIONS/0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md)).
+[ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)).

 **What goes away:** the `/etc/hosts` floor. It exists because a node had to reach the mesh
 database before its own DNS existed; with [ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)
@@ -361,24 +364,47 @@ not a list of containers.
 somebody starts by hand is not the mesh's to configure, and reaching into every container on a
 machine — declared or not — is what a nameserver in `resolv.conf` would be for.

-### One resolver for the mesh
+### The mesh's resolvers

-*2026-10-03.* **The mesh's names live in one place: the module holding `mesh-resolver`**, a mesh-scoped
-seat of capacity one, placed on the node every tunnel converges on. It holds one wildcard per node —
-`<node>.internal` and everything under it — and listens on the private network only. It answers the
-mesh's names from what it holds and forwards every other name, giving the public answer.
+*2026-10-03, revised 2026-10-05.* **The mesh's names live in one module, held on two machines: the
+holders of `mesh-dns-resolver`**, a mesh-scoped seat that is *replicated* — held on the anchor and on
+the home server, each running the same module with the same machine list and the same zones, rendered
+by the controller into each. Each holds one wildcard per node — `<node>.internal` and everything
+under it — and one host record per node, listens on its private address and loopback only, answers
+the mesh's names from what it holds and forwards every other name, giving the public answer
+([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md),
+[ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)).
+Each holder is on record, added by `seat mesh-dns-resolver --add <node>/<module>`; an assignment of
+the module not on record stands beside them, eligible and silent, and a seat held once stays held
+once. *Checked by the controller's resolution tests with two holders on record, with two claimants
+and nothing on record, and on a seat held once.*

-**Every node asks it for everything, and a public resolver only when it is silent.** The module
-holding `node-resolver-config` writes `/etc/resolv.conf` naming `mesh-resolver` first and a public
-resolver second, with a short timeout and one attempt: the C library moves to the second only when the
-first does not answer — the anchor or the tunnel down, a captive portal holding the tunnel back — so
-public names keep resolving then, and `.internal` is never asked of a public resolver while the mesh's
-answers. Containers take the same two from their machine, the runtime copying non-loopback resolvers
-into every container, so the runtime is given no `dns` of its own
+**Every node asks them for everything, and nothing else.** The module holding
+`node-resolver-config` writes `/etc/resolv.conf` listing every holder's private address — the holder
+on the machine itself first if it is one, then the rest by name — with a short timeout and two
+attempts, and **no public resolver**. ADR 0196 listed a public resolver second, for the anchor being
+unreachable; a C library that asks every listed server at once and takes the first reply — musl, so
+every Alpine container — took the public resolver's "no such name" for a mesh name, and every build on
+the home server failed. With only the mesh's resolvers listed, whichever answers first gives the one
+answer. The cost is stated: a machine that reaches no mesh resolver has no names until it does, and
+with the anchor down the second resolver is reachable only on its own machine and its own LAN.
+Containers take the same lines from their machine, the runtime copying non-loopback resolvers into
+every container, so the runtime is given no `dns` of its own
 ([ADR 0196](../../02-DECISIONS/0196-a-node-asks-the-meshs-resolver-first-and-a-public-one-only-when-it-is-silent.md),
 replacing ADR 0194's per-node `systemd-resolved` stub). `resolv-conf` is the one module the catalogue
 offers for it; the systemd-resolved split-DNS module that once claimed the same seat is gone
 ([ADR 0220](../../02-DECISIONS/0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md)).
+*Checked by the controller's composition tests on both holders and on a third machine, and live by
+each machine's `/etc/resolv.conf` and an Alpine container on the home server resolving the anchor's
+name every time.*
+
+**Next, decided and not yet built** ([ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)):
+`/etc/resolv.conf` becomes the file of the `node-uplink` holder, given the resolvers by the mesh —
+NetworkManager through its global DNS configuration, dhcpcd through static nameservers,
+systemd-networkd's module declaring the file — and `resolv-conf`, `node-resolver-config` and its need
+for the uplink below retire. And a machine's own names become one seat's: the `hosts` module is
+renamed `hostname` and holds `node-hostname`, writing `/etc/hostname` and the machine's `127.0.1.1`
+line and keeping the operator's lines.

 **That file stays the mesh's only beside the uplink.** A network manager rewrites `/etc/resolv.conf`
 on every connectivity change unless it is told not to, and the module holding `node-uplink` is what
@@ -388,17 +414,18 @@ machine with no uplink holder is refused, naming the managers' modules that coul
 the last uplink holder from beneath it is refused too
 ([ADR 0220](../../02-DECISIONS/0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md)). *Checked by the controller's seat-dependency tests and by `assign` live.*

-**No node holds a copy.** The per-node resolver, its zones file and the mesh's region of `/etc/hosts`
+**No node holds a copy of its own.** The two holders hold the same rendering of one roster, never a
+list anyone edits. The per-node resolver, its zones file and the mesh's region of `/etc/hosts`
 go, and the per-node resolver's seat with them, deleted from the set once nothing claimed it
 ([ADR 0220](../../02-DECISIONS/0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md)): every resolution fault found on 2026-10-03 was a copy disagreeing with the truth — a hosts file
 read once at start, an operator's old line beside the mesh's, a node's resolver lent to a LAN. No
 member's resolver answers a LAN; a router pointing at one is moved first. *Checked by each node's
-`/etc/resolv.conf` naming `mesh-resolver` then a public resolver, by no node but the holder answering
+`/etc/resolv.conf` listing the resolver's holders and nothing else, by no node but a holder answering
 DNS on any address, and by the router's DHCP DNS option naming the router.*

 **Names that are neither a node nor a route.** A module that answers names declares a zone (a
 setting) and the listen that answers it; the controller hands the `mesh-dns-resolver` holder every
-zone with its module's node address and published port, and the holder forwards that zone there and
+zone with its module's node address and published port, and each holder forwards that zone there and
 answers nothing in it itself — the lab answers `<machine>.incus` for its running scenarios this way.
 An operator's own names, unrelated to the mesh, live in `/etc/hosts`'s kept region, held per node by
 the `node-hosts-file` seat's holder and changed through its tools; the controller holds none of them
@@ -7,12 +7,15 @@ code:
  - mesh-controller internal/catalogue/seat_dependencies.go
  - mesh-controller internal/inventory/seats.go
  - mesh-controller internal/inventory/migrations/0039-a-seat-is-held-by-one-assignment-on-record.sql
+  - mesh-controller internal/inventory/migrations/0062-a-replicated-seat-has-several-holders-on-record.sql
+  - mesh-controller cmd/mesh-controller/holdings.go
  - mesh-controller cmd/mesh-controller/seats.go
  - mesh-controller cmd/mesh-controller/source.go
  - mesh-controller internal/inventory/migrations/0032-a-source-may-live-on-a-seat.sql
  - mesh-catalog modules/gitea/module.json
 updated: 2026-10-05
 decisions:
+  - 02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md
  - 02-DECISIONS/0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md
  - 02-DECISIONS/0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md
  - 02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md
@@ -38,7 +41,7 @@ A seat has four properties, fixed by the mesh rather than by any module:
 | property | is |
 |---|---|
 | name | what a definition names and an assignment holds, and what a person reads in the list |
-| scope | node, site or mesh: where its capacity applies. Every seat in the set has a capacity of one, so one holder per scope. A bench, a seat with several holders, is a word the glossary keeps and no seat uses yet |
+| scope | node, site or mesh: where its capacity applies. Every seat in the set has a capacity of one, so one holder per scope — except a *replicated* mesh seat, which may be held on several machines at once, one holder per machine, each on record ([ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)). `mesh-dns-resolver` is the only one. Replicated is part of the seat's definition, compiled with the set and never stored |
 | delivers | the provision its holder answers for, or nothing |
 | decision | the record that made it a seat |

@@ -69,6 +72,17 @@ needs ([28](28-building-the-bus.md), task 5.3). **And a holding is the assignmen
 holder takes the row with it, so a seat never points at something that is not running anywhere, and
 the seat falls back to derivation rather than to nothing.

+**A replicated seat has several holders on record** (revision, 2026-10-05,
+[ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)).
+`seat <name> --add <node>/<module>` records one more holder beside those on record, judged exactly
+as a handover is; `--to` still leaves exactly one. Each holder is added by that act and never by being
+assigned: an assignment not on record is eligible and silent, and two claimants with nothing on record
+are refused, as for any mesh seat. `--add` on a seat held once is refused, naming `--to`, and a store
+recording two holders of a seat held once is refused at resolution, naming the seat. A requirement the
+seat delivers is answered on a holder by itself, and elsewhere by the first holder in name order. What
+the holders are is given to a module's roster template, its own machine first — how every machine's
+resolver file lists both of the mesh's resolvers.
+
 The handover refuses what would make the new holder wrong before anything is written: the seat must
 exist, the assignment must exist, and the module must be able to hold the seat — claim it at its scope
 and provide what it delivers, judged against the store's row and not against anything compiled into a
@@ -130,13 +144,13 @@ convention, which later seats departed from.
 | `mesh-npm-package-registry` | `npm-package-registry` | mesh | `npm-package-registry` | the forge |
 | `mesh-git` | `git` | mesh | `git` | the forge |
 | `mesh-build-machine` | `the-build-machine` | node | — | a builder |
-| `mesh-resolver` | — | mesh | — | the mesh's one resolver, holding every node's internal domain ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)) |
+| `mesh-resolver` | — | mesh | — | the mesh's resolvers, each holding every node's internal domain ([ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md)); replicated, held on the anchor and the home server ([ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)) |
 | ~~`mesh-dns-port`~~ | `the-dns-port` | node | — | retired by [ADR 0194](../../02-DECISIONS/0194-the-mesh-has-one-resolver-and-every-node-asks-it-for-the-meshs-names.md): the local resolver became the mesh's one; deleted from the set, and from the store's table, once nothing claimed it ([ADR 0220](../../02-DECISIONS/0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md)) |
-| `node-hosts-file` | — | node | — | owns `/etc/hosts`: the machine's own lines and the operator's kept region, changed through its verbs `entries`, `add`, `remove` ([ADR 0199](../../02-DECISIONS/0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md)) |
+| `node-hosts-file` | — | node | — | owns `/etc/hosts`: the machine's own lines and the operator's kept region, changed through its verbs `entries`, `add`, `remove` ([ADR 0199](../../02-DECISIONS/0199-a-module-that-answers-names-declares-its-zone-and-a-nodes-hosts-file-is-one-modules.md)). Next, decided: renamed `node-hostname`, held by the module renamed `hostname`, owning `/etc/hostname` as well ([ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)) |
 | `mesh-intrusion-prevention` | `the-intrusion-prevention` | node | — | an intrusion-prevention service |
 | `mesh-packet-filter` | `the-packet-filter` | node | — | the packet filter |
 | `mesh-private-network` | `the-private-network` | node | — | the private network the mesh runs over |
-| `mesh-resolver-configuration` | `the-resolver-configuration` | node | — | the module writing `/etc/resolv.conf`; its holder needs the uplink's held on the same node ([ADR 0220](../../02-DECISIONS/0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md)) |
+| `mesh-resolver-configuration` | `the-resolver-configuration` | node | — | the module writing `/etc/resolv.conf`; its holder needs the uplink's held on the same node ([ADR 0220](../../02-DECISIONS/0220-what-a-machine-asks-needs-its-uplink-held-and-the-retired-resolver-pieces-go.md)). Next, decided: retired, the file becoming the uplink holder's ([ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)) |
 | `mesh-showcase` | `the-showcase` | node | — | the showcase module |

 The controller holds **the mesh's own** entries in code, and a test asserts their size and that
@@ -280,6 +294,7 @@ checked as their tables say:
 | A singular fact about machines is a placement of capacity one, refused by name | 0161: the overlay command's test for a second hub; the store's unique index. |
 | A holder of `node-uplink` is the dialect the machine runs | 0161: the host reports `uplink-<manager>` in its profile with every report; a resolution test refuses the other holder naming the capability. |
 | A seat's holder has the seats it needs beside it: the resolver configuration is refused at `assign` without the uplink held on its node, and the uplink's last holder cannot be taken from beneath it | 0220: seat-dependency tests on the definition, on a synthetic claimant, at assign, at unassign and at composition, and one reading the catalogue for the uplink's possible holders. |
+| A replicated seat has every holder on record and composes on each; a seat held once still refuses a second holder, on record or not; `--add` is refused for it | 0223: resolution and composition tests with two holders on record and a third machine, with two claimants and nothing on record, and on `mesh-store`; a unit test that only `mesh-dns-resolver` is replicated, surviving the store's rows; store tests keeping several holders once each, a handover leaving one, and unassigning one taking only its row. |
 | A retired seat leaves the set once nothing claims it, from the compiled set and the store's table both | 0220: the closed-set test's count, and the store migration that deletes the row. |
 | Holdings are derived, and the overview lists every seat | 0118: the `seats` command test, including an unheld seat. |
 | A build source on the seat records no address; an unheld seat refuses only self-hosted builds | 0111's tests. |
@@ -44,3 +44,7 @@ answered only by the wildcard, still answers NXDOMAIN for IPv6, exactly as it di
 Ask the mesh's resolver for the IPv6 address of a machine's name. The answer must be NOERROR with no
 records. Today that check is done by hand. A controller test that renders the machine list and
 requires one host record per machine should be added.
+
+*2026-10-05:* that test is added with [ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)
+— mesh-controller's composition test renders the resolver's machine list and requires exactly one
+host record per machine. The live check stays by hand, on each resolver.