diff --git a/01-RESEARCH/033-split-dns-with-a-vpn-client/00-overview.md b/01-RESEARCH/033-split-dns-with-a-vpn-client/00-overview.md new file mode 100644 index 00000000..e7ecf4ab --- /dev/null +++ b/01-RESEARCH/033-split-dns-with-a-vpn-client/00-overview.md @@ -0,0 +1,69 @@ +--- +status: active +initiated: 2026-10-07 +touches: + - 02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md + - 02-DECISIONS/0117-a-machines-uplink-is-a-seat.md + - 03-DESIGN/01-to-be/08-connectivity.md + - the node-uplink seat and its holders + - the node-engine's reconcile of a file another program rewrites +--- + +# 033 — Split DNS with a VPN client + +## What is investigated + +How a machine of the mesh resolves three kinds of name at once — the mesh's (`*.internal`), a +corporate network's reached through a VPN client, and public ones — when the VPN client writes +`/etc/resolv.conf` itself. + +[ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md) gives +every machine one resolver file listing the mesh's two resolvers and nothing else, written by the holder +of the machine's uplink ([ADR 0117](../../02-DECISIONS/0117-a-machines-uplink-is-a-seat.md)). On the +laptop, a corporate VPN client replaces that file when it connects, with its own servers and eight search +domains. The node-engine finds the file changed and writes the mesh's back on its next reconcile. From +then on, for the rest of the session, the corporate names do not resolve; in the minutes before it, the +mesh's names did not. Neither program knows the other exists. + +The questions: + +1. **How the VPN client sets DNS** on Linux: does it overwrite the file, use `resolvconf`, or talk to + systemd-resolved or NetworkManager; what it backs up and restores; whether it re-asserts its file. +2. **What it pushes**: how many servers, which domains, over which link, and what routes. +3. **How often the two collide** on the live mesh, and what each collision costs. +4. **The options**: per-link routing in systemd-resolved, adding the VPN's servers to the mesh's file, + forwarding the corporate domains from the mesh's resolvers, standing back while the VPN is up, and + any other — and what each means for ADR 0223 and for the machines that run no VPN. + +## Why + +The operator uses the VPN for work, every working day. While it is up the laptop is either cut off from +the mesh (agents saw lookups fail) or cut off from the corporate network, depending on which program +wrote the file last. A rule of the mesh that a machine's own work program silently breaks — or that +breaks the machine's owner's work — is a rule that cannot be kept as stated. + +A sibling decision proposed alongside this effort, ADR 0241 (machine network health), makes the +node-engine detect and report an outside writer of the resolver file. That says the fault; this effort +asks how it stops being one. + +## What it touches + +ADR 0223's rule that a machine lists only the mesh's resolvers, and its rejection of a local forwarder; +ADR 0117's list of what an uplink holder declares; the networkmanager module (the uplink holder on the +laptop); the node-engine's correction of a file changed on the machine; connectivity §2. + +## Where it stands + +Evidence gathered read-only on the laptop and options weighed, 2026-10-07: + +- [01 — The evidence](01-evidence.md): the VPN client always *moves* the resolver file aside and writes + its own (two servers, eight search domains), restores its backup on disconnect, and never re-asserts; + it does not configure systemd-resolved or NetworkManager per link on this machine. In every one of the + nine sessions since the node-engine's journal begins, the node-engine wrote the mesh's file back within + 1 s to 3.5 min; once the VPN client's restore of a stale backup was itself corrected. +- [02 — Options](02-options.md): six, from systemd-resolved per-link routing to replacing the VPN + client. +- [03 — Recommendation](03-recommendation.md): on a machine that declares a VPN client, the uplink's + holder runs systemd-resolved as a local router on the machine's private address, and hands it the + VPN's servers and domains the moment the VPN client writes them; every other machine is unchanged. + A proposed decision text with how each rule is checked. diff --git a/01-RESEARCH/033-split-dns-with-a-vpn-client/01-evidence.md b/01-RESEARCH/033-split-dns-with-a-vpn-client/01-evidence.md new file mode 100644 index 00000000..dea4a565 --- /dev/null +++ b/01-RESEARCH/033-split-dns-with-a-vpn-client/01-evidence.md @@ -0,0 +1,100 @@ +# 01 — The evidence + +Gathered read-only on the laptop, 2026-10-07, with the VPN disconnected: the VPN client's installed +files, its own log (kept since its install, about five weeks), the strings of its binaries, the +node-engine's journal (kept for eleven days), and the machine's network state. Nothing was changed and +nothing restarted. The corporate network's domains and addresses are not reproduced here. + +## The machine + +- **Uplink:** NetworkManager, held by the mesh's `networkmanager` module, which tells it `dns=none` and + leaves the private network's interface (`mesh0`) unmanaged. The resolver file is the module's own, + declared from one template: the two mesh resolvers, `options timeout:1 attempts:2 edns0`. +- **systemd-resolved** is installed, as part of systemd, but disabled and not running. The name service + order is `files dns … resolve [!UNAVAIL=return]`, so with resolved stopped the resolver file is the + only DNS path. +- **The mesh's names are not in `/etc/hosts`** any more (ADR 0148 moved every name to the resolvers): + with the resolver file replaced, no mesh name resolves on the machine itself, nor in any container. +- **Five containers**: two on the host network (they read the machine's file as it is, at each lookup), + three on bridges (they read it, or the runtime's embedded resolver copies its servers, when they start). + +## The VPN client + +**FortiClient 7.4, the vendor's own Linux client** (not openfortivpn), its scheduler running as a system +service; the tunnel is set up on demand by its `vpn` process. + +**What it does at connect**, from its log, identical in all eleven sessions recorded: + +1. `Inherit local DNS: No` and `DNS service resetting interval: 0` — settings pushed with the profile. + The first means the machine's own resolvers are *not* kept beside the corporate ones; the second that + it never re-asserts its file during a session. +2. It configures the tunnel device with `ip` ("fall back to using ip command") — so NetworkManager sees + the tunnel as an external device it does not manage. +3. **`Moving /etc/resolv.conf to /etc/resolv.conf.forticlient.backup`** — a rename, so the mesh's file, + inode and all, becomes the backup. +4. It writes a new `/etc/resolv.conf`: **two nameservers** (in a private RFC 1918 range, reached through + the tunnel) and **eight search domains** (the corporate network's internal and cloud zones). Its + binary carries the header it writes: `# Dynamic resolv.conf(5) file for glibc resolver(3) generated + by forticlient`, and a line saying the original is backed up and restored after the VPN disconnects. +5. **Split tunnel**: 119 routes through the tunnel — 110 host routes, mostly to a CDN's addresses, and + nine networks including the whole of `10.0.0.0/8` and `172.16.0.0/12`. The mesh's private /24 sits + inside the first, and survives only because a more specific route on `mesh0` wins. + +**At disconnect:** `Moving /etc/resolv.conf.forticlient.backup to /etc/resolv.conf` — the backup taken +at connect is put back as it was then, whatever has happened to the file since. With no backup present +it logs "No DNS backup file was found. Skip." + +**What it does not do on this machine.** Its binary knows systemd-resolved (if running, it *reads* the +resolvers from resolved's own file and flushes resolved's cache) and NetworkManager (it can allocate the +tunnel through `nmcli`, set a connection's DNS search domains, and drop a configuration file into +NetworkManager's `conf.d`, keeping its own backups beside the resolver file). Here it took neither path. +In no path found in the binary does it give systemd-resolved a per-link server or routing domain; it +always writes `/etc/resolv.conf` itself. Public reports agree: with systemd-resolved running, the client +replaces the stub symlink with its static file. + +## The collision + +The node-engine corrects a declared file that was "changed on the machine since this host last wrote it" +on its next apply or reconcile. Matching its journal to the VPN client's log: + +| Session | VPN wrote its file | Node-engine wrote the mesh's back | Gap | +|---|---|---|---| +| 1 | 21:20:18 | 21:21:38 | 1 min 20 s | +| 2 | 11:51:33 | 11:54:48 | 3 min 15 s | +| 3 | 12:59:48 | 13:00:59 | 1 min 11 s | +| 4 | 12:26:30 | 12:29:55 | 3 min 25 s | +| 5 | 12:31:36 | 12:34:54 | 3 min 18 s | +| 6 | 12:42:58 | 12:44:54 | 1 min 56 s | +| 7 | 13:53:22 | 13:54:06 | 44 s | +| 8 | 14:58:37 | 15:00:35 | 1 min 58 s | +| 9 | 15:46:13 | 15:46:14 | 1 s | + +- **Nine of nine sessions** since the journal begins were overwritten by the mesh, within 1 s to 3.5 min + (median about 1 min 56 s). Two earlier sessions predate the journal. +- **Each session then ran on the mesh's file** — sessions lasted from five minutes to fourteen hours — + so for all but the first minutes **no corporate name resolved**, while the tunnel and its routes stood. + The VPN client never noticed: its resetting interval is 0. +- **Sessions 4–6 were three connects within sixteen minutes**, two ended by hand after five and seven + minutes — the shape of a person reconnecting because something stopped working. +- **The VPN client's restore is itself an outside write.** On one night the uplink's holder changed (the + resolver file moved from one module's template to another's) while a session was up; when the session + ended at 02:43 the client put back its backup — the old module's file — and the node-engine corrected + it two minutes later. A restore puts back *a* mesh file, not necessarily the current one. +- **In the minutes before each correction**, the machine had only the corporate servers: no mesh name + resolved (the agents' lookups failed), and public names depended on the corporate servers. The + operator reported agents seeing `ENOTFOUND` for the AI API's public name in such a window; it is not in + any journal kept on the machine, so how often public names failed is not measured. + +## What the other machines have + +No VPN client runs on the anchor, the home server or the workstation. The two mesh resolvers have no +route to the corporate network's servers: they sit behind the laptop's tunnel, in ranges the laptop +routes only to itself. + +## Two related faults, noted + +- **One mesh resolver briefly slow under load** ([issue 277](../../04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md)): + any design that adds a hop must keep the two-resolver answer and its short timeouts. +- **The self-check asks the resolvers only from the control node** (to-be 45 §4, D2). A machine whose + own resolver file is wrong is invisible to it — the gap ADR 0241 (proposed) closes from the machine's + side. diff --git a/01-RESEARCH/033-split-dns-with-a-vpn-client/02-options.md b/01-RESEARCH/033-split-dns-with-a-vpn-client/02-options.md new file mode 100644 index 00000000..36f38d75 --- /dev/null +++ b/01-RESEARCH/033-split-dns-with-a-vpn-client/02-options.md @@ -0,0 +1,113 @@ +# 02 — Options + +What any answer must give, from the [evidence](01-evidence.md) and ADR 0223: + +- **R1** mesh names resolve on the machine and in its containers while the VPN is up; +- **R2** the corporate domains resolve through the VPN's servers while it is up; +- **R3** public names resolve, VPN up or down; +- **R4** no listed set of servers can give two different answers to one name — musl asks every listed + server at once and takes the first reply, glibc takes the first server's "no such name" as final + ([issue 262](../../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md), + ADR 0223); +- **R5** the VPN client keeps working unmodified — it is the employer's software, configured by the + employer's gateway; the mesh cannot change what it pushes or how it writes; +- **R6** no change on a machine that runs no VPN. + +A constraint every option meets: **the VPN client always writes `/etc/resolv.conf` itself**, even with +systemd-resolved running. No option makes it stop; each decides what happens next. + +## 1. systemd-resolved with per-link DNS and routing domains + +resolved runs on the machine. The `mesh0` link gets the two mesh resolvers with the routing domain +`~internal` and the default route (`~.`); the tunnel link gets the VPN's two servers and its eight +domains as routing domains. resolved sends each question to the link whose domain matches longest, the +rest to the default route. + +- **R1–R3 met**, by routing rather than by listing: one answer per name. +- **Containers cannot reach resolved's stub on `127.0.0.53`.** A container on the host network can; one + on a bridge cannot — it reads the machine's file and gets an address that is its own loopback. resolved + can listen on further addresses (`DNSStubListenerExtra=`): on the machine's private mesh address, which + ADR 0223 already relies on being reachable from containers on a holder. The machine's file then lists + that one address. **R4 met**: one server listed. +- **Who gives the tunnel link its servers?** Not the VPN client (it never sets per-link DNS) and not + NetworkManager (the tunnel is not its device). Something must read what the VPN pushed — and the VPN + client writes exactly that, in its file, within a second of connecting. A path watch on the resolver + file, declared by the uplink's holder, can read the file's `nameserver` and `search` lines, give them to + the tunnel link with `resolvectl dns` and `resolvectl domain`, and write the mesh's file back at once — + not at the next reconcile. When the tunnel goes, resolved forgets its link's configuration by itself. +- **The VPN client's restore at disconnect** puts back the mesh's file (or an older one: the node-engine + corrects the content, as today). Nothing points at a link that is gone. +- **Cost:** a daemon on the machine (already installed; part of systemd), its configuration, the watch. +- **For ADR 0223:** it is the local forwarder 0223 rejected (its option 2), on fewer machines and with + its three objections met: not on every machine — only on one that declares a VPN client; reachable + from containers — on the private address; and it holds no copy of the mesh's names — every `.internal` + question still goes to the two resolvers, so nothing can disagree with the truth. +- **R6 met** if it is scoped to machines that declare a VPN client. + +## 2. The uplink's holder adds the VPN's servers to the mesh's file + +When the tunnel appears, the holder writes the mesh's two resolvers *and* the VPN's two servers, and the +VPN's search domains. + +- **R4 broken, by construction.** The VPN's servers answer "no such name" for `.internal`; the mesh's + resolvers forward a corporate name to the public internet, which answers "no such name" or a public + record. glibc stops at the first listed server's answer, musl takes whichever is fastest. Whatever order + is chosen, one kind of name fails — the exact fault ADR 0223 removed the public resolver for. +- Rejected. It only works with something that routes by domain in front — which is option 1 or 5. + +## 3. The mesh's resolvers forward the corporate domains while the VPN is up + +- **They cannot reach the VPN's servers**: those sit behind the laptop's tunnel, routed only on the + laptop. Forwarding through the laptop would make the mesh's resolvers depend on one laptop's session. +- It would also put an employer's zones into the mesh's shared configuration, answered for every machine. +- Rejected. + +## 4. Stand back: detect the VPN and defer + +While the resolver file carries the VPN client's marker and its backup holds the mesh's file, the +node-engine does not correct it, and the machine's health says *deferred to a VPN client* rather than +*rewritten by another program*. On disconnect the client restores the mesh's file. + +- **R2, R3 met; R1 broken for the whole session**: no mesh name resolves on the laptop, nor in its + containers. The node-engine reaches the bus by name; agents and tools on the laptop lose the mesh. +- **Cheap**: no daemon, one rule in the node-engine (and the content check on restore stays). +- Today's behaviour is the opposite trade — R1 met after a few minutes, R2 broken for the rest of the + session — and the evidence shows the operator living with it by reconnecting. +- Acceptable only as a stated, chosen degradation for a machine where the owner's work comes first; + better than the fight, worse than routing. **Useful as the fallback** of option 1 while it is not + built, if the operator prefers work names to mesh names. + +## 5. A local forwarder of the mesh's own (dnsmasq) on the VPN machine + +As option 1, with the catalogue's resolver program in place of resolved: `server=//…` +lines for the VPN's domains, everything else to the two mesh resolvers, listening on the private address. + +- Meets R1–R4 like option 1. +- **Costs more than option 1**: a package to declare, and a configuration file to rewrite and a + process to reload on every connect and disconnect, where resolved takes per-link settings live over its + own interface and drops them with the link. dnsmasq is already the mesh's resolver module, which makes + "a second copy of the resolver on this machine" easy to misread as a third mesh resolver. +- Kept as the fallback for a machine without systemd. + +## 6. Replace the VPN client + +openfortivpn, through NetworkManager's plugin, hands the VPN's DNS to NetworkManager instead of writing +the file; NetworkManager with `dns=systemd-resolved` then gives resolved per-link servers and domains — +option 1 with no watch at all. + +- **Not the mesh's to decide** (R5): the employer's gateway may require its own client (posture checks, + single sign-on through the vendor's flow). It would also hand the file back to NetworkManager, against + ADR 0117's `dns=none`. +- Recorded so the option is not rediscovered; not proposed. + +## Summary + +| | R1 mesh | R2 corporate | R3 public | R4 one answer | R6 others untouched | New on the machine | +|---|---|---|---|---|---|---| +| 1 resolved, per link | yes | yes | yes | yes | yes, if scoped | resolved, a path watch | +| 2 add the VPN's servers | some | some | yes | **no** | yes | nothing | +| 3 forward from the mesh | yes | **no** | yes | yes | **no** | nothing | +| 4 defer | **no** | yes | yes | yes | yes | one rule | +| 5 dnsmasq locally | yes | yes | yes | yes | yes, if scoped | a package, a reload per connect | +| 6 replace the client | yes | yes | yes | yes | yes | not the mesh's call | +| today (the fight) | after minutes | **no**, after minutes | mostly | yes | yes | — | diff --git a/01-RESEARCH/033-split-dns-with-a-vpn-client/03-recommendation.md b/01-RESEARCH/033-split-dns-with-a-vpn-client/03-recommendation.md new file mode 100644 index 00000000..0a4ac2de --- /dev/null +++ b/01-RESEARCH/033-split-dns-with-a-vpn-client/03-recommendation.md @@ -0,0 +1,89 @@ +# 03 — Recommendation + +From the [evidence](01-evidence.md) and the [options](02-options.md): **route by domain on the one +machine that runs a VPN client, and leave every other machine as ADR 0223 has it.** On a machine whose +operator declares a VPN client, the uplink's holder runs systemd-resolved as the machine's resolver, +listening on the machine's private address; the mesh's link carries the two mesh resolvers for the +mesh's domain and as the default route; the moment the VPN client writes its own resolver file, the +holder hands that file's servers and domains to the tunnel link and writes the mesh's file back. The +fight ends because the two programs no longer want different things: the VPN's servers are used, for the +VPN's domains only. + +Why not the cheaper ones: adding the VPN's servers to the file breaks the one-answer rule that ADR 0223 +exists for (option 2); the mesh's resolvers cannot reach the VPN's servers (option 3); standing back cuts +the laptop off from the mesh for whole working days (option 4). Option 4 is kept as what the machine does +*until* this is built, if the operator prefers it to today's fight — a choice for the operator, stated in +the proposed text as an interim setting, not as the decision. + +What it asks of the sibling proposal, ADR 0241 (machine network health): on a machine that declares a VPN +client, the VPN client's file is not an outside writer's fault but an expected handover — it is reported +as *adopted* once the holder has taken its servers, and as a fault only if it stands longer than the +watch's bound (a few seconds), which means the watch did not fire. + +## Proposed decision text + +Ready for graduation through playbook 02. The number is assigned then. + +> **A machine with a VPN client routes names by domain, and the VPN's file is taken, not fought** +> +> **Context.** ADR 0223 gives every machine one resolver file listing the mesh's two resolvers and +> nothing else, written by the uplink's holder (ADR 0117). A corporate VPN client on the laptop moves that +> file aside when it connects and writes its own — two servers reached through its tunnel, eight search +> domains — and puts its backup back on disconnect; it never re-asserts its file and never gives +> systemd-resolved per-link settings, even when resolved runs. The node-engine wrote the mesh's file back +> in nine of nine sessions, within 1 s to 3.5 min; from then on no corporate name resolved for the rest of +> sessions lasting up to fourteen hours, and before it no mesh name did. No resolver file can list both +> sets of servers: musl takes the first reply and glibc the first server's "no such name", so one kind of +> name would fail (issue 262). +> +> **Decision.** +> +> 1. **A machine declares that it runs a VPN client**, as a setting of its uplink's holder, naming the +> client. Nothing changes on a machine that does not. +> 2. **On such a machine the uplink's holder runs systemd-resolved** as the machine's only resolver, with +> its stub also listening on the machine's private address, so the machine's containers reach it. The +> private network's link carries the mesh's resolvers, in ADR 0223's order, as the routing domain of the +> mesh's names and as the default route. `/etc/resolv.conf` lists the machine's private address alone, +> with ADR 0223's options. resolved holds no copy of any mesh name: every mesh name is asked of the two +> resolvers, as on every other machine. +> 3. **The VPN client's file is taken, not fought.** The holder watches the resolver file; when the +> declared client writes it, the holder gives the tunnel link the file's servers and its search domains +> as routing domains, and writes the mesh's file back, within seconds — not at the next reconcile. When +> the tunnel goes, resolved drops its link's settings; the client's restore of its backup is corrected +> by content, as any change to the file is. +> 4. **The VPN's domains are never the mesh's.** They are not written to the mesh's store, its resolvers +> or any other machine; they live in resolved's state for the life of the tunnel. +> 5. **The machine says it.** The machine's network health (ADR 0241, proposed) states the VPN's file as +> *adopted*, with the tunnel link and the count of domains routed, and raises it as an outside writer +> only if it stands longer than the watch's bound or the client is not the declared one. +> 6. **Until rules 2–3 run on a machine, its operator may choose to defer**: the node-engine leaves the +> declared client's file in place while the client's backup holds the mesh's file, and the machine's +> health says *deferred to the VPN client — mesh names do not resolve* for as long as it lasts. +> +> **Consequences.** ADR 0223's rejection of a local forwarder stands for every machine but one that +> declares a VPN client; there its three objections are answered — one machine, not every machine; +> reachable from containers on the private address; no copy of the mesh's names. A machine that +> declares a VPN client and whose resolved is down has no names; its network health says so. The +> routes the VPN client adds are untouched: its routes include `10.0.0.0/8`, and the +> mesh's private network keeps working only because its own route is more specific — a fact to keep in +> mind when choosing a private range, and a check of its own. +> +> **How it is checked.** +> +> | Rule | Checked by | +> |---|---| +> | 1 Declared, nothing changes elsewhere | a controller composition test: a machine without the setting composes exactly as before; with it, the holder declares resolved and the watch | +> | 2 resolved, on the private address, mesh link and default route | a composition test reading the holder's files; live on the laptop: `resolvectl status` shows the mesh link with `~internal` and `~.`, the stub on the private address, and the machine's file lists that address alone | +> | 3 Taken within seconds | a lab drill: a container plays the VPN client (writes a file with its own header, servers and search lines, adds a tunnel link); the tunnel link carries the servers and domains and the mesh's file is back within 5 s; removing the link drops them | +> | 3 Containers | the lab drill: an Alpine container on a bridge resolves a mesh name ten times out of ten while the fake tunnel is up | +> | 4 Never the mesh's | a test that the store, the controller's renders and the other machines' files carry none of the VPN's domains after the drill | +> | 5 Said | the machine network health test: the VPN's file read as *adopted*, not raised; raised when it stands past the bound | +> | 6 Defer, while not built | a node-engine test: with the setting and the client's backup present, the file is not corrected and the health says deferred; without the backup it is corrected | +> | the routes | the machine network health's check that a mesh address still routes over the private link (ADR 0241, proposed) | + +## What this does not decide + +- Which VPN client the operator uses (option 6): the employer's. +- Resolution for a machine with *two* VPN clients — none exists. +- Whether the operator's corporate names should resolve inside the mesh's containers on the laptop: they + will, through the same router; restricting them is a later question if it matters.