Merge pull request 'Research 033: split DNS with a VPN client' (#167) from research/033-split-dns-with-a-vpn-client into main

This commit was merged in pull request #167.
This commit is contained in:
2026-10-07 16:54:28 +00:00
4 changed files with 371 additions and 0 deletions
@@ -0,0 +1,69 @@
---
status: active
initiated: 2026-10-07
touches:
- 02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md
- 02-DECISIONS/0117-a-machines-uplink-is-a-seat.md
- 03-DESIGN/01-to-be/08-connectivity.md
- the node-uplink seat and its holders
- the node-engine's reconcile of a file another program rewrites
---
# 033 — Split DNS with a VPN client
## What is investigated
How a machine of the mesh resolves three kinds of name at once — the mesh's (`*.internal`), a
corporate network's reached through a VPN client, and public ones — when the VPN client writes
`/etc/resolv.conf` itself.
[ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md) gives
every machine one resolver file listing the mesh's two resolvers and nothing else, written by the holder
of the machine's uplink ([ADR 0117](../../02-DECISIONS/0117-a-machines-uplink-is-a-seat.md)). On the
laptop, a corporate VPN client replaces that file when it connects, with its own servers and eight search
domains. The node-engine finds the file changed and writes the mesh's back on its next reconcile. From
then on, for the rest of the session, the corporate names do not resolve; in the minutes before it, the
mesh's names did not. Neither program knows the other exists.
The questions:
1. **How the VPN client sets DNS** on Linux: does it overwrite the file, use `resolvconf`, or talk to
systemd-resolved or NetworkManager; what it backs up and restores; whether it re-asserts its file.
2. **What it pushes**: how many servers, which domains, over which link, and what routes.
3. **How often the two collide** on the live mesh, and what each collision costs.
4. **The options**: per-link routing in systemd-resolved, adding the VPN's servers to the mesh's file,
forwarding the corporate domains from the mesh's resolvers, standing back while the VPN is up, and
any other — and what each means for ADR 0223 and for the machines that run no VPN.
## Why
The operator uses the VPN for work, every working day. While it is up the laptop is either cut off from
the mesh (agents saw lookups fail) or cut off from the corporate network, depending on which program
wrote the file last. A rule of the mesh that a machine's own work program silently breaks — or that
breaks the machine's owner's work — is a rule that cannot be kept as stated.
A sibling decision proposed alongside this effort, ADR 0241 (machine network health), makes the
node-engine detect and report an outside writer of the resolver file. That says the fault; this effort
asks how it stops being one.
## What it touches
ADR 0223's rule that a machine lists only the mesh's resolvers, and its rejection of a local forwarder;
ADR 0117's list of what an uplink holder declares; the networkmanager module (the uplink holder on the
laptop); the node-engine's correction of a file changed on the machine; connectivity §2.
## Where it stands
Evidence gathered read-only on the laptop and options weighed, 2026-10-07:
- [01 — The evidence](01-evidence.md): the VPN client always *moves* the resolver file aside and writes
its own (two servers, eight search domains), restores its backup on disconnect, and never re-asserts;
it does not configure systemd-resolved or NetworkManager per link on this machine. In every one of the
nine sessions since the node-engine's journal begins, the node-engine wrote the mesh's file back within
1 s to 3.5 min; once the VPN client's restore of a stale backup was itself corrected.
- [02 — Options](02-options.md): six, from systemd-resolved per-link routing to replacing the VPN
client.
- [03 — Recommendation](03-recommendation.md): on a machine that declares a VPN client, the uplink's
holder runs systemd-resolved as a local router on the machine's private address, and hands it the
VPN's servers and domains the moment the VPN client writes them; every other machine is unchanged.
A proposed decision text with how each rule is checked.
@@ -0,0 +1,100 @@
# 01 — The evidence
Gathered read-only on the laptop, 2026-10-07, with the VPN disconnected: the VPN client's installed
files, its own log (kept since its install, about five weeks), the strings of its binaries, the
node-engine's journal (kept for eleven days), and the machine's network state. Nothing was changed and
nothing restarted. The corporate network's domains and addresses are not reproduced here.
## The machine
- **Uplink:** NetworkManager, held by the mesh's `networkmanager` module, which tells it `dns=none` and
leaves the private network's interface (`mesh0`) unmanaged. The resolver file is the module's own,
declared from one template: the two mesh resolvers, `options timeout:1 attempts:2 edns0`.
- **systemd-resolved** is installed, as part of systemd, but disabled and not running. The name service
order is `files dns … resolve [!UNAVAIL=return]`, so with resolved stopped the resolver file is the
only DNS path.
- **The mesh's names are not in `/etc/hosts`** any more (ADR 0148 moved every name to the resolvers):
with the resolver file replaced, no mesh name resolves on the machine itself, nor in any container.
- **Five containers**: two on the host network (they read the machine's file as it is, at each lookup),
three on bridges (they read it, or the runtime's embedded resolver copies its servers, when they start).
## The VPN client
**FortiClient 7.4, the vendor's own Linux client** (not openfortivpn), its scheduler running as a system
service; the tunnel is set up on demand by its `vpn` process.
**What it does at connect**, from its log, identical in all eleven sessions recorded:
1. `Inherit local DNS: No` and `DNS service resetting interval: 0` — settings pushed with the profile.
The first means the machine's own resolvers are *not* kept beside the corporate ones; the second that
it never re-asserts its file during a session.
2. It configures the tunnel device with `ip` ("fall back to using ip command") — so NetworkManager sees
the tunnel as an external device it does not manage.
3. **`Moving /etc/resolv.conf to /etc/resolv.conf.forticlient.backup`** — a rename, so the mesh's file,
inode and all, becomes the backup.
4. It writes a new `/etc/resolv.conf`: **two nameservers** (in a private RFC 1918 range, reached through
the tunnel) and **eight search domains** (the corporate network's internal and cloud zones). Its
binary carries the header it writes: `# Dynamic resolv.conf(5) file for glibc resolver(3) generated
by forticlient`, and a line saying the original is backed up and restored after the VPN disconnects.
5. **Split tunnel**: 119 routes through the tunnel — 110 host routes, mostly to a CDN's addresses, and
nine networks including the whole of `10.0.0.0/8` and `172.16.0.0/12`. The mesh's private /24 sits
inside the first, and survives only because a more specific route on `mesh0` wins.
**At disconnect:** `Moving /etc/resolv.conf.forticlient.backup to /etc/resolv.conf` — the backup taken
at connect is put back as it was then, whatever has happened to the file since. With no backup present
it logs "No DNS backup file was found. Skip."
**What it does not do on this machine.** Its binary knows systemd-resolved (if running, it *reads* the
resolvers from resolved's own file and flushes resolved's cache) and NetworkManager (it can allocate the
tunnel through `nmcli`, set a connection's DNS search domains, and drop a configuration file into
NetworkManager's `conf.d`, keeping its own backups beside the resolver file). Here it took neither path.
In no path found in the binary does it give systemd-resolved a per-link server or routing domain; it
always writes `/etc/resolv.conf` itself. Public reports agree: with systemd-resolved running, the client
replaces the stub symlink with its static file.
## The collision
The node-engine corrects a declared file that was "changed on the machine since this host last wrote it"
on its next apply or reconcile. Matching its journal to the VPN client's log:
| Session | VPN wrote its file | Node-engine wrote the mesh's back | Gap |
|---|---|---|---|
| 1 | 21:20:18 | 21:21:38 | 1 min 20 s |
| 2 | 11:51:33 | 11:54:48 | 3 min 15 s |
| 3 | 12:59:48 | 13:00:59 | 1 min 11 s |
| 4 | 12:26:30 | 12:29:55 | 3 min 25 s |
| 5 | 12:31:36 | 12:34:54 | 3 min 18 s |
| 6 | 12:42:58 | 12:44:54 | 1 min 56 s |
| 7 | 13:53:22 | 13:54:06 | 44 s |
| 8 | 14:58:37 | 15:00:35 | 1 min 58 s |
| 9 | 15:46:13 | 15:46:14 | 1 s |
- **Nine of nine sessions** since the journal begins were overwritten by the mesh, within 1 s to 3.5 min
(median about 1 min 56 s). Two earlier sessions predate the journal.
- **Each session then ran on the mesh's file** — sessions lasted from five minutes to fourteen hours —
so for all but the first minutes **no corporate name resolved**, while the tunnel and its routes stood.
The VPN client never noticed: its resetting interval is 0.
- **Sessions 4–6 were three connects within sixteen minutes**, two ended by hand after five and seven
minutes — the shape of a person reconnecting because something stopped working.
- **The VPN client's restore is itself an outside write.** On one night the uplink's holder changed (the
resolver file moved from one module's template to another's) while a session was up; when the session
ended at 02:43 the client put back its backup — the old module's file — and the node-engine corrected
it two minutes later. A restore puts back *a* mesh file, not necessarily the current one.
- **In the minutes before each correction**, the machine had only the corporate servers: no mesh name
resolved (the agents' lookups failed), and public names depended on the corporate servers. The
operator reported agents seeing `ENOTFOUND` for the AI API's public name in such a window; it is not in
any journal kept on the machine, so how often public names failed is not measured.
## What the other machines have
No VPN client runs on the anchor, the home server or the workstation. The two mesh resolvers have no
route to the corporate network's servers: they sit behind the laptop's tunnel, in ranges the laptop
routes only to itself.
## Two related faults, noted
- **One mesh resolver briefly slow under load** ([issue 277](../../04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md)):
any design that adds a hop must keep the two-resolver answer and its short timeouts.
- **The self-check asks the resolvers only from the control node** (to-be 45 §4, D2). A machine whose
own resolver file is wrong is invisible to it — the gap ADR 0241 (proposed) closes from the machine's
side.
@@ -0,0 +1,113 @@
# 02 — Options
What any answer must give, from the [evidence](01-evidence.md) and ADR 0223:
- **R1** mesh names resolve on the machine and in its containers while the VPN is up;
- **R2** the corporate domains resolve through the VPN's servers while it is up;
- **R3** public names resolve, VPN up or down;
- **R4** no listed set of servers can give two different answers to one name — musl asks every listed
server at once and takes the first reply, glibc takes the first server's "no such name" as final
([issue 262](../../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md),
ADR 0223);
- **R5** the VPN client keeps working unmodified — it is the employer's software, configured by the
employer's gateway; the mesh cannot change what it pushes or how it writes;
- **R6** no change on a machine that runs no VPN.
A constraint every option meets: **the VPN client always writes `/etc/resolv.conf` itself**, even with
systemd-resolved running. No option makes it stop; each decides what happens next.
## 1. systemd-resolved with per-link DNS and routing domains
resolved runs on the machine. The `mesh0` link gets the two mesh resolvers with the routing domain
`~internal` and the default route (`~.`); the tunnel link gets the VPN's two servers and its eight
domains as routing domains. resolved sends each question to the link whose domain matches longest, the
rest to the default route.
- **R1–R3 met**, by routing rather than by listing: one answer per name.
- **Containers cannot reach resolved's stub on `127.0.0.53`.** A container on the host network can; one
on a bridge cannot — it reads the machine's file and gets an address that is its own loopback. resolved
can listen on further addresses (`DNSStubListenerExtra=`): on the machine's private mesh address, which
ADR 0223 already relies on being reachable from containers on a holder. The machine's file then lists
that one address. **R4 met**: one server listed.
- **Who gives the tunnel link its servers?** Not the VPN client (it never sets per-link DNS) and not
NetworkManager (the tunnel is not its device). Something must read what the VPN pushed — and the VPN
client writes exactly that, in its file, within a second of connecting. A path watch on the resolver
file, declared by the uplink's holder, can read the file's `nameserver` and `search` lines, give them to
the tunnel link with `resolvectl dns` and `resolvectl domain`, and write the mesh's file back at once —
not at the next reconcile. When the tunnel goes, resolved forgets its link's configuration by itself.
- **The VPN client's restore at disconnect** puts back the mesh's file (or an older one: the node-engine
corrects the content, as today). Nothing points at a link that is gone.
- **Cost:** a daemon on the machine (already installed; part of systemd), its configuration, the watch.
- **For ADR 0223:** it is the local forwarder 0223 rejected (its option 2), on fewer machines and with
its three objections met: not on every machine — only on one that declares a VPN client; reachable
from containers — on the private address; and it holds no copy of the mesh's names — every `.internal`
question still goes to the two resolvers, so nothing can disagree with the truth.
- **R6 met** if it is scoped to machines that declare a VPN client.
## 2. The uplink's holder adds the VPN's servers to the mesh's file
When the tunnel appears, the holder writes the mesh's two resolvers *and* the VPN's two servers, and the
VPN's search domains.
- **R4 broken, by construction.** The VPN's servers answer "no such name" for `.internal`; the mesh's
resolvers forward a corporate name to the public internet, which answers "no such name" or a public
record. glibc stops at the first listed server's answer, musl takes whichever is fastest. Whatever order
is chosen, one kind of name fails — the exact fault ADR 0223 removed the public resolver for.
- Rejected. It only works with something that routes by domain in front — which is option 1 or 5.
## 3. The mesh's resolvers forward the corporate domains while the VPN is up
- **They cannot reach the VPN's servers**: those sit behind the laptop's tunnel, routed only on the
laptop. Forwarding through the laptop would make the mesh's resolvers depend on one laptop's session.
- It would also put an employer's zones into the mesh's shared configuration, answered for every machine.
- Rejected.
## 4. Stand back: detect the VPN and defer
While the resolver file carries the VPN client's marker and its backup holds the mesh's file, the
node-engine does not correct it, and the machine's health says *deferred to a VPN client* rather than
*rewritten by another program*. On disconnect the client restores the mesh's file.
- **R2, R3 met; R1 broken for the whole session**: no mesh name resolves on the laptop, nor in its
containers. The node-engine reaches the bus by name; agents and tools on the laptop lose the mesh.
- **Cheap**: no daemon, one rule in the node-engine (and the content check on restore stays).
- Today's behaviour is the opposite trade — R1 met after a few minutes, R2 broken for the rest of the
session — and the evidence shows the operator living with it by reconnecting.
- Acceptable only as a stated, chosen degradation for a machine where the owner's work comes first;
better than the fight, worse than routing. **Useful as the fallback** of option 1 while it is not
built, if the operator prefers work names to mesh names.
## 5. A local forwarder of the mesh's own (dnsmasq) on the VPN machine
As option 1, with the catalogue's resolver program in place of resolved: `server=/<corporate domain>/…`
lines for the VPN's domains, everything else to the two mesh resolvers, listening on the private address.
- Meets R1–R4 like option 1.
- **Costs more than option 1**: a package to declare, and a configuration file to rewrite and a
process to reload on every connect and disconnect, where resolved takes per-link settings live over its
own interface and drops them with the link. dnsmasq is already the mesh's resolver module, which makes
"a second copy of the resolver on this machine" easy to misread as a third mesh resolver.
- Kept as the fallback for a machine without systemd.
## 6. Replace the VPN client
openfortivpn, through NetworkManager's plugin, hands the VPN's DNS to NetworkManager instead of writing
the file; NetworkManager with `dns=systemd-resolved` then gives resolved per-link servers and domains —
option 1 with no watch at all.
- **Not the mesh's to decide** (R5): the employer's gateway may require its own client (posture checks,
single sign-on through the vendor's flow). It would also hand the file back to NetworkManager, against
ADR 0117's `dns=none`.
- Recorded so the option is not rediscovered; not proposed.
## Summary
| | R1 mesh | R2 corporate | R3 public | R4 one answer | R6 others untouched | New on the machine |
|---|---|---|---|---|---|---|
| 1 resolved, per link | yes | yes | yes | yes | yes, if scoped | resolved, a path watch |
| 2 add the VPN's servers | some | some | yes | **no** | yes | nothing |
| 3 forward from the mesh | yes | **no** | yes | yes | **no** | nothing |
| 4 defer | **no** | yes | yes | yes | yes | one rule |
| 5 dnsmasq locally | yes | yes | yes | yes | yes, if scoped | a package, a reload per connect |
| 6 replace the client | yes | yes | yes | yes | yes | not the mesh's call |
| today (the fight) | after minutes | **no**, after minutes | mostly | yes | yes | — |
@@ -0,0 +1,89 @@
# 03 — Recommendation
From the [evidence](01-evidence.md) and the [options](02-options.md): **route by domain on the one
machine that runs a VPN client, and leave every other machine as ADR 0223 has it.** On a machine whose
operator declares a VPN client, the uplink's holder runs systemd-resolved as the machine's resolver,
listening on the machine's private address; the mesh's link carries the two mesh resolvers for the
mesh's domain and as the default route; the moment the VPN client writes its own resolver file, the
holder hands that file's servers and domains to the tunnel link and writes the mesh's file back. The
fight ends because the two programs no longer want different things: the VPN's servers are used, for the
VPN's domains only.
Why not the cheaper ones: adding the VPN's servers to the file breaks the one-answer rule that ADR 0223
exists for (option 2); the mesh's resolvers cannot reach the VPN's servers (option 3); standing back cuts
the laptop off from the mesh for whole working days (option 4). Option 4 is kept as what the machine does
*until* this is built, if the operator prefers it to today's fight — a choice for the operator, stated in
the proposed text as an interim setting, not as the decision.
What it asks of the sibling proposal, ADR 0241 (machine network health): on a machine that declares a VPN
client, the VPN client's file is not an outside writer's fault but an expected handover — it is reported
as *adopted* once the holder has taken its servers, and as a fault only if it stands longer than the
watch's bound (a few seconds), which means the watch did not fire.
## Proposed decision text
Ready for graduation through playbook 02. The number is assigned then.
> **A machine with a VPN client routes names by domain, and the VPN's file is taken, not fought**
>
> **Context.** ADR 0223 gives every machine one resolver file listing the mesh's two resolvers and
> nothing else, written by the uplink's holder (ADR 0117). A corporate VPN client on the laptop moves that
> file aside when it connects and writes its own — two servers reached through its tunnel, eight search
> domains — and puts its backup back on disconnect; it never re-asserts its file and never gives
> systemd-resolved per-link settings, even when resolved runs. The node-engine wrote the mesh's file back
> in nine of nine sessions, within 1 s to 3.5 min; from then on no corporate name resolved for the rest of
> sessions lasting up to fourteen hours, and before it no mesh name did. No resolver file can list both
> sets of servers: musl takes the first reply and glibc the first server's "no such name", so one kind of
> name would fail (issue 262).
>
> **Decision.**
>
> 1. **A machine declares that it runs a VPN client**, as a setting of its uplink's holder, naming the
> client. Nothing changes on a machine that does not.
> 2. **On such a machine the uplink's holder runs systemd-resolved** as the machine's only resolver, with
> its stub also listening on the machine's private address, so the machine's containers reach it. The
> private network's link carries the mesh's resolvers, in ADR 0223's order, as the routing domain of the
> mesh's names and as the default route. `/etc/resolv.conf` lists the machine's private address alone,
> with ADR 0223's options. resolved holds no copy of any mesh name: every mesh name is asked of the two
> resolvers, as on every other machine.
> 3. **The VPN client's file is taken, not fought.** The holder watches the resolver file; when the
> declared client writes it, the holder gives the tunnel link the file's servers and its search domains
> as routing domains, and writes the mesh's file back, within seconds — not at the next reconcile. When
> the tunnel goes, resolved drops its link's settings; the client's restore of its backup is corrected
> by content, as any change to the file is.
> 4. **The VPN's domains are never the mesh's.** They are not written to the mesh's store, its resolvers
> or any other machine; they live in resolved's state for the life of the tunnel.
> 5. **The machine says it.** The machine's network health (ADR 0241, proposed) states the VPN's file as
> *adopted*, with the tunnel link and the count of domains routed, and raises it as an outside writer
> only if it stands longer than the watch's bound or the client is not the declared one.
> 6. **Until rules 2–3 run on a machine, its operator may choose to defer**: the node-engine leaves the
> declared client's file in place while the client's backup holds the mesh's file, and the machine's
> health says *deferred to the VPN client — mesh names do not resolve* for as long as it lasts.
>
> **Consequences.** ADR 0223's rejection of a local forwarder stands for every machine but one that
> declares a VPN client; there its three objections are answered — one machine, not every machine;
> reachable from containers on the private address; no copy of the mesh's names. A machine that
> declares a VPN client and whose resolved is down has no names; its network health says so. The
> routes the VPN client adds are untouched: its routes include `10.0.0.0/8`, and the
> mesh's private network keeps working only because its own route is more specific — a fact to keep in
> mind when choosing a private range, and a check of its own.
>
> **How it is checked.**
>
> | Rule | Checked by |
> |---|---|
> | 1 Declared, nothing changes elsewhere | a controller composition test: a machine without the setting composes exactly as before; with it, the holder declares resolved and the watch |
> | 2 resolved, on the private address, mesh link and default route | a composition test reading the holder's files; live on the laptop: `resolvectl status` shows the mesh link with `~internal` and `~.`, the stub on the private address, and the machine's file lists that address alone |
> | 3 Taken within seconds | a lab drill: a container plays the VPN client (writes a file with its own header, servers and search lines, adds a tunnel link); the tunnel link carries the servers and domains and the mesh's file is back within 5 s; removing the link drops them |
> | 3 Containers | the lab drill: an Alpine container on a bridge resolves a mesh name ten times out of ten while the fake tunnel is up |
> | 4 Never the mesh's | a test that the store, the controller's renders and the other machines' files carry none of the VPN's domains after the drill |
> | 5 Said | the machine network health test: the VPN's file read as *adopted*, not raised; raised when it stands past the bound |
> | 6 Defer, while not built | a node-engine test: with the setting and the client's backup present, the file is not corrected and the health says deferred; without the backup it is corrected |
> | the routes | the machine network health's check that a mesh address still routes over the private link (ADR 0241, proposed) |
## What this does not decide
- Which VPN client the operator uses (option 6): the employer's.
- Resolution for a machine with *two* VPN clients — none exists.
- Whether the operator's corporate names should resolve inside the mesh's containers on the laptop: they
will, through the same router; restricting them is a later question if it matters.