novox/hq
6
0
Fork 0
Code Issues Pull Requests 7 Releases Wiki Activity

ADR 0241 and to-be 48 §10: a machine says how its network is, and an outside writer of a mesh file is a finding

Browse Source 
A VPN client rewrote the laptop's resolver file and every mesh name failed
while each module read healthy: nothing judged the machine under them.
This commit is contained in:
jochen
2026-10-07 20:33:23 +02:00
parent f6aa1ced5e
commit a22a1661e1
3 changed files with 254 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md
+186
View File
@@ -0,0 +1,186 @@
---
topic: what runs on it
status: accepted
date: 2026-10-07
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md
---
# 241. A machine says how its network is, and an outside writer of a mesh file is a finding
## Context
[ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) has the node-engine
judge everything a module runs. It judges nothing *under* the modules: the machine's own networking, which
every module on it uses and none of them owns.
**On the laptop, a corporate VPN client rewrites `/etc/resolv.conf` when it connects.** It moves the file
the uplink's holder wrote ([ADR 0117](0117-a-machines-uplink-is-a-seat.md),
[ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)) aside and writes its own, naming
its own resolvers. Mesh names then fail on the machine and in its containers, and agents saw "no such host"
for a public service they call. The node-engine writes the mesh's file back at its next reconcile, which ends
the VPN's names for the rest of the session; measured for
[research 033](../01-RESEARCH/033-split-dns-with-a-vpn-client/00-overview.md), in nine of nine VPN sessions
since the node-engine's journal begins, one second to three and a half minutes after each connect. **Nothing
said any of it.** Every module's check was green, because no check asked the machine.
Two more facts of the same kind: one mesh resolver answered slowly under load
([issue 277](../04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md)),
and the self-check's resolver probe asks the resolvers only from the control node, so a machine that cannot
reach them is not seen. An Alpine container failed a mesh name because a resolver answered "no such name"
for its IPv6 address
([issue 262](../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md)).
The operator's direction, 2026-10-07: *the health check should, from now on, also catch DNS issues or other
networking issues.*
**Checked against GENESIS.** *Failure must be loud*: a machine whose names stop resolving while every
module reads healthy is quiet failure. *The mesh notices when something is wrong before you do*: the
operator found this by reconnecting the VPN three times in sixteen minutes. *The mesh is a guest on a
personal node*: on the laptop the VPN client is the employer's, so the mesh says what it finds and repairs
nothing it does not own. Nothing here conflicts with GENESIS.
## Considered Options
**Who looks.**
1. *The controller's self-check probes each machine's resolvers from the control node.* Rejected: this is
what exists, and it is blind to the machine that cannot reach them. A machine's names are a fact on that
machine.
2. **The node-engine on each machine looks at its own networking, beside its liveness looks.** Chosen. It
already reads the machine for ADR 0240, and it applied the file it compares.
**What is looked at.** Rejected: a full network test (throughput, latency to every machine), which costs
more than it tells. Chosen: five cheap parts, each a fact some incident needed (Decision §1).
**What an outside writer is.**
1. *A cause of the names failing, said only in the names' finding.* Rejected: it hides the actionable fact.
The operator can do something about another program owning a mesh file, and cannot do anything about
names failing.
2. **Its own finding, naming the writer where it can.** Chosen. The names that fail through the rewritten
file are listed as what it costs, not raised a second time.
**What the gate does with a network condition.** Under
[issue 281](../04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine/00-report.md),
a machine-level condition raised since the send holds the machine as a whole, and it fails at the bound.
1. *Leave it so.* Rejected for the outside writer: a VPN connecting during a send would put back a good build.
2. *Ignore network conditions in the gate.* Rejected: a send that breaks the machine's network is exactly
what the gate exists to catch.
3. **What is shown to be another's waits; what is the machine's own stays the machine's.** Chosen.
## Decision
**1. The node-engine judges its machine's networking every 30 s, in five parts:**
- **resolv-conf**: `/etc/resolv.conf` is, byte for byte apart from surrounding whitespace, the file the
uplink's holder declared and the engine last applied. A file nothing declares whole is not judged.
- **names**: every resolver the file lists (up to three, as the C library reads them) answers the mesh's
name with an address, its IPv6 question with "none" and never "no such name" (issue 262), and a public
name with an address. Each answer must come within the time the file tells the C library to wait. The
mesh's name is the bus's own, the name the machine needs most. The public name is one reserved for
documentation, which belongs to no installation.
- **tunnel**: the mesh's interface has handshaken with the hub within five minutes. On the hub, any peer has.
- **bus**: the link to the bus is open.
- **route**: the machine has a default route.
The resolvers are asked concurrently, so a look costs one wait however many are silent. A part that cannot be
judged on the machine, such as a machine with no tunnel tool, is left out, not failed.
**2. The two-look rule is the engine's.** A part is unhealthy on its second failing look in a row, and
healthy on its first passing one. One unanswered datagram is not a finding (issue 277). Each part says
its reason in words that hold no address, path or domain. The detail, with addresses, is evidence and stays
inside the mesh ([ADR 0234](0234-the-mesh-holds-a-conversation-with-its-operator.md) §6).
**3. An outside writer is named where the machine shows it, and said as a guess when it is one.** In order:
1. the file's own header, since every writer puts its name there;
2. a backup beside the file named for its writer, or one changed when the file was;
3. a program known to write the file, running now, said with a question mark.
A link put in place of the file names what it points at. Nothing found is said as nothing found.
**4. The statement carries it.** The engine's health statement (ADR 0240 §4) gains the machine's network:
its worst state, since when, and each part with its reason, evidence, writer, the module whose file it is,
and what a failure points at (the hub, or each resolver's address). It travels in every report and with the
health event, on the same subject, under the same grant. An engine older than this says no network, and
the controller reads that as not known: never healthy, never raised.
**5. The controller raises three kinds, from every machine's newest statement together:**
- **`machine.<m>.<owner>.rewritten`**: the file the module `<owner>` writes on `<m>` was rewritten by
another program, naming it, and what it costs (names that do not resolve) "until the node-engine writes it
back at its next reconcile, or that program gives it back". The names failing through the rewritten file
are this finding's, not raised again.
- **`machine.<m>.network`**: what is the machine's own, meaning its route, its tunnel, its bus, or a
resolver that is no mesh machine's.
- **`machine.<x>.unreachable`**: what points at another machine is said once, there, as ADR 0240 rule 5
says a provider. A failure toward the hub or toward a mesh resolver is held under that machine when it is
down on the record: its silence is open, its own network is unhealthy, or a second machine finds the same.
The waiting machines are listed there and raise nothing of their own. When that machine's own network
condition is open, they are listed on it. One machine alone failing toward a healthy one is its own.
Each is a warning. It is urgent on the control node or the hub, when the bus cannot be reached, or when other
machines wait on it. It clears on the first statement that no longer says it. `node show` lists each part.
**6. The gate waits on what is shown to be another's.** A send whose machine raises
`…rewritten` for a file the send did not move, or `…unreachable` for another machine, waits: no pass, and
no failure at the bound. A `…rewritten` naming a module the send moved is that module's, under issue 281's
rule. A machine's own `…network` holds the machine as a whole, as before.
**7. The engine reads and never acts** (ADR 0240 rule 6). It does not write the file back sooner, restart a
link, or ask for a reconcile. The reconcile holds the file as it always has. How the laptop should share
its names with a VPN client is [research 033](../01-RESEARCH/033-split-dns-with-a-vpn-client/00-overview.md)'s
question, and is not decided here.
## Consequences
- **The VPN rewriting the laptop's file is said within about a minute,** naming FortiClient from its
header. It clears when the reconcile writes the file back. While research 033 is open it is raised on every
connect. That is the point: the fight between the two writers is now visible, not silent.
- **A resolver failing from one machine is that machine's; from two, the resolver's.** The control node's
probe stays, and the machines' own looks now cover the paths it cannot see.
- **About six datagrams per resolver and three small reads, every 30 s, on every machine.** A silent
resolver costs one wait per look (the file's timeout, one second on the mesh's own file).
- **Harder:** a writer the engine cannot name is said as "another program". The mesh cannot see who wrote a
file after the fact, only what the machine shows.
- **Harder:** a machine whose file nothing declares whole (an adopted machine, one whose uplink holder does
not write it) has its names judged but not its file.
- **The gate's waiting has no bound,** as it already has none for a provider down. A rewrite that never
ends keeps the laptop's sends waiting, and the condition says why.
## What this does not decide
- How the laptop resolves both the mesh's names and the VPN's (research 033).
- Whether a healer writes the file back sooner, under [ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md).
- A judging of other mesh-owned files for outside writers. This record judges the one that broke. Another
is its own decision, made the same way when one breaks.
## How it is checked
| Rule | Checked by |
|---|---|
| 1 The five parts | mesh-host `internal/network` tests: a healthy machine is healthy on its first look; the file rewritten is a finding; a resolver answering NXDOMAIN for a mesh name's IPv6 address is a finding naming that resolver; a stale handshake with the hub, the bus unlinked and no default route are each said; on the hub, one fresh peer is a healthy tunnel; a file nothing declares and a machine without the tunnel tool are not judged; a real UDP resolver raised by the test is read for an address, for none and for no such name |
| 2 Two looks | the same tests: one failing look is not a finding, a pass between two failures resets it, and recovery is healthy on the first passing look; the reason never holds an address the evidence holds |
| 3 The writer named | tests naming FortiClient from its header, from its backup beside the file (kept with the old file's time, as its rename leaves it), a running VPN client as a guess, and systemd-resolved from a link put in place of the file |
| 4 The statement | mesh-host `cmd/mesh-host` tests: the file judged is the one a module declares whole (not a seed); the mesh name asked is the bus's; the statement carries the network and an engine with no network judge says none. The controller keeps it in `node_health.network` (migration 0077) |
| 5 Three kinds, said once | mesh-controller `machine_network_test.go`: the rewrite is one finding naming its writer and its cost, with no address in its summary; one machine failing toward a healthy hub is its own; two are one condition at the hub, urgent, listing both; a silent hub holds it; a hub whose own network is unhealthy lists who cannot reach it; a mesh resolver failing from one machine is that machine's and from two is the resolver's machine's; the control node and the bus are urgent; an engine that says no network raises nothing; raised from the statement through the store and cleared when the file is written back |
| 6 The gate | the same file: a rewrite the send did not make waits; one of the file a moved module owns is that module's; the machine's own network holds the machine as a whole; a machine that cannot reach a down hub waits |
| 7 Reads only | the network package holds no write, restart or reconcile, and its tests run against files and fakes alone |
| drill | mesh-host's `TestDrill…`, run in a throwaway container: declared, rewritten as the VPN client rewrites it, written back: healthy, healthy after one failing look, unhealthy naming FortiClient and the names it costs, healthy. Its statements are replayed by mesh-controller's `TestTheDrillsStatementsRaiseAndClearTheRewrite`, which raises the rewrite on the fourth statement alone and clears it on the fifth |
| live | after rollout, the node-engine first and then the controller: every machine's `node show` lists its network's five parts (four where there is no tunnel tool); the laptop's next VPN connect raises `machine.<laptop>.<uplink holder>.rewritten` naming FortiClient, and the reconcile that writes it back clears it |
## References
- [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md), which this extends from
modules to their machine. Its rules 2, 4, 5 and 6 are the shape here.
- [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md) and
[ADR 0117](0117-a-machines-uplink-is-a-seat.md): whose file it is, and what it lists.
- [Research 033](../01-RESEARCH/033-split-dns-with-a-vpn-client/00-overview.md): the VPN client's behaviour,
measured, and how the laptop could share names with it.
- [To-be 48](../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md) §10, the design.
- Issues 262, 277 and 281.
- mesh-host `internal/network`, `cmd/mesh-host`; mesh-controller `cmd/mesh-controller/machine_network.go`,
`gate.go`, migration 0077.
02-DECISIONS/README.md
+1
View File
@@ -339,6 +339,7 @@ python3 00-META/checks/index.py fail if stale
- **0233** — [A module declares the data it holds, and the mesh protects and watches it from that declaration](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md)
- **0235** — [The bus is backed up by its own snapshot of each stream, taken under the bus module's account](0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md)
- **0240** — [A module says how it is healthy, and the node-engine judges it](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md)
- **0241** — [A machine says how its network is, and an outside writer of a mesh file is a finding](0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md)
- **0242** — [A recorded build moves only by a person's push, and a send says what it recreates](0242-a-recorded-build-moves-only-by-a-persons-push-and-a-send-says-what-it-recreates.md) *(proposed)*
- **0243** — [The agent module removes a home item it did not place only on the person's word, and keeps a copy](0243-the-agent-module-removes-a-home-item-it-did-not-place-only-on-the-persons-word-and-keeps-a-copy.md)
03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md
+67
View File
@@ -5,6 +5,7 @@ code: [mesh-host, mesh-controller, mesh-lab, mesh-catalog]
updated: 2026-10-07
decisions:
- 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md
- 02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md
- 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md
- 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
---
@@ -218,6 +219,47 @@ nothing long-lived and need no declaration.
**Until then, and for ever for a module running nothing long-lived:** liveness of everything it runs that
stays up, and the gate's points as they stand.
## 10. The machine under the modules: its network
([ADR 0241](../../02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md).)
Every module on a machine uses its network, and none of them owns it. When a VPN client rewrote the laptop's
resolver file, every module's check stayed green while no mesh name resolved. So the node-engine also judges
the machine itself, beside its liveness looks, and says it in the same statement.
```
node-engine, every 30 s controller, from every machine's newest statement
┌──────────────────────────────────────────────┐ ┌────────────────────────────────────────────────┐
│ resolv-conf the uplink holder's file, as │ │ rewritten machine.<m>.<owner>.rewritten │
│ applied? if not, who wrote it │ statement│ (names failing are its cost) │
│ names each listed resolver: mesh name, │ ───────► │ own machine.<m>.network │
│ its IPv6 "none", a public name │ │ points at x held at x when x is down on the │
│ tunnel handshake with the hub │ │ record or a second machine agrees: │
│ bus the link open │ │ machine.<x>.unreachable, said once │
│ route a default route │ │ gate another's: waits; its own: as 281 │
│ two looks to unhealthy, one back to healthy │ └────────────────────────────────────────────────┘
└──────────────────────────────────────────────┘
```
- **Five parts, each cheap.** The file is compared with the one the uplink's holder declared
([ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)). The names
are asked of every resolver the file lists *now*, whoever wrote it, so the finding is what the machine's
programs see. The answer must come within the wait the file itself gives the C library. A mesh name's IPv6
question must be answered "none", never "no such name"
([issue 262](../../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md)).
The tunnel is the newest handshake with the hub, the bus is the engine's own link, and the route is the
routing table's default.
- **The writer is named where the machine shows it**: the file's own header, a backup named for its writer,
or a known writer running (said as a guess). A link in place of the file names what it points at.
- **Said once.** The names failing through a rewritten file are the rewrite's cost, not a second condition.
A failure toward another machine is held there, as a consumer's under its provider (§6), when that machine
is down on the record or a second machine finds the same.
- **Severity.** Warning; urgent on the control node or the hub, when the bus cannot be reached, or when
others wait on it.
- **The gate.** A rewrite of a file the send did not move, and another machine unreachable, make the judging
*wait*. The machine's own network fault holds the machine as a whole
([issue 281](../../04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine/00-report.md)).
- **It reads and never acts** (§7). The reconcile writes the file back as it always has.
## Phases
| Phase | Repository | Delivers | Done when |
@@ -229,6 +271,7 @@ stays up, and the gate's points as they stand.
| C — the provider hold | mesh-controller | `needs` read against the provider composed for the consumer; the consumer's finding held under the provider's condition; the consumer's gate waiting | the rule 5 test (one provider, three consumers, one condition) passes |
| D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine |
| E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses |
| F — the machine's network | mesh-host (node-engine), mesh-controller | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile |
Phases A and B may be built together; A is live first, because it judges without a single declaration.
@@ -338,6 +381,30 @@ Still to do for these phases' "done when": each read on the live mesh once rolle
node-engine, then the controller, then the catalogue's declarations; the count's first live reading; and
the bed's first run on a catalogue change at the build seat.
## As built — Phase F
One pull request in each of mesh-host and mesh-controller. What the build chose where this design left it
open:
- **The look.** Every 30 s, on the liveness loop's tick. The resolvers are asked concurrently, three
questions each, with a reply read over raw UDP so "no such name" and "none" are told apart. The tunnel is
read with the tunnel tool's handshake and allowed-address listings, never its dump, which holds the
private key. The hub is the peer whose allowed addresses are a range. The judge keeps nothing on disk.
Its two looks start afresh when the engine starts.
- **The statement.** A `network` beside `resources` in the health statement. The controller reads the
statement leniently, so the engine rolls out first and the controller after, and an older controller
ignores the field. The controller keeps it in a column of the machine's health row, added by a numbered
migration.
- **The keys.** The rewrite is keyed by machine and owner, so a send that moved the owner is held on it by
the rule the gate already had for a condition naming a moved module.
- **The drill.** A test of the engine's network package, run in a throwaway container, rewrites the
container's file as the VPN client does and writes it back. Its five statements are recorded, with the
mesh's names replaced by the test mesh's, and replayed by the controller's test through its store and
keeper.
Still to do for its "done when": the live reading once rolled out in order: the node-engine, then the
controller.
## What is not decided here
- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231).