diff --git a/02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md b/02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md new file mode 100644 index 00000000..cde4dc73 --- /dev/null +++ b/02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md @@ -0,0 +1,186 @@ +--- +topic: what runs on it +status: accepted +date: 2026-10-07 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md +--- + +# 241. A machine says how its network is, and an outside writer of a mesh file is a finding + +## Context + +[ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) has the node-engine +judge everything a module runs. It judges nothing *under* the modules: the machine's own networking, which +every module on it uses and none of them owns. + +**On the laptop, a corporate VPN client rewrites `/etc/resolv.conf` when it connects.** It moves the file +the uplink's holder wrote ([ADR 0117](0117-a-machines-uplink-is-a-seat.md), +[ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)) aside and writes its own, naming +its own resolvers. Mesh names then fail on the machine and in its containers, and agents saw "no such host" +for a public service they call. The node-engine writes the mesh's file back at its next reconcile, which ends +the VPN's names for the rest of the session; measured for +[research 033](../01-RESEARCH/033-split-dns-with-a-vpn-client/00-overview.md), in nine of nine VPN sessions +since the node-engine's journal begins, one second to three and a half minutes after each connect. **Nothing +said any of it.** Every module's check was green, because no check asked the machine. + +Two more facts of the same kind: one mesh resolver answered slowly under load +([issue 277](../04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md)), +and the self-check's resolver probe asks the resolvers only from the control node, so a machine that cannot +reach them is not seen. An Alpine container failed a mesh name because a resolver answered "no such name" +for its IPv6 address +([issue 262](../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md)). + +The operator's direction, 2026-10-07: *the health check should, from now on, also catch DNS issues or other +networking issues.* + +**Checked against GENESIS.** *Failure must be loud*: a machine whose names stop resolving while every +module reads healthy is quiet failure. *The mesh notices when something is wrong before you do*: the +operator found this by reconnecting the VPN three times in sixteen minutes. *The mesh is a guest on a +personal node*: on the laptop the VPN client is the employer's, so the mesh says what it finds and repairs +nothing it does not own. Nothing here conflicts with GENESIS. + +## Considered Options + +**Who looks.** + +1. *The controller's self-check probes each machine's resolvers from the control node.* Rejected: this is + what exists, and it is blind to the machine that cannot reach them. A machine's names are a fact on that + machine. +2. **The node-engine on each machine looks at its own networking, beside its liveness looks.** Chosen. It + already reads the machine for ADR 0240, and it applied the file it compares. + +**What is looked at.** Rejected: a full network test (throughput, latency to every machine), which costs +more than it tells. Chosen: five cheap parts, each a fact some incident needed (Decision §1). + +**What an outside writer is.** + +1. *A cause of the names failing, said only in the names' finding.* Rejected: it hides the actionable fact. + The operator can do something about another program owning a mesh file, and cannot do anything about + names failing. +2. **Its own finding, naming the writer where it can.** Chosen. The names that fail through the rewritten + file are listed as what it costs, not raised a second time. + +**What the gate does with a network condition.** Under +[issue 281](../04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine/00-report.md), +a machine-level condition raised since the send holds the machine as a whole, and it fails at the bound. + +1. *Leave it so.* Rejected for the outside writer: a VPN connecting during a send would put back a good build. +2. *Ignore network conditions in the gate.* Rejected: a send that breaks the machine's network is exactly + what the gate exists to catch. +3. **What is shown to be another's waits; what is the machine's own stays the machine's.** Chosen. + +## Decision + +**1. The node-engine judges its machine's networking every 30 s, in five parts:** + +- **resolv-conf**: `/etc/resolv.conf` is, byte for byte apart from surrounding whitespace, the file the + uplink's holder declared and the engine last applied. A file nothing declares whole is not judged. +- **names**: every resolver the file lists (up to three, as the C library reads them) answers the mesh's + name with an address, its IPv6 question with "none" and never "no such name" (issue 262), and a public + name with an address. Each answer must come within the time the file tells the C library to wait. The + mesh's name is the bus's own, the name the machine needs most. The public name is one reserved for + documentation, which belongs to no installation. +- **tunnel**: the mesh's interface has handshaken with the hub within five minutes. On the hub, any peer has. +- **bus**: the link to the bus is open. +- **route**: the machine has a default route. + +The resolvers are asked concurrently, so a look costs one wait however many are silent. A part that cannot be +judged on the machine, such as a machine with no tunnel tool, is left out, not failed. + +**2. The two-look rule is the engine's.** A part is unhealthy on its second failing look in a row, and +healthy on its first passing one. One unanswered datagram is not a finding (issue 277). Each part says +its reason in words that hold no address, path or domain. The detail, with addresses, is evidence and stays +inside the mesh ([ADR 0234](0234-the-mesh-holds-a-conversation-with-its-operator.md) §6). + +**3. An outside writer is named where the machine shows it, and said as a guess when it is one.** In order: + +1. the file's own header, since every writer puts its name there; +2. a backup beside the file named for its writer, or one changed when the file was; +3. a program known to write the file, running now, said with a question mark. + +A link put in place of the file names what it points at. Nothing found is said as nothing found. + +**4. The statement carries it.** The engine's health statement (ADR 0240 §4) gains the machine's network: +its worst state, since when, and each part with its reason, evidence, writer, the module whose file it is, +and what a failure points at (the hub, or each resolver's address). It travels in every report and with the +health event, on the same subject, under the same grant. An engine older than this says no network, and +the controller reads that as not known: never healthy, never raised. + +**5. The controller raises three kinds, from every machine's newest statement together:** + +- **`machine...rewritten`**: the file the module `` writes on `` was rewritten by + another program, naming it, and what it costs (names that do not resolve) "until the node-engine writes it + back at its next reconcile, or that program gives it back". The names failing through the rewritten file + are this finding's, not raised again. +- **`machine..network`**: what is the machine's own, meaning its route, its tunnel, its bus, or a + resolver that is no mesh machine's. +- **`machine..unreachable`**: what points at another machine is said once, there, as ADR 0240 rule 5 + says a provider. A failure toward the hub or toward a mesh resolver is held under that machine when it is + down on the record: its silence is open, its own network is unhealthy, or a second machine finds the same. + The waiting machines are listed there and raise nothing of their own. When that machine's own network + condition is open, they are listed on it. One machine alone failing toward a healthy one is its own. + +Each is a warning. It is urgent on the control node or the hub, when the bus cannot be reached, or when other +machines wait on it. It clears on the first statement that no longer says it. `node show` lists each part. + +**6. The gate waits on what is shown to be another's.** A send whose machine raises +`…rewritten` for a file the send did not move, or `…unreachable` for another machine, waits: no pass, and +no failure at the bound. A `…rewritten` naming a module the send moved is that module's, under issue 281's +rule. A machine's own `…network` holds the machine as a whole, as before. + +**7. The engine reads and never acts** (ADR 0240 rule 6). It does not write the file back sooner, restart a +link, or ask for a reconcile. The reconcile holds the file as it always has. How the laptop should share +its names with a VPN client is [research 033](../01-RESEARCH/033-split-dns-with-a-vpn-client/00-overview.md)'s +question, and is not decided here. + +## Consequences + +- **The VPN rewriting the laptop's file is said within about a minute,** naming FortiClient from its + header. It clears when the reconcile writes the file back. While research 033 is open it is raised on every + connect. That is the point: the fight between the two writers is now visible, not silent. +- **A resolver failing from one machine is that machine's; from two, the resolver's.** The control node's + probe stays, and the machines' own looks now cover the paths it cannot see. +- **About six datagrams per resolver and three small reads, every 30 s, on every machine.** A silent + resolver costs one wait per look (the file's timeout, one second on the mesh's own file). +- **Harder:** a writer the engine cannot name is said as "another program". The mesh cannot see who wrote a + file after the fact, only what the machine shows. +- **Harder:** a machine whose file nothing declares whole (an adopted machine, one whose uplink holder does + not write it) has its names judged but not its file. +- **The gate's waiting has no bound,** as it already has none for a provider down. A rewrite that never + ends keeps the laptop's sends waiting, and the condition says why. + +## What this does not decide + +- How the laptop resolves both the mesh's names and the VPN's (research 033). +- Whether a healer writes the file back sooner, under [ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md). +- A judging of other mesh-owned files for outside writers. This record judges the one that broke. Another + is its own decision, made the same way when one breaks. + +## How it is checked + +| Rule | Checked by | +|---|---| +| 1 The five parts | mesh-host `internal/network` tests: a healthy machine is healthy on its first look; the file rewritten is a finding; a resolver answering NXDOMAIN for a mesh name's IPv6 address is a finding naming that resolver; a stale handshake with the hub, the bus unlinked and no default route are each said; on the hub, one fresh peer is a healthy tunnel; a file nothing declares and a machine without the tunnel tool are not judged; a real UDP resolver raised by the test is read for an address, for none and for no such name | +| 2 Two looks | the same tests: one failing look is not a finding, a pass between two failures resets it, and recovery is healthy on the first passing look; the reason never holds an address the evidence holds | +| 3 The writer named | tests naming FortiClient from its header, from its backup beside the file (kept with the old file's time, as its rename leaves it), a running VPN client as a guess, and systemd-resolved from a link put in place of the file | +| 4 The statement | mesh-host `cmd/mesh-host` tests: the file judged is the one a module declares whole (not a seed); the mesh name asked is the bus's; the statement carries the network and an engine with no network judge says none. The controller keeps it in `node_health.network` (migration 0077) | +| 5 Three kinds, said once | mesh-controller `machine_network_test.go`: the rewrite is one finding naming its writer and its cost, with no address in its summary; one machine failing toward a healthy hub is its own; two are one condition at the hub, urgent, listing both; a silent hub holds it; a hub whose own network is unhealthy lists who cannot reach it; a mesh resolver failing from one machine is that machine's and from two is the resolver's machine's; the control node and the bus are urgent; an engine that says no network raises nothing; raised from the statement through the store and cleared when the file is written back | +| 6 The gate | the same file: a rewrite the send did not make waits; one of the file a moved module owns is that module's; the machine's own network holds the machine as a whole; a machine that cannot reach a down hub waits | +| 7 Reads only | the network package holds no write, restart or reconcile, and its tests run against files and fakes alone | +| drill | mesh-host's `TestDrill…`, run in a throwaway container: declared, rewritten as the VPN client rewrites it, written back: healthy, healthy after one failing look, unhealthy naming FortiClient and the names it costs, healthy. Its statements are replayed by mesh-controller's `TestTheDrillsStatementsRaiseAndClearTheRewrite`, which raises the rewrite on the fourth statement alone and clears it on the fifth | +| live | after rollout, the node-engine first and then the controller: every machine's `node show` lists its network's five parts (four where there is no tunnel tool); the laptop's next VPN connect raises `machine...rewritten` naming FortiClient, and the reconcile that writes it back clears it | + +## References + +- [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md), which this extends from + modules to their machine. Its rules 2, 4, 5 and 6 are the shape here. +- [ADR 0223](0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md) and + [ADR 0117](0117-a-machines-uplink-is-a-seat.md): whose file it is, and what it lists. +- [Research 033](../01-RESEARCH/033-split-dns-with-a-vpn-client/00-overview.md): the VPN client's behaviour, + measured, and how the laptop could share names with it. +- [To-be 48](../03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md) §10, the design. +- Issues 262, 277 and 281. +- mesh-host `internal/network`, `cmd/mesh-host`; mesh-controller `cmd/mesh-controller/machine_network.go`, + `gate.go`, migration 0077. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 2ccd561e..d1fdc25e 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -339,6 +339,7 @@ python3 00-META/checks/index.py fail if stale - **0233** — [A module declares the data it holds, and the mesh protects and watches it from that declaration](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md) - **0235** — [The bus is backed up by its own snapshot of each stream, taken under the bus module's account](0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md) - **0240** — [A module says how it is healthy, and the node-engine judges it](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) +- **0241** — [A machine says how its network is, and an outside writer of a mesh file is a finding](0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md) - **0242** — [A recorded build moves only by a person's push, and a send says what it recreates](0242-a-recorded-build-moves-only-by-a-persons-push-and-a-send-says-what-it-recreates.md) *(proposed)* - **0243** — [The agent module removes a home item it did not place only on the person's word, and keeps a copy](0243-the-agent-module-removes-a-home-item-it-did-not-place-only-on-the-persons-word-and-keeps-a-copy.md) diff --git a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md index 72d8c27d..e8eb17a6 100644 --- a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md +++ b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md @@ -5,6 +5,7 @@ code: [mesh-host, mesh-controller, mesh-lab, mesh-catalog] updated: 2026-10-07 decisions: - 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md + - 02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md - 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md --- @@ -218,6 +219,47 @@ nothing long-lived and need no declaration. **Until then, and for ever for a module running nothing long-lived:** liveness of everything it runs that stays up, and the gate's points as they stand. +## 10. The machine under the modules: its network + +([ADR 0241](../../02-DECISIONS/0241-a-machine-says-how-its-network-is-and-an-outside-writer-of-a-mesh-file-is-a-finding.md).) +Every module on a machine uses its network, and none of them owns it. When a VPN client rewrote the laptop's +resolver file, every module's check stayed green while no mesh name resolved. So the node-engine also judges +the machine itself, beside its liveness looks, and says it in the same statement. + +``` + node-engine, every 30 s controller, from every machine's newest statement + ┌──────────────────────────────────────────────┐ ┌────────────────────────────────────────────────┐ + │ resolv-conf the uplink holder's file, as │ │ rewritten machine...rewritten │ + │ applied? if not, who wrote it │ statement│ (names failing are its cost) │ + │ names each listed resolver: mesh name, │ ───────► │ own machine..network │ + │ its IPv6 "none", a public name │ │ points at x held at x when x is down on the │ + │ tunnel handshake with the hub │ │ record or a second machine agrees: │ + │ bus the link open │ │ machine..unreachable, said once │ + │ route a default route │ │ gate another's: waits; its own: as 281 │ + │ two looks to unhealthy, one back to healthy │ └────────────────────────────────────────────────┘ + └──────────────────────────────────────────────┘ +``` + +- **Five parts, each cheap.** The file is compared with the one the uplink's holder declared + ([ADR 0223](../../02-DECISIONS/0223-the-mesh-has-two-resolvers-and-a-machine-lists-only-them.md)). The names + are asked of every resolver the file lists *now*, whoever wrote it, so the finding is what the machine's + programs see. The answer must come within the wait the file itself gives the C library. A mesh name's IPv6 + question must be answered "none", never "no such name" + ([issue 262](../../04-ISSUES/262-an-alpine-container-could-not-find-a-machine-by-its-mesh-name/00-report.md)). + The tunnel is the newest handshake with the hub, the bus is the engine's own link, and the route is the + routing table's default. +- **The writer is named where the machine shows it**: the file's own header, a backup named for its writer, + or a known writer running (said as a guess). A link in place of the file names what it points at. +- **Said once.** The names failing through a rewritten file are the rewrite's cost, not a second condition. + A failure toward another machine is held there, as a consumer's under its provider (§6), when that machine + is down on the record or a second machine finds the same. +- **Severity.** Warning; urgent on the control node or the hub, when the bus cannot be reached, or when + others wait on it. +- **The gate.** A rewrite of a file the send did not move, and another machine unreachable, make the judging + *wait*. The machine's own network fault holds the machine as a whole + ([issue 281](../../04-ISSUES/281-a-tier-sent-one-module-at-a-time-blamed-a-module-for-its-machine/00-report.md)). +- **It reads and never acts** (§7). The reconcile writes the file back as it always has. + ## Phases | Phase | Repository | Delivers | Done when | @@ -229,6 +271,7 @@ stays up, and the gate's points as they stand. | C — the provider hold | mesh-controller | `needs` read against the provider composed for the consumer; the consumer's finding held under the provider's condition; the consumer's gate waiting | the rule 5 test (one provider, three consumers, one condition) passes | | D — the proof | mesh-lab, mesh-catalog | the bed; the catalogue's check starting every changed resource on it; adopted image checks proved | the replay of the studio's false *unhealthy* fails the bed, not a machine | | E — the migration | mesh-catalog, mesh-controller | the declarations of §9 steps 2–5; the count kept in the catalogue and its test; `module check` refusing after the date | the count is zero, or the date has passed and `module check` refuses | +| F — the machine's network | mesh-host (node-engine), mesh-controller | §10: the five parts judged on two looks and stated; the three conditions said once; the gate waiting on another's; `node show` | the tests of ADR 0241 pass; the drill's replay raises and clears; a VPN connect on the laptop is raised naming its writer and cleared by the reconcile | Phases A and B may be built together; A is live first, because it judges without a single declaration. @@ -338,6 +381,30 @@ Still to do for these phases' "done when": each read on the live mesh once rolle node-engine, then the controller, then the catalogue's declarations; the count's first live reading; and the bed's first run on a catalogue change at the build seat. +## As built — Phase F + +One pull request in each of mesh-host and mesh-controller. What the build chose where this design left it +open: + +- **The look.** Every 30 s, on the liveness loop's tick. The resolvers are asked concurrently, three + questions each, with a reply read over raw UDP so "no such name" and "none" are told apart. The tunnel is + read with the tunnel tool's handshake and allowed-address listings, never its dump, which holds the + private key. The hub is the peer whose allowed addresses are a range. The judge keeps nothing on disk. + Its two looks start afresh when the engine starts. +- **The statement.** A `network` beside `resources` in the health statement. The controller reads the + statement leniently, so the engine rolls out first and the controller after, and an older controller + ignores the field. The controller keeps it in a column of the machine's health row, added by a numbered + migration. +- **The keys.** The rewrite is keyed by machine and owner, so a send that moved the owner is held on it by + the rule the gate already had for a condition naming a moved module. +- **The drill.** A test of the engine's network package, run in a throwaway container, rewrites the + container's file as the VPN client does and writes it back. Its five statements are recorded, with the + mesh's names replaced by the test mesh's, and replayed by the controller's test through its store and + keeper. + +Still to do for its "done when": the live reading once rolled out in order: the node-engine, then the +controller. + ## What is not decided here - Whether a healer restarts what stays unhealthy (its own record, under ADR 0231).