diff --git a/02-DECISIONS/0137-a-machine-says-which-networks-it-routes.md b/02-DECISIONS/0137-a-machine-says-which-networks-it-routes.md new file mode 100644 index 0000000..2c3904e --- /dev/null +++ b/02-DECISIONS/0137-a-machine-says-which-networks-it-routes.md @@ -0,0 +1,116 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-28 +deciders: jochen +extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md +reconstructed: false +--- + +# 137. A machine says which networks it routes + +## Context + +The filter the mesh derives denies forwarding by default, because without a forward chain it says +nothing about a container's published port +([ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md), [issue 047](../04-ISSUES/047-the-firewall-does-not-cover-published-container-ports/00-report.md)). +To keep a machine's own containers working it then allows two ranges: the container runtime's +default bridge pool, and the pool its compose files are given. Those two are named in the +controller's code, with a comment saying what the gap is: + +> A machine whose runtime is configured with something else needs this to say so — which is a thing +> the mesh cannot derive and a reason this list is named here rather than computed. + +**There was no way to say so.** The list was a constant. A machine whose guests live anywhere else +was filtered by a rule that looked deliberate and was a guess. + +**Measured, on the day a workstation was converged.** Flipping it cut egress for five of its +container networks at once, and for every network its test beds create — the beds allocate a fresh +range per run, from a pool neither default covers. Nothing reported a fault. The containers could +not reach anything, the machine went on reporting that it had applied what it was told, and the +converge preview had said nothing about it either, because the preview lists what *listens* and +routing is not a listener. + +**And two questions, not one.** A guest also asks its host for an address and for names. Both arrive +at the input chain, where nothing declared them, so denying by default left the guests of a routed +network with no address and no resolution — which is not a closed port but a network that does not +function, asked for by this machine's own guest. + +**Why the machine cannot simply be read.** A test bed creates its bridge while it runs, between one +declaration and the next, so a filter derived from what the machine last reported would be correct +only for the networks that already existed when it was composed. A declared range covers the ones +that do not exist yet. + +## Decision + +**A machine says which networks it routes for what it hosts, and the filter forwards them.** A +node-level fact, beside the node's public domain +([ADR 0066](0066-public-routing-is-name-agnostic.md)) and for the same reason: the +machine routes them, and the module that loads the filter holds a seat and may be replaced. + +**Added to the runtime's defaults, never replacing them.** A machine that names one range has not +stopped hosting whatever was already on the runtime's own pools, and replacing would trade one +silent breakage for another. + +**Their guests keep address and name service.** For a network that was named, the input chain admits +that network's own DHCP and DNS, and nothing else: everything else a guest might want from its host +is a port somebody declares, like every other port on this machine. + +**Said in CIDR form and checked when it is said.** An entry that does not parse is a line nftables +refuses, and a refused ruleset is a machine filtering nothing while its unit reports a fault — so +the refusal happens where a person can read it, not on the machine. + +**A machine that says nothing is filtered exactly as before.** Every machine already converged is +untouched by this. + +## Options considered + +1. **Leave it constant and edit the code per installation.** Rejected: the value is a property of + one machine, the code is the whole mesh's, and the two ranges as they stand describe a machine + whose runtime was left at its defaults. It is also how this got here. +2. **Derive it from what the machine reports.** Rejected as insufficient, not as wrong: it cannot + cover a network created between two declarations, which is precisely the case that was broken. It + would also make the filter follow whatever appeared on the machine, which is a firewall that + widens itself. +3. **A per-node setting on the module that loads the filter.** Rejected: the machine routes the + networks. The filter module holds a node-scoped seat and is meant to be replaceable, and a + replacement must not lose the machine's own truth. +4. **Replace the defaults with what is said.** Rejected: see the decision. The first machine to name + its bed range would lose its containers. +5. **Admit all input from a routed network, not only address and name service.** Rejected: that is + every port on the machine open to anything it hosts, which is the derivation abandoned. + +## Consequences + +**The converge preview says what a machine routes**, including when it routes nothing but the +defaults, with the command that changes it. The preview's own sentence about traffic it cannot +preview stays, because a tunnel and the found firewall's NAT are still not previewable. + +**A machine whose guests are already broken by an earlier flip is fixed by saying its networks and +pushing**, with no flip to undo. + +**The list is one more thing that can be wrong and stale.** A range removed from the machine and +left here keeps forwarding for a network that no longer exists, which admits nothing, because there +is no guest on it to admit. That is the safe direction of being out of date. + +## How this is checked + +- **What a machine says it routes is forwarded, and its guests keep address and name service.** A + test renders a ruleset for a machine that names one range and asserts both chains, per chain body + so a line in the wrong chain cannot pass it. It fails against the previous behaviour, which is how + it was written. +- **The runtime's own defaults survive naming a range.** Asserted in the same test. +- **A machine that names nothing renders byte-identically to one that names nil**, so every machine + already behind this filter is untouched. +- **Each family is matched in its own syntax.** A test with one v4 and one v6 network asserts + `ip saddr` and `ip6 saddr`, because one set holding both is a syntax error and a ruleset that does + not load is a machine filtering nothing. +- **An entry that is not a network is refused where it is said**, by the parse in the setter. + +## References + +- [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) — the derived filter this completes +- [ADR 0066](0066-public-routing-is-name-agnostic.md) — the precedent for a node-level fact +- [issue 047](../04-ISSUES/047-the-firewall-does-not-cover-published-container-ports/00-report.md) — why there is a forward chain at all +- [issue 137](../04-ISSUES/137-converging-a-machine-cut-off-its-own-guests/00-report.md) — the measurement that produced this +- mesh-controller `internal/catalogue/filtering.go` — the constant whose own comment named this gap diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index be0909d..cac9d25 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -217,6 +217,7 @@ python3 00-META/checks/index.py fail if stale - **0133** — [A module owns its migrations, and the mesh owns when they run](0133-a-module-owns-its-migrations-and-the-mesh-owns-when-they-run.md) *(superseded)* - **0135** — [A module version prepares its state before it runs](0135-a-module-version-prepares-its-state-before-it-runs.md) - **0136** — [A step gates its module, not the machine](0136-a-step-gates-its-module-not-the-machine.md) +- **0137** — [A machine says which networks it routes](0137-a-machine-says-which-networks-it-routes.md) ### How it is built diff --git a/04-ISSUES/137-converging-a-machine-cut-off-its-own-guests/00-report.md b/04-ISSUES/137-converging-a-machine-cut-off-its-own-guests/00-report.md new file mode 100644 index 0000000..c1921e7 --- /dev/null +++ b/04-ISSUES/137-converging-a-machine-cut-off-its-own-guests/00-report.md @@ -0,0 +1,74 @@ +--- +status: resolved +opened: 2026-09-28 +located-in: [mesh-controller internal/catalogue] +fixed-by: mesh-controller — a machine says which networks it routes and the derived filter forwards them, their guests keeping address and name service ([ADR 0137](../../02-DECISIONS/0137-a-machine-says-which-networks-it-routes.md)). +amended-design: +--- + +# 137 — Converging a machine cut off its own guests, and nothing said so + +## What was observed + +A workstation was flipped from adopted to converged, so the mesh's derived filter replaced what was +there. The flip reported success, the machine reported that it had applied its declaration, and every +surface of the mesh read green. + +A container on one of that machine's networks could no longer reach anything: + +``` +192.168.64.2/20 +OUTBOUND BLOCKED +``` + +Five of the machine's container networks were affected, and every network its test beds create. The +reason is in the filter's forward chain, which denies by default and then allows two ranges: + +``` +ip saddr 172.16.0.0/12 accept # the container runtime's bridge networks +ip saddr 192.168.128.0/17 accept # the networks its compose files are given +``` + +Those two are constants in the controller. The machine's guests were allocated from neither: its +compose networks from other parts of `192.168/16`, and each test bed a fresh `10.x/24`. So the rules +were correct for a machine whose runtime was left at its defaults, and a guess on this one. + +Two further things were closed by the same flip, and for the same reason nobody saw them: a guest asks +its host for an address over DHCP and for names over DNS, both of which arrive at the input chain, +where no module had declared them. + +## Why it matters beyond this instance + +**The preview could not have warned.** It lists what the machine reported as *listening*, and says so +honestly: it ends with a line that traffic the machine routes is "not previewed". What it did not say +is that routing was about to be denied by default, or which ranges would survive. An operator reading +a 350-line preview approves what it shows. + +**It is the second time today that a constant stood in for something the mesh cannot know.** The +intrusion-prevention module named a firewall front-end two machines do not have +([issue 136](../136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md)), and the +filter names the address ranges one runtime happens to use. Both were true where they were written and +silently false elsewhere. + +**And the code already knew.** The comment above those two lines says a machine configured otherwise +"needs this to say so — which is a thing the mesh cannot derive and a reason this list is named here +rather than computed". The gap was documented at the point where it was introduced, and the way to say +it was never built. A comment naming a missing mechanism is a rule that is not enforced. + +## What was done + +A machine says which networks it routes; the filter forwards them and admits their guests' address and +name service. Added to the runtime's defaults rather than replacing them, so a machine that names one +range keeps the others. Node-level, because the machine routes them and the module that loads the +filter may be replaced. The converge preview now says what a machine routes, and what it will keep +forwarding if it says nothing. + +## What is still true + +**The flip is still the moment a machine's unmanaged services close.** That is what converging means +and the preview names each one. This issue is not about the ports that were meant to close; it is about +the ones nothing could name. + +**Egress is still not previewed per network.** The preview says which ranges will be forwarded, not +which of the machine's guests sit inside them. Deriving that would need the machine to report its +bridges, and a bed's bridge does not exist until the bed runs.