From eba24a72af07f35dc15a7e606539307f1adbe1b7 Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 29 Sep 2026 13:32:01 +0200 Subject: [PATCH] ADR 0144: anything on a machine may call anything on it, superseding 0143 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Everything should be able to call what runs on the same machine, another machine's service exposed to the private network, and another machine's service exposed publicly. Three cases; the filter had two. The first was expressed as the machines' own addresses on the private network. A caller on the machine carries such an address; a caller in one of its containers carries a bridge address and matched nothing — measured, same destination and same machine: src 10.10.0.1 against src 172.17.0.8. The second case worked by accident, because the tunnel rewrites a caller's address to the sending machine's. Two of three working is why it read as correct. 0143 answered the wrong question. It proposed verifying each grant from the consumer's own network position and went to length about which position, because whether a caller sat in a container changed the answer — and that difference was the bug. Observing a configuration error is not its remedy. Superseded, and nothing replaces it; whether the mesh should check a grant is still open in issue 145 and must stand on its own. And a module is not a container: 61 of 72 happen to use one, 11 do not, and a rule reasoning about containers describes most of the mesh rather than the mesh. --- ...consumer-verifies-the-grant-it-is-given.md | 3 +- ...ng-on-a-machine-may-call-anything-on-it.md | 121 ++++++++++++++++++ 02-DECISIONS/README.md | 3 +- 03-DESIGN/01-to-be/10-delivery.md | 47 +++---- .../00-report.md | 26 ++-- 5 files changed, 164 insertions(+), 36 deletions(-) create mode 100644 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md diff --git a/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md b/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md index bce1367..1e3a3f2 100644 --- a/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md +++ b/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md @@ -1,10 +1,11 @@ --- topic: what runs on it -status: accepted +status: superseded date: 2026-09-29 deciders: jochen reconstructed: false extends: 02-DECISIONS/0010-delivery.md +superseded-by: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md --- # 143. A consumer verifies the grant it is given diff --git a/02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md b/02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md new file mode 100644 index 0000000..f958e79 --- /dev/null +++ b/02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md @@ -0,0 +1,121 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-29 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md +supersedes: 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md +--- + +# 144. Anything on a machine may call anything on it, and that is the whole of "local" + +## Context + +Everything in the mesh should be able to call: + +- what runs on the same machine; +- another machine's service over the private network, if that service is exposed there; +- another machine's service over the public network, if it is exposed there. + +Three cases. The filter had two of them. + +**The first was broken and the break was invisible.** A service exposed to the private network rendered +as the machines' own addresses on it. A caller on the machine carries such an address; a caller inside +one of that machine's containers carries a bridge address and matched nothing. Measured: + +``` +the machine: local 10.10.0.1 dev lo src 10.10.0.1 +a container: 10.10.0.1 via 172.17.0.1 dev eth0 src 172.17.0.8 +``` + +Same destination, same machine, two source addresses. The rule named the first and silently refused the +second, so a module reaching its database on its own machine's name timed out for eleven hours +([issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)). + +**The second case works, and by accident.** A caller on another machine reaches the private network over +the tunnel, and arrives carrying that machine's own address — so the rule matches. It would not have +matched the caller's own address either; the tunnel rewrites it. That two of three cases worked is why +this looked correct. + +**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) answered the wrong question.** Written +hours earlier, it proposed that a consumer verify each grant it is given by opening a connection from +its own network position — and it went to some length about *which* position, because whether a caller +sat in a container changed the answer. That difference was the bug. A verification mechanism would have +reported this outage sooner and would not have prevented it, and the machinery it needed existed only +because the rule was wrong. The remedy for a configuration error is the correct configuration. + +**And a module is not a container.** A module is software that delivers one or more services, and it may +do that as a container, an installed package with a unit, a binary, or files something else reads. Of 72 +modules in the catalogue, 61 happen to use a container and 11 do not — among them the resolver, the ssh +daemon and the intrusion-prevention module. A rule that reasons about containers describes most of the +mesh and not the mesh. + +## Considered Options + +1. **A line per service admitting the machine's own callers.** Rejected: it is what was written first, + and it only ever covers the services somebody remembered to think about. It also states, service by + service, a thing that is true of the machine. +2. **Verify each grant from the consumer's position** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)). + Rejected as a remedy: it observes the fault rather than removing it, and the question it agonised over + — which network position — exists only while the fault does. +3. **Enumerate the addresses a machine's callers may have.** Rejected for the reason no address is named + anywhere in this filter any more ([ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md)): + a range describes one machine and goes stale in silence. +4. **Local is not filtered, stated once.** Adopted. + +## Decision + +**Anything on a machine may call anything on that machine, and the filter says so once.** Not per +service, not per port, and not by naming who the callers are: traffic that did not arrive from outside +the machine and did not arrive over the private network is the machine's own, and is admitted. It is +asked by the link the traffic arrived on, because that is a fact about the machine rather than a list +that describes one. + +**Local is not a boundary this mesh draws.** Whether a caller is a container, a unit, or the operator's +shell changes nothing, because the thing being decided is "is this the same machine" and the answer does +not depend on the form the caller takes. + +**The other two cases are unchanged and are now legible beside it.** A service exposed to the private +network admits the machines on it; a service exposed publicly admits anything. Three cases, three lines, +and a reader can see all three at once. + +**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) is superseded and nothing replaces it.** +Whether the mesh should check that a grant works is a real question — it reported this machine healthy +for eleven hours — but it is a question about what the mesh can say, not about what it should do, and it +must stand on its own rather than as the remedy for a rule that was wrong. It is not built. + +## Consequences + +- **The three things everything should be able to call are three lines**, and the first is one line + rather than one per service, so a service added tomorrow is reachable locally without anybody + remembering to say so. +- **A form of module stops mattering to the filter.** The 11 modules that are not containers were never + affected by this bug and were never the reason it was hard to see; they are the reason the rule should + never have mentioned containers. +- **The mesh still cannot say when a grant stops working.** That is the live gap, recorded in issue 145 + and no longer pretending to have an answer. +- **What got harder:** nothing. This removes a line per service and replaces it with one. + +## How it is checked + +- **A caller on the machine reaches a service on it, in the input chain**, asserted on that chain's own + body — because the forward chain carries the same line in the same words, and an assertion on the + whole rendered file passed with the input chain's copy deleted. That is what + [ADR 0137](0137-a-machine-says-which-networks-it-routes.md)'s tests already say to do. +- **It is one rule, not one per service.** Asserted by rendering two services of different reach and + refusing a per-port local line. +- **The three reaches render as three lines**, asserted together, so the whole of what the filter says + about who may call what is one test. +- **The measured case:** from a container on the machine, a service exposed to the private network on + that machine answers. This is the outage, and it fails against the rule this replaces. + +## References + +- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the filter is the sum + of what its modules listen on +- [ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md) — why no address is named +- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public, + the other two cases +- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded here +- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md) diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 54a1778..f732795 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -223,7 +223,8 @@ python3 00-META/checks/index.py fail if stale - **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md) *(superseded)* - **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md) - **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md) -- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) +- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) *(superseded)* +- **0144** — [Anything on a machine may call anything on it, and that is the whole of "local"](0144-anything-on-a-machine-may-call-anything-on-it.md) ### How it is built diff --git a/03-DESIGN/01-to-be/10-delivery.md b/03-DESIGN/01-to-be/10-delivery.md index ad80cdf..94f98fc 100644 --- a/03-DESIGN/01-to-be/10-delivery.md +++ b/03-DESIGN/01-to-be/10-delivery.md @@ -7,7 +7,6 @@ code: - mesh-controller internal/inventory/builds.go updated: 2026-09-29 decisions: - - 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md - 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md - 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md - 02-DECISIONS/0010-delivery.md @@ -228,37 +227,33 @@ id, whatever the words; three make the machine stuck, and `status` says so besid knows, not what the machine is told. *How it is checked:* an inventory test counts three identical reports, a different one, and a clean apply; the status test asserts the word appears. -## A consumer verifies the grant it is given +## Everything may call what is exposed to it, and local is not a boundary *2026-09-29, from an outage that ran eleven hours — [issue 145](../../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md), -settled by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md).* +settled by [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md).* A grant is four facts and a credential: the provision, the machine, the port, and who the consumer is -when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and the -mesh asserted it without ever finding out whether it was true. +when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and it +rests on three things being callable — what runs on the same machine, another machine's service over the +private network where it is exposed there, and another machine's service over the public network where it +is exposed there. -What that cost: a machine was converged, the path from a container to a port on its own machine closed, -and for eleven hours the mesh answered *all heard from, every module current with its source* while a -module logged a connection timeout to its database six thousand times. Every check the mesh makes passed, -because every one is about the relationship between the mesh and a machine — applied, current, running. -None asks whether a consumer can reach what it requires. +The filter had two of those. A service exposed to the private network admitted the machines' own addresses +on it; a caller on the machine carries such an address, and a caller inside one of that machine's +containers carries a bridge address and matched nothing. Measured, same destination and same machine: +`src 10.10.0.1` from the machine, `src 172.17.0.8` from a container on it. So a module reaching its +database on its own machine's name timed out for eleven hours while the mesh called the machine healthy. -**So a consumer verifies its own grants, from its own network position.** Not the control plane, which -reaches the address by a path no consumer uses; not the machine, whose own packets carried a source -address the filter admitted while every container's was refused. The position is the point: a check -somewhere else is testing something nobody asked about. +The second case worked by accident: a caller on another machine arrives over the tunnel carrying that +machine's address, which the rule matched. Two of three working is why this read as correct. -It opens a connection and nothing more. Whether the credential is right or the schema current is the -provider's to answer and the consumer's to discover; a check that spoke each provision's protocol would -be a second implementation of every provision. +**So local is not a boundary this mesh draws, and the filter says so once.** Traffic that did not arrive +from outside the machine and did not arrive over the private network is the machine's own, and is +admitted — for every service there, not per service. Whether the caller is a container, a unit or a shell +decides nothing, because the question is "is this the same machine". -One failure is not news — a provider restarting is ordinary — so a grant reads unreachable only after -consecutive reconciles, and the machine reports the count rather than the last attempt, which is what -separates "briefly away" from "never worked". A consumer that is not running has no position to dial -from: that grant reads unchecked, which is a different sentence from broken and must not be written as -one. - -*How it is checked* is stated with the decision, and the first of them is the outage itself: a bed drops -the path from a consumer's position while leaving the machine's own open, and the grant must read -unreachable — which fails both against reporting nothing and against a check run from the machine. +A verification mechanism was drafted for this and withdrawn. It would have reported the outage sooner and +would not have prevented it, and the part of it that was hard — deciding which network position to check +from — existed only because the rule was wrong. Whether the mesh should check that a grant works is still +open, in issue 145; it is not the remedy for a configuration error. diff --git a/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md b/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md index 0ffddcc..2a1e122 100644 --- a/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md +++ b/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md @@ -74,15 +74,25 @@ What is not fixed is that nothing in the mesh would have told anybody. ## What was decided -*2026-09-29, the same day.* The first open question below — should a grant be checked, and from where — -is answered by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md): the -consumer verifies it, from its own network position, because that is the only position that answers what -a grant claims. A check run by the control plane or by the machine would have passed throughout this -outage, since the rule in force admitted the machines' own addresses and it was the containers that were -refused. +*2026-09-29, the same day, in two steps and the first was wrong.* -The remaining questions below stand, and the record answers two of them: one failure is not news, only -consecutive ones, and a grant that cannot be checked is reported unchecked rather than assumed good. +The first answer was [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md): +the consumer verifies each grant from its own network position, because whether a caller sat in a +container changed whether it could reach the provider. **That difference was the fault**, and the record +is superseded. A verification mechanism would have reported this sooner and would not have prevented it, +and the part of it that was difficult — deciding which network position to check from — existed only +while the rule was wrong. + +The remedy is [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md): +anything on a machine may call anything on it, said once rather than per service, and asked by the link +traffic arrives on rather than the address it carries. Everything should be able to call what runs on the +same machine, another machine's service exposed to the private network, and another machine's service +exposed publicly. The filter had the second and third and expressed the first as a list of addresses that +no container could match. + +**The question this issue is actually about is still open.** Nothing in the mesh would have said a grant +had stopped working, and nothing does now. That is not answered by a configuration fix, and it should not +be answered by a mechanism adopted as a remedy for one. ## Open questions -- 2.54.0