diff --git a/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md b/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md index bce1367..1e3a3f2 100644 --- a/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md +++ b/02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md @@ -1,10 +1,11 @@ --- topic: what runs on it -status: accepted +status: superseded date: 2026-09-29 deciders: jochen reconstructed: false extends: 02-DECISIONS/0010-delivery.md +superseded-by: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md --- # 143. A consumer verifies the grant it is given diff --git a/02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md b/02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md new file mode 100644 index 0000000..f958e79 --- /dev/null +++ b/02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md @@ -0,0 +1,121 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-29 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md +supersedes: 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md +--- + +# 144. Anything on a machine may call anything on it, and that is the whole of "local" + +## Context + +Everything in the mesh should be able to call: + +- what runs on the same machine; +- another machine's service over the private network, if that service is exposed there; +- another machine's service over the public network, if it is exposed there. + +Three cases. The filter had two of them. + +**The first was broken and the break was invisible.** A service exposed to the private network rendered +as the machines' own addresses on it. A caller on the machine carries such an address; a caller inside +one of that machine's containers carries a bridge address and matched nothing. Measured: + +``` +the machine: local 10.10.0.1 dev lo src 10.10.0.1 +a container: 10.10.0.1 via 172.17.0.1 dev eth0 src 172.17.0.8 +``` + +Same destination, same machine, two source addresses. The rule named the first and silently refused the +second, so a module reaching its database on its own machine's name timed out for eleven hours +([issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)). + +**The second case works, and by accident.** A caller on another machine reaches the private network over +the tunnel, and arrives carrying that machine's own address — so the rule matches. It would not have +matched the caller's own address either; the tunnel rewrites it. That two of three cases worked is why +this looked correct. + +**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) answered the wrong question.** Written +hours earlier, it proposed that a consumer verify each grant it is given by opening a connection from +its own network position — and it went to some length about *which* position, because whether a caller +sat in a container changed the answer. That difference was the bug. A verification mechanism would have +reported this outage sooner and would not have prevented it, and the machinery it needed existed only +because the rule was wrong. The remedy for a configuration error is the correct configuration. + +**And a module is not a container.** A module is software that delivers one or more services, and it may +do that as a container, an installed package with a unit, a binary, or files something else reads. Of 72 +modules in the catalogue, 61 happen to use a container and 11 do not — among them the resolver, the ssh +daemon and the intrusion-prevention module. A rule that reasons about containers describes most of the +mesh and not the mesh. + +## Considered Options + +1. **A line per service admitting the machine's own callers.** Rejected: it is what was written first, + and it only ever covers the services somebody remembered to think about. It also states, service by + service, a thing that is true of the machine. +2. **Verify each grant from the consumer's position** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)). + Rejected as a remedy: it observes the fault rather than removing it, and the question it agonised over + — which network position — exists only while the fault does. +3. **Enumerate the addresses a machine's callers may have.** Rejected for the reason no address is named + anywhere in this filter any more ([ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md)): + a range describes one machine and goes stale in silence. +4. **Local is not filtered, stated once.** Adopted. + +## Decision + +**Anything on a machine may call anything on that machine, and the filter says so once.** Not per +service, not per port, and not by naming who the callers are: traffic that did not arrive from outside +the machine and did not arrive over the private network is the machine's own, and is admitted. It is +asked by the link the traffic arrived on, because that is a fact about the machine rather than a list +that describes one. + +**Local is not a boundary this mesh draws.** Whether a caller is a container, a unit, or the operator's +shell changes nothing, because the thing being decided is "is this the same machine" and the answer does +not depend on the form the caller takes. + +**The other two cases are unchanged and are now legible beside it.** A service exposed to the private +network admits the machines on it; a service exposed publicly admits anything. Three cases, three lines, +and a reader can see all three at once. + +**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) is superseded and nothing replaces it.** +Whether the mesh should check that a grant works is a real question — it reported this machine healthy +for eleven hours — but it is a question about what the mesh can say, not about what it should do, and it +must stand on its own rather than as the remedy for a rule that was wrong. It is not built. + +## Consequences + +- **The three things everything should be able to call are three lines**, and the first is one line + rather than one per service, so a service added tomorrow is reachable locally without anybody + remembering to say so. +- **A form of module stops mattering to the filter.** The 11 modules that are not containers were never + affected by this bug and were never the reason it was hard to see; they are the reason the rule should + never have mentioned containers. +- **The mesh still cannot say when a grant stops working.** That is the live gap, recorded in issue 145 + and no longer pretending to have an answer. +- **What got harder:** nothing. This removes a line per service and replaces it with one. + +## How it is checked + +- **A caller on the machine reaches a service on it, in the input chain**, asserted on that chain's own + body — because the forward chain carries the same line in the same words, and an assertion on the + whole rendered file passed with the input chain's copy deleted. That is what + [ADR 0137](0137-a-machine-says-which-networks-it-routes.md)'s tests already say to do. +- **It is one rule, not one per service.** Asserted by rendering two services of different reach and + refusing a per-port local line. +- **The three reaches render as three lines**, asserted together, so the whole of what the filter says + about who may call what is one test. +- **The measured case:** from a container on the machine, a service exposed to the private network on + that machine answers. This is the outage, and it fails against the rule this replaces. + +## References + +- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the filter is the sum + of what its modules listen on +- [ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md) — why no address is named +- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public, + the other two cases +- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded here +- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md) diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 54a1778..f732795 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -223,7 +223,8 @@ python3 00-META/checks/index.py fail if stale - **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md) *(superseded)* - **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md) - **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md) -- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) +- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) *(superseded)* +- **0144** — [Anything on a machine may call anything on it, and that is the whole of "local"](0144-anything-on-a-machine-may-call-anything-on-it.md) ### How it is built diff --git a/03-DESIGN/01-to-be/10-delivery.md b/03-DESIGN/01-to-be/10-delivery.md index ad80cdf..94f98fc 100644 --- a/03-DESIGN/01-to-be/10-delivery.md +++ b/03-DESIGN/01-to-be/10-delivery.md @@ -7,7 +7,6 @@ code: - mesh-controller internal/inventory/builds.go updated: 2026-09-29 decisions: - - 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md - 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md - 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md - 02-DECISIONS/0010-delivery.md @@ -228,37 +227,33 @@ id, whatever the words; three make the machine stuck, and `status` says so besid knows, not what the machine is told. *How it is checked:* an inventory test counts three identical reports, a different one, and a clean apply; the status test asserts the word appears. -## A consumer verifies the grant it is given +## Everything may call what is exposed to it, and local is not a boundary *2026-09-29, from an outage that ran eleven hours — [issue 145](../../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md), -settled by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md).* +settled by [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md).* A grant is four facts and a credential: the provision, the machine, the port, and who the consumer is -when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and the -mesh asserted it without ever finding out whether it was true. +when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and it +rests on three things being callable — what runs on the same machine, another machine's service over the +private network where it is exposed there, and another machine's service over the public network where it +is exposed there. -What that cost: a machine was converged, the path from a container to a port on its own machine closed, -and for eleven hours the mesh answered *all heard from, every module current with its source* while a -module logged a connection timeout to its database six thousand times. Every check the mesh makes passed, -because every one is about the relationship between the mesh and a machine — applied, current, running. -None asks whether a consumer can reach what it requires. +The filter had two of those. A service exposed to the private network admitted the machines' own addresses +on it; a caller on the machine carries such an address, and a caller inside one of that machine's +containers carries a bridge address and matched nothing. Measured, same destination and same machine: +`src 10.10.0.1` from the machine, `src 172.17.0.8` from a container on it. So a module reaching its +database on its own machine's name timed out for eleven hours while the mesh called the machine healthy. -**So a consumer verifies its own grants, from its own network position.** Not the control plane, which -reaches the address by a path no consumer uses; not the machine, whose own packets carried a source -address the filter admitted while every container's was refused. The position is the point: a check -somewhere else is testing something nobody asked about. +The second case worked by accident: a caller on another machine arrives over the tunnel carrying that +machine's address, which the rule matched. Two of three working is why this read as correct. -It opens a connection and nothing more. Whether the credential is right or the schema current is the -provider's to answer and the consumer's to discover; a check that spoke each provision's protocol would -be a second implementation of every provision. +**So local is not a boundary this mesh draws, and the filter says so once.** Traffic that did not arrive +from outside the machine and did not arrive over the private network is the machine's own, and is +admitted — for every service there, not per service. Whether the caller is a container, a unit or a shell +decides nothing, because the question is "is this the same machine". -One failure is not news — a provider restarting is ordinary — so a grant reads unreachable only after -consecutive reconciles, and the machine reports the count rather than the last attempt, which is what -separates "briefly away" from "never worked". A consumer that is not running has no position to dial -from: that grant reads unchecked, which is a different sentence from broken and must not be written as -one. - -*How it is checked* is stated with the decision, and the first of them is the outage itself: a bed drops -the path from a consumer's position while leaving the machine's own open, and the grant must read -unreachable — which fails both against reporting nothing and against a check run from the machine. +A verification mechanism was drafted for this and withdrawn. It would have reported the outage sooner and +would not have prevented it, and the part of it that was hard — deciding which network position to check +from — existed only because the rule was wrong. Whether the mesh should check that a grant works is still +open, in issue 145; it is not the remedy for a configuration error. diff --git a/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md b/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md index 0ffddcc..2a1e122 100644 --- a/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md +++ b/04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md @@ -74,15 +74,25 @@ What is not fixed is that nothing in the mesh would have told anybody. ## What was decided -*2026-09-29, the same day.* The first open question below — should a grant be checked, and from where — -is answered by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md): the -consumer verifies it, from its own network position, because that is the only position that answers what -a grant claims. A check run by the control plane or by the machine would have passed throughout this -outage, since the rule in force admitted the machines' own addresses and it was the containers that were -refused. +*2026-09-29, the same day, in two steps and the first was wrong.* -The remaining questions below stand, and the record answers two of them: one failure is not news, only -consecutive ones, and a grant that cannot be checked is reported unchecked rather than assumed good. +The first answer was [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md): +the consumer verifies each grant from its own network position, because whether a caller sat in a +container changed whether it could reach the provider. **That difference was the fault**, and the record +is superseded. A verification mechanism would have reported this sooner and would not have prevented it, +and the part of it that was difficult — deciding which network position to check from — existed only +while the rule was wrong. + +The remedy is [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md): +anything on a machine may call anything on it, said once rather than per service, and asked by the link +traffic arrives on rather than the address it carries. Everything should be able to call what runs on the +same machine, another machine's service exposed to the private network, and another machine's service +exposed publicly. The filter had the second and third and expressed the first as a list of addresses that +no container could match. + +**The question this issue is actually about is still open.** Nothing in the mesh would have said a grant +had stopped working, and nothing does now. That is not answered by a configuration fix, and it should not +be answered by a mechanism adopted as a remedy for one. ## Open questions