ADR 0144: anything on a machine may call anything on it #181
@@ -1,10 +1,11 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
status: superseded
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0010-delivery.md
|
||||
superseded-by: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md
|
||||
---
|
||||
|
||||
# 143. A consumer verifies the grant it is given
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md
|
||||
supersedes: 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md
|
||||
---
|
||||
|
||||
# 144. Anything on a machine may call anything on it, and that is the whole of "local"
|
||||
|
||||
## Context
|
||||
|
||||
Everything in the mesh should be able to call:
|
||||
|
||||
- what runs on the same machine;
|
||||
- another machine's service over the private network, if that service is exposed there;
|
||||
- another machine's service over the public network, if it is exposed there.
|
||||
|
||||
Three cases. The filter had two of them.
|
||||
|
||||
**The first was broken and the break was invisible.** A service exposed to the private network rendered
|
||||
as the machines' own addresses on it. A caller on the machine carries such an address; a caller inside
|
||||
one of that machine's containers carries a bridge address and matched nothing. Measured:
|
||||
|
||||
```
|
||||
the machine: local 10.10.0.1 dev lo src 10.10.0.1
|
||||
a container: 10.10.0.1 via 172.17.0.1 dev eth0 src 172.17.0.8
|
||||
```
|
||||
|
||||
Same destination, same machine, two source addresses. The rule named the first and silently refused the
|
||||
second, so a module reaching its database on its own machine's name timed out for eleven hours
|
||||
([issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)).
|
||||
|
||||
**The second case works, and by accident.** A caller on another machine reaches the private network over
|
||||
the tunnel, and arrives carrying that machine's own address — so the rule matches. It would not have
|
||||
matched the caller's own address either; the tunnel rewrites it. That two of three cases worked is why
|
||||
this looked correct.
|
||||
|
||||
**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) answered the wrong question.** Written
|
||||
hours earlier, it proposed that a consumer verify each grant it is given by opening a connection from
|
||||
its own network position — and it went to some length about *which* position, because whether a caller
|
||||
sat in a container changed the answer. That difference was the bug. A verification mechanism would have
|
||||
reported this outage sooner and would not have prevented it, and the machinery it needed existed only
|
||||
because the rule was wrong. The remedy for a configuration error is the correct configuration.
|
||||
|
||||
**And a module is not a container.** A module is software that delivers one or more services, and it may
|
||||
do that as a container, an installed package with a unit, a binary, or files something else reads. Of 72
|
||||
modules in the catalogue, 61 happen to use a container and 11 do not — among them the resolver, the ssh
|
||||
daemon and the intrusion-prevention module. A rule that reasons about containers describes most of the
|
||||
mesh and not the mesh.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A line per service admitting the machine's own callers.** Rejected: it is what was written first,
|
||||
and it only ever covers the services somebody remembered to think about. It also states, service by
|
||||
service, a thing that is true of the machine.
|
||||
2. **Verify each grant from the consumer's position** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)).
|
||||
Rejected as a remedy: it observes the fault rather than removing it, and the question it agonised over
|
||||
— which network position — exists only while the fault does.
|
||||
3. **Enumerate the addresses a machine's callers may have.** Rejected for the reason no address is named
|
||||
anywhere in this filter any more ([ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md)):
|
||||
a range describes one machine and goes stale in silence.
|
||||
4. **Local is not filtered, stated once.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**Anything on a machine may call anything on that machine, and the filter says so once.** Not per
|
||||
service, not per port, and not by naming who the callers are: traffic that did not arrive from outside
|
||||
the machine and did not arrive over the private network is the machine's own, and is admitted. It is
|
||||
asked by the link the traffic arrived on, because that is a fact about the machine rather than a list
|
||||
that describes one.
|
||||
|
||||
**Local is not a boundary this mesh draws.** Whether a caller is a container, a unit, or the operator's
|
||||
shell changes nothing, because the thing being decided is "is this the same machine" and the answer does
|
||||
not depend on the form the caller takes.
|
||||
|
||||
**The other two cases are unchanged and are now legible beside it.** A service exposed to the private
|
||||
network admits the machines on it; a service exposed publicly admits anything. Three cases, three lines,
|
||||
and a reader can see all three at once.
|
||||
|
||||
**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) is superseded and nothing replaces it.**
|
||||
Whether the mesh should check that a grant works is a real question — it reported this machine healthy
|
||||
for eleven hours — but it is a question about what the mesh can say, not about what it should do, and it
|
||||
must stand on its own rather than as the remedy for a rule that was wrong. It is not built.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The three things everything should be able to call are three lines**, and the first is one line
|
||||
rather than one per service, so a service added tomorrow is reachable locally without anybody
|
||||
remembering to say so.
|
||||
- **A form of module stops mattering to the filter.** The 11 modules that are not containers were never
|
||||
affected by this bug and were never the reason it was hard to see; they are the reason the rule should
|
||||
never have mentioned containers.
|
||||
- **The mesh still cannot say when a grant stops working.** That is the live gap, recorded in issue 145
|
||||
and no longer pretending to have an answer.
|
||||
- **What got harder:** nothing. This removes a line per service and replaces it with one.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **A caller on the machine reaches a service on it, in the input chain**, asserted on that chain's own
|
||||
body — because the forward chain carries the same line in the same words, and an assertion on the
|
||||
whole rendered file passed with the input chain's copy deleted. That is what
|
||||
[ADR 0137](0137-a-machine-says-which-networks-it-routes.md)'s tests already say to do.
|
||||
- **It is one rule, not one per service.** Asserted by rendering two services of different reach and
|
||||
refusing a per-port local line.
|
||||
- **The three reaches render as three lines**, asserted together, so the whole of what the filter says
|
||||
about who may call what is one test.
|
||||
- **The measured case:** from a container on the machine, a service exposed to the private network on
|
||||
that machine answers. This is the outage, and it fails against the rule this replaces.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the filter is the sum
|
||||
of what its modules listen on
|
||||
- [ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md) — why no address is named
|
||||
- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public,
|
||||
the other two cases
|
||||
- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded here
|
||||
- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
|
||||
@@ -223,7 +223,8 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md) *(superseded)*
|
||||
- **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md)
|
||||
- **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)
|
||||
- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md)
|
||||
- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) *(superseded)*
|
||||
- **0144** — [Anything on a machine may call anything on it, and that is the whole of "local"](0144-anything-on-a-machine-may-call-anything-on-it.md)
|
||||
|
||||
### How it is built
|
||||
|
||||
|
||||
@@ -7,7 +7,6 @@ code:
|
||||
- mesh-controller internal/inventory/builds.go
|
||||
updated: 2026-09-29
|
||||
decisions:
|
||||
- 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md
|
||||
- 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
|
||||
- 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
|
||||
- 02-DECISIONS/0010-delivery.md
|
||||
@@ -228,37 +227,33 @@ id, whatever the words; three make the machine stuck, and `status` says so besid
|
||||
knows, not what the machine is told. *How it is checked:* an inventory test counts three identical
|
||||
reports, a different one, and a clean apply; the status test asserts the word appears.
|
||||
|
||||
## A consumer verifies the grant it is given
|
||||
## Everything may call what is exposed to it, and local is not a boundary
|
||||
|
||||
*2026-09-29, from an outage that ran eleven hours —
|
||||
[issue 145](../../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
|
||||
settled by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md).*
|
||||
settled by [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md).*
|
||||
|
||||
A grant is four facts and a credential: the provision, the machine, the port, and who the consumer is
|
||||
when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and the
|
||||
mesh asserted it without ever finding out whether it was true.
|
||||
when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and it
|
||||
rests on three things being callable — what runs on the same machine, another machine's service over the
|
||||
private network where it is exposed there, and another machine's service over the public network where it
|
||||
is exposed there.
|
||||
|
||||
What that cost: a machine was converged, the path from a container to a port on its own machine closed,
|
||||
and for eleven hours the mesh answered *all heard from, every module current with its source* while a
|
||||
module logged a connection timeout to its database six thousand times. Every check the mesh makes passed,
|
||||
because every one is about the relationship between the mesh and a machine — applied, current, running.
|
||||
None asks whether a consumer can reach what it requires.
|
||||
The filter had two of those. A service exposed to the private network admitted the machines' own addresses
|
||||
on it; a caller on the machine carries such an address, and a caller inside one of that machine's
|
||||
containers carries a bridge address and matched nothing. Measured, same destination and same machine:
|
||||
`src 10.10.0.1` from the machine, `src 172.17.0.8` from a container on it. So a module reaching its
|
||||
database on its own machine's name timed out for eleven hours while the mesh called the machine healthy.
|
||||
|
||||
**So a consumer verifies its own grants, from its own network position.** Not the control plane, which
|
||||
reaches the address by a path no consumer uses; not the machine, whose own packets carried a source
|
||||
address the filter admitted while every container's was refused. The position is the point: a check
|
||||
somewhere else is testing something nobody asked about.
|
||||
The second case worked by accident: a caller on another machine arrives over the tunnel carrying that
|
||||
machine's address, which the rule matched. Two of three working is why this read as correct.
|
||||
|
||||
It opens a connection and nothing more. Whether the credential is right or the schema current is the
|
||||
provider's to answer and the consumer's to discover; a check that spoke each provision's protocol would
|
||||
be a second implementation of every provision.
|
||||
**So local is not a boundary this mesh draws, and the filter says so once.** Traffic that did not arrive
|
||||
from outside the machine and did not arrive over the private network is the machine's own, and is
|
||||
admitted — for every service there, not per service. Whether the caller is a container, a unit or a shell
|
||||
decides nothing, because the question is "is this the same machine".
|
||||
|
||||
One failure is not news — a provider restarting is ordinary — so a grant reads unreachable only after
|
||||
consecutive reconciles, and the machine reports the count rather than the last attempt, which is what
|
||||
separates "briefly away" from "never worked". A consumer that is not running has no position to dial
|
||||
from: that grant reads unchecked, which is a different sentence from broken and must not be written as
|
||||
one.
|
||||
|
||||
*How it is checked* is stated with the decision, and the first of them is the outage itself: a bed drops
|
||||
the path from a consumer's position while leaving the machine's own open, and the grant must read
|
||||
unreachable — which fails both against reporting nothing and against a check run from the machine.
|
||||
A verification mechanism was drafted for this and withdrawn. It would have reported the outage sooner and
|
||||
would not have prevented it, and the part of it that was hard — deciding which network position to check
|
||||
from — existed only because the rule was wrong. Whether the mesh should check that a grant works is still
|
||||
open, in issue 145; it is not the remedy for a configuration error.
|
||||
|
||||
+18
-8
@@ -74,15 +74,25 @@ What is not fixed is that nothing in the mesh would have told anybody.
|
||||
|
||||
## What was decided
|
||||
|
||||
*2026-09-29, the same day.* The first open question below — should a grant be checked, and from where —
|
||||
is answered by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md): the
|
||||
consumer verifies it, from its own network position, because that is the only position that answers what
|
||||
a grant claims. A check run by the control plane or by the machine would have passed throughout this
|
||||
outage, since the rule in force admitted the machines' own addresses and it was the containers that were
|
||||
refused.
|
||||
*2026-09-29, the same day, in two steps and the first was wrong.*
|
||||
|
||||
The remaining questions below stand, and the record answers two of them: one failure is not news, only
|
||||
consecutive ones, and a grant that cannot be checked is reported unchecked rather than assumed good.
|
||||
The first answer was [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md):
|
||||
the consumer verifies each grant from its own network position, because whether a caller sat in a
|
||||
container changed whether it could reach the provider. **That difference was the fault**, and the record
|
||||
is superseded. A verification mechanism would have reported this sooner and would not have prevented it,
|
||||
and the part of it that was difficult — deciding which network position to check from — existed only
|
||||
while the rule was wrong.
|
||||
|
||||
The remedy is [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md):
|
||||
anything on a machine may call anything on it, said once rather than per service, and asked by the link
|
||||
traffic arrives on rather than the address it carries. Everything should be able to call what runs on the
|
||||
same machine, another machine's service exposed to the private network, and another machine's service
|
||||
exposed publicly. The filter had the second and third and expressed the first as a list of addresses that
|
||||
no container could match.
|
||||
|
||||
**The question this issue is actually about is still open.** Nothing in the mesh would have said a grant
|
||||
had stopped working, and nothing does now. That is not answered by a configuration fix, and it should not
|
||||
be answered by a mechanism adopted as a remedy for one.
|
||||
|
||||
## Open questions
|
||||
|
||||
|
||||
Reference in New Issue
Block a user