Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9de25994e9 | ||
|
|
64ea47b11d | ||
|
|
eba24a72af | ||
|
|
78d4873f4f |
@@ -1,10 +1,11 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
status: superseded
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0010-delivery.md
|
||||
superseded-by: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md
|
||||
---
|
||||
|
||||
# 143. A consumer verifies the grant it is given
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md
|
||||
supersedes: 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md
|
||||
---
|
||||
|
||||
# 144. Anything on a machine may call anything on it, and that is the whole of "local"
|
||||
|
||||
## Context
|
||||
|
||||
Everything in the mesh should be able to call:
|
||||
|
||||
- what runs on the same machine;
|
||||
- another machine's service over the private network, if that service is exposed there;
|
||||
- another machine's service over the public network, if it is exposed there.
|
||||
|
||||
Three cases. The filter had two of them.
|
||||
|
||||
**The first was broken and the break was invisible.** A service exposed to the private network rendered
|
||||
as the machines' own addresses on it. A caller on the machine carries such an address; a caller inside
|
||||
one of that machine's containers carries a bridge address and matched nothing. Measured:
|
||||
|
||||
```
|
||||
the machine: local 10.10.0.1 dev lo src 10.10.0.1
|
||||
a container: 10.10.0.1 via 172.17.0.1 dev eth0 src 172.17.0.8
|
||||
```
|
||||
|
||||
Same destination, same machine, two source addresses. The rule named the first and silently refused the
|
||||
second, so a module reaching its database on its own machine's name timed out for eleven hours
|
||||
([issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)).
|
||||
|
||||
**The second case works, and by accident.** A caller on another machine reaches the private network over
|
||||
the tunnel, and arrives carrying that machine's own address — so the rule matches. It would not have
|
||||
matched the caller's own address either; the tunnel rewrites it. That two of three cases worked is why
|
||||
this looked correct.
|
||||
|
||||
**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) answered the wrong question.** Written
|
||||
hours earlier, it proposed that a consumer verify each grant it is given by opening a connection from
|
||||
its own network position — and it went to some length about *which* position, because whether a caller
|
||||
sat in a container changed the answer. That difference was the bug. A verification mechanism would have
|
||||
reported this outage sooner and would not have prevented it, and the machinery it needed existed only
|
||||
because the rule was wrong. The remedy for a configuration error is the correct configuration.
|
||||
|
||||
**And a module is not a container.** A module is software that delivers one or more services, and it may
|
||||
do that as a container, an installed package with a unit, a binary, or files something else reads. Of 72
|
||||
modules in the catalogue, 61 happen to use a container and 11 do not — among them the resolver, the ssh
|
||||
daemon and the intrusion-prevention module. A rule that reasons about containers describes most of the
|
||||
mesh and not the mesh.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **A line per service admitting the machine's own callers.** Rejected: it is what was written first,
|
||||
and it only ever covers the services somebody remembered to think about. It also states, service by
|
||||
service, a thing that is true of the machine.
|
||||
2. **Verify each grant from the consumer's position** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)).
|
||||
Rejected as a remedy: it observes the fault rather than removing it, and the question it agonised over
|
||||
— which network position — exists only while the fault does.
|
||||
3. **Enumerate the addresses a machine's callers may have.** Rejected for the reason no address is named
|
||||
anywhere in this filter any more ([ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md)):
|
||||
a range describes one machine and goes stale in silence.
|
||||
4. **Local is not filtered, stated once.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**Anything on a machine may call anything on that machine, and the filter says so once.** Not per
|
||||
service, not per port, and not by naming who the callers are: traffic that did not arrive from outside
|
||||
the machine and did not arrive over the private network is the machine's own, and is admitted. It is
|
||||
asked by the link the traffic arrived on, because that is a fact about the machine rather than a list
|
||||
that describes one.
|
||||
|
||||
**Local is not a boundary this mesh draws.** Whether a caller is a container, a unit, or the operator's
|
||||
shell changes nothing, because the thing being decided is "is this the same machine" and the answer does
|
||||
not depend on the form the caller takes.
|
||||
|
||||
**The other two cases are unchanged and are now legible beside it.** A service exposed to the private
|
||||
network admits the machines on it; a service exposed publicly admits anything. Three cases, three lines,
|
||||
and a reader can see all three at once.
|
||||
|
||||
**[ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) is superseded and nothing replaces it.**
|
||||
Whether the mesh should check that a grant works is a real question — it reported this machine healthy
|
||||
for eleven hours — but it is a question about what the mesh can say, not about what it should do, and it
|
||||
must stand on its own rather than as the remedy for a rule that was wrong. It is not built.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The three things everything should be able to call are three lines**, and the first is one line
|
||||
rather than one per service, so a service added tomorrow is reachable locally without anybody
|
||||
remembering to say so.
|
||||
- **A form of module stops mattering to the filter.** The 11 modules that are not containers were never
|
||||
affected by this bug and were never the reason it was hard to see; they are the reason the rule should
|
||||
never have mentioned containers.
|
||||
- **The mesh still cannot say when a grant stops working.** That is the live gap, recorded in issue 145
|
||||
and no longer pretending to have an answer.
|
||||
- **What got harder:** nothing. This removes a line per service and replaces it with one.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **A caller on the machine reaches a service on it, in the input chain**, asserted on that chain's own
|
||||
body — because the forward chain carries the same line in the same words, and an assertion on the
|
||||
whole rendered file passed with the input chain's copy deleted. That is what
|
||||
[ADR 0137](0137-a-machine-says-which-networks-it-routes.md)'s tests already say to do.
|
||||
- **It is one rule, not one per service.** Asserted by rendering two services of different reach and
|
||||
refusing a per-port local line.
|
||||
- **The three reaches render as three lines**, asserted together, so the whole of what the filter says
|
||||
about who may call what is one test.
|
||||
- **The measured case:** from a container on the machine, a service exposed to the private network on
|
||||
that machine answers. This is the outage, and it fails against the rule this replaces.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) — the filter is the sum
|
||||
of what its modules listen on
|
||||
- [ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md) — why no address is named
|
||||
- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public,
|
||||
the other two cases
|
||||
- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded here
|
||||
- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
topic: what runs on it
|
||||
status: accepted
|
||||
date: 2026-09-29
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md
|
||||
---
|
||||
|
||||
# 145. A module checks what the mesh claims is reachable, and it checks itself
|
||||
|
||||
## Context
|
||||
|
||||
The mesh asserts three things are callable ([ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md)):
|
||||
what runs on the same machine, another machine's service exposed to the private network, and another
|
||||
machine's service exposed publicly. It has never checked any of them.
|
||||
|
||||
[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md):
|
||||
the first of the three was broken for eleven hours and the mesh answered *all heard from, every module
|
||||
current with its source* throughout. Every check it makes is about the relationship between the mesh and
|
||||
a machine — applied, current, containers running — and none about whether anything can reach anything.
|
||||
|
||||
**A first answer was drafted and withdrawn.** [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)
|
||||
put the check inside the host, verifying each grant from the consumer's network position. It was
|
||||
superseded because the difference it worked so hard to reproduce — whether a caller sat in a container —
|
||||
was the bug itself. What survives from it is the part that was right: a check run from the wrong place
|
||||
proves nothing, and the mesh's own reports are not evidence about the network.
|
||||
|
||||
**The mesh already has the shape for this and it is a module.** A module can declare a container that
|
||||
runs on a cadence ([ADR 0053](0053-a-step-that-runs-on-a-schedule.md), and three modules already use
|
||||
`*/5 * * * *`), can be given the mesh's roster as a rendered fact — every machine's name, address and
|
||||
this node's own identity, the same mechanism the resolver and the operator's ssh configuration use — and
|
||||
can emit what it found on the bus. Nothing new is needed to build this except the module.
|
||||
|
||||
**What it must not check is the trap.** The obvious probe target is ssh: present on every machine, never
|
||||
closed by design. Dialling it would have passed throughout the outage, because ssh is admitted
|
||||
unconditionally and the thing that broke was a service exposed to the private network. A checker whose
|
||||
probe is unconditionally open measures the one path that cannot fail, which is the failure this whole
|
||||
sequence keeps producing — a check that reads as verification and verifies nothing.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **The host verifies each grant** ([ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md)).
|
||||
Superseded. It needed the host to act from another network position, which is machinery that exists
|
||||
only while local calls are filtered wrongly.
|
||||
2. **The control plane dials every node.** Rejected: it sits on one machine and reaches the others by a
|
||||
path no ordinary caller uses. It would have passed throughout the outage.
|
||||
3. **Probe an existing service.** Rejected for the target problem above: the services guaranteed on every
|
||||
machine are the ones that are never closed, so they cannot fail the way the mesh fails.
|
||||
4. **A module on every machine that serves its own probe and dials the others'.** Adopted.
|
||||
|
||||
## Decision
|
||||
|
||||
**A module runs on every machine, serves an endpoint of its own, and dials every other machine's.** The
|
||||
probe is the module's own endpoint, declared reachable over the private network — so the thing being
|
||||
dialled is admitted by exactly the rule that governs every other internally-exposed service, and fails
|
||||
when that rule is wrong. A second endpoint, declared public, does the same for the public path where a
|
||||
machine has one.
|
||||
|
||||
**It checks the three cases the mesh claims, by name:**
|
||||
|
||||
- its **own machine**, by dialling its own machine's address — the case that broke, and the only one that
|
||||
distinguishes a caller on the machine from a caller in one of its containers;
|
||||
- **each other machine over the private network**;
|
||||
- **each machine's public path**, where one is recorded.
|
||||
|
||||
**It resolves before it dials, and says which failed.** A name that does not resolve and a port that does
|
||||
not answer are different faults with different owners, and a checker that reports one sentence for both
|
||||
sends a reader to the wrong place.
|
||||
|
||||
**It runs where the callers run.** The module's own code in its own container, on the cadence the mesh
|
||||
already has, from the same position as every other module on that machine. It is not the host and not the
|
||||
control plane, and that is the whole point.
|
||||
|
||||
**It says what it found and nothing else.** It emits results; it repairs nothing, opens nothing and holds
|
||||
no credential beyond its own. A checker that fixes things is a second control plane.
|
||||
|
||||
**One failure is not a fault.** A machine rebooting is ordinary. A path is reported broken after it has
|
||||
failed on consecutive runs, and the count travels with the result so a reader can tell "briefly away"
|
||||
from "never worked" — the one thing [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) got
|
||||
right and worth keeping.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The mesh gains the ability to be wrong out loud about the network.** Eleven hours becomes two runs.
|
||||
- **It is a module, so it is assigned, built, pushed and reported on like everything else** — no new host
|
||||
capability, no new vocabulary, nothing in the control plane that has to know about checking.
|
||||
- **Its own endpoint is the instrument.** That is what makes it able to fail; it also means the checker
|
||||
must be assigned to a machine before that machine can be checked, and a machine without it is
|
||||
unchecked rather than healthy.
|
||||
- **It cannot check what it cannot be told.** The roster gives it machines; it does not give it every
|
||||
module's endpoints, so this checks the paths the mesh claims and not every grant in the mesh. That is
|
||||
the honest scope of a first one, and the difference is worth saying rather than growing quietly.
|
||||
- **What got harder:** one more module on every machine, and a module whose whole purpose is to fail
|
||||
visibly when something else is wrong. Its own failures will be read as the mesh's, which is the cost of
|
||||
an instrument.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- **It catches the measured outage.** A bed closes the path from a container to a service exposed to the
|
||||
private network on its own machine — issue 145's shape — and the checker reports its own machine
|
||||
unreachable while every other path still reads reachable. This fails against a probe on a port that is
|
||||
never closed, which is the wrong target this record exists to name.
|
||||
- **A machine rebooting is not a fault**: one failed run reports nothing, the count rises and falls.
|
||||
- **A name that does not resolve is reported as that**, not as a port that did not answer.
|
||||
- **It reports and does not act**: asserted by giving it a broken path and checking nothing on the machine
|
||||
changed.
|
||||
- **A machine without the module reads unchecked**, never healthy — asserted on what the mesh says about
|
||||
a machine it is not assigned to.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md) — the three things that must be callable
|
||||
- [ADR 0143](0143-a-consumer-verifies-the-grant-it-is-given.md) — superseded; what survives is that the
|
||||
position matters
|
||||
- [ADR 0053](0053-a-step-that-runs-on-a-schedule.md) — the cadence
|
||||
- [ADR 0138](0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md) — internal and public,
|
||||
which the probe endpoints declare
|
||||
- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
|
||||
@@ -223,7 +223,9 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md) *(superseded)*
|
||||
- **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md)
|
||||
- **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)
|
||||
- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md)
|
||||
- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) *(superseded)*
|
||||
- **0144** — [Anything on a machine may call anything on it, and that is the whole of "local"](0144-anything-on-a-machine-may-call-anything-on-it.md)
|
||||
- **0145** — [A module checks what the mesh claims is reachable, and it checks itself](0145-a-module-checks-what-the-mesh-claims-is-reachable.md)
|
||||
|
||||
### How it is built
|
||||
|
||||
|
||||
@@ -7,7 +7,6 @@ code:
|
||||
- mesh-controller internal/inventory/builds.go
|
||||
updated: 2026-09-29
|
||||
decisions:
|
||||
- 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md
|
||||
- 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
|
||||
- 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
|
||||
- 02-DECISIONS/0010-delivery.md
|
||||
@@ -228,37 +227,33 @@ id, whatever the words; three make the machine stuck, and `status` says so besid
|
||||
knows, not what the machine is told. *How it is checked:* an inventory test counts three identical
|
||||
reports, a different one, and a clean apply; the status test asserts the word appears.
|
||||
|
||||
## A consumer verifies the grant it is given
|
||||
## Everything may call what is exposed to it, and local is not a boundary
|
||||
|
||||
*2026-09-29, from an outage that ran eleven hours —
|
||||
[issue 145](../../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
|
||||
settled by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md).*
|
||||
settled by [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md).*
|
||||
|
||||
A grant is four facts and a credential: the provision, the machine, the port, and who the consumer is
|
||||
when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and the
|
||||
mesh asserted it without ever finding out whether it was true.
|
||||
when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and it
|
||||
rests on three things being callable — what runs on the same machine, another machine's service over the
|
||||
private network where it is exposed there, and another machine's service over the public network where it
|
||||
is exposed there.
|
||||
|
||||
What that cost: a machine was converged, the path from a container to a port on its own machine closed,
|
||||
and for eleven hours the mesh answered *all heard from, every module current with its source* while a
|
||||
module logged a connection timeout to its database six thousand times. Every check the mesh makes passed,
|
||||
because every one is about the relationship between the mesh and a machine — applied, current, running.
|
||||
None asks whether a consumer can reach what it requires.
|
||||
The filter had two of those. A service exposed to the private network admitted the machines' own addresses
|
||||
on it; a caller on the machine carries such an address, and a caller inside one of that machine's
|
||||
containers carries a bridge address and matched nothing. Measured, same destination and same machine:
|
||||
`src 10.10.0.1` from the machine, `src 172.17.0.8` from a container on it. So a module reaching its
|
||||
database on its own machine's name timed out for eleven hours while the mesh called the machine healthy.
|
||||
|
||||
**So a consumer verifies its own grants, from its own network position.** Not the control plane, which
|
||||
reaches the address by a path no consumer uses; not the machine, whose own packets carried a source
|
||||
address the filter admitted while every container's was refused. The position is the point: a check
|
||||
somewhere else is testing something nobody asked about.
|
||||
The second case worked by accident: a caller on another machine arrives over the tunnel carrying that
|
||||
machine's address, which the rule matched. Two of three working is why this read as correct.
|
||||
|
||||
It opens a connection and nothing more. Whether the credential is right or the schema current is the
|
||||
provider's to answer and the consumer's to discover; a check that spoke each provision's protocol would
|
||||
be a second implementation of every provision.
|
||||
**So local is not a boundary this mesh draws, and the filter says so once.** Traffic that did not arrive
|
||||
from outside the machine and did not arrive over the private network is the machine's own, and is
|
||||
admitted — for every service there, not per service. Whether the caller is a container, a unit or a shell
|
||||
decides nothing, because the question is "is this the same machine".
|
||||
|
||||
One failure is not news — a provider restarting is ordinary — so a grant reads unreachable only after
|
||||
consecutive reconciles, and the machine reports the count rather than the last attempt, which is what
|
||||
separates "briefly away" from "never worked". A consumer that is not running has no position to dial
|
||||
from: that grant reads unchecked, which is a different sentence from broken and must not be written as
|
||||
one.
|
||||
|
||||
*How it is checked* is stated with the decision, and the first of them is the outage itself: a bed drops
|
||||
the path from a consumer's position while leaving the machine's own open, and the grant must read
|
||||
unreachable — which fails both against reporting nothing and against a check run from the machine.
|
||||
A verification mechanism was drafted for this and withdrawn. It would have reported the outage sooner and
|
||||
would not have prevented it, and the part of it that was hard — deciding which network position to check
|
||||
from — existed only because the rule was wrong. Whether the mesh should check that a grant works is still
|
||||
open, in issue 145; it is not the remedy for a configuration error.
|
||||
|
||||
+26
-8
@@ -74,15 +74,33 @@ What is not fixed is that nothing in the mesh would have told anybody.
|
||||
|
||||
## What was decided
|
||||
|
||||
*2026-09-29, the same day.* The first open question below — should a grant be checked, and from where —
|
||||
is answered by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md): the
|
||||
consumer verifies it, from its own network position, because that is the only position that answers what
|
||||
a grant claims. A check run by the control plane or by the machine would have passed throughout this
|
||||
outage, since the rule in force admitted the machines' own addresses and it was the containers that were
|
||||
refused.
|
||||
*2026-09-29, the same day, in two steps and the first was wrong.*
|
||||
|
||||
The remaining questions below stand, and the record answers two of them: one failure is not news, only
|
||||
consecutive ones, and a grant that cannot be checked is reported unchecked rather than assumed good.
|
||||
The first answer was [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md):
|
||||
the consumer verifies each grant from its own network position, because whether a caller sat in a
|
||||
container changed whether it could reach the provider. **That difference was the fault**, and the record
|
||||
is superseded. A verification mechanism would have reported this sooner and would not have prevented it,
|
||||
and the part of it that was difficult — deciding which network position to check from — existed only
|
||||
while the rule was wrong.
|
||||
|
||||
The remedy is [ADR 0144](../../02-DECISIONS/0144-anything-on-a-machine-may-call-anything-on-it.md):
|
||||
anything on a machine may call anything on it, said once rather than per service, and asked by the link
|
||||
traffic arrives on rather than the address it carries. Everything should be able to call what runs on the
|
||||
same machine, another machine's service exposed to the private network, and another machine's service
|
||||
exposed publicly. The filter had the second and third and expressed the first as a list of addresses that
|
||||
no container could match.
|
||||
|
||||
**And then the question this issue is actually about was answered on its own terms.**
|
||||
[ADR 0145](../../02-DECISIONS/0145-a-module-checks-what-the-mesh-claims-is-reachable.md): a module on
|
||||
every machine serves an endpoint of its own and dials every other machine's, from the position the
|
||||
callers are in. Its probe is its own endpoint declared reachable over the private network, so it is
|
||||
admitted by exactly the rule that governs every internally-exposed service and fails when that rule is
|
||||
wrong — where a probe on a service every machine has would have passed for all eleven hours, because the
|
||||
services every machine has are the ones never closed.
|
||||
|
||||
Adopted on its merits rather than as the remedy for a configuration error, which is what 0143 was and
|
||||
why it went. The module is written and merged; it is not yet assigned, so every machine currently reads
|
||||
unchecked.
|
||||
|
||||
## Open questions
|
||||
|
||||
|
||||
Reference in New Issue
Block a user