Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ddfd62edf6 | ||
|
|
f65664640a | ||
|
|
492ac7be18 | ||
|
|
5ac77e3cef | ||
|
|
a619022c35 | ||
|
|
49c065c204 | ||
|
|
ddb3980f09 | ||
|
|
75a8f6abc7 | ||
|
|
862253518f | ||
|
|
51ef3eb7e2 |
@@ -99,6 +99,31 @@ composed, so it is not certified.
|
|||||||
binding. The per-node source override becomes its reach, widened from the filter alone to the names
|
binding. The per-node source override becomes its reach, widened from the filter alone to the names
|
||||||
and the certificate as well.
|
and the certificate as well.
|
||||||
|
|
||||||
|
## Progressive insight — 2026-09-29, from building it
|
||||||
|
|
||||||
|
**Reach does not mean the same thing to the filter for an endpoint the proxy serves.** The decision
|
||||||
|
above says `internal` means "the filter opens the machine port to the private network" and `public`
|
||||||
|
means "the filter opens it to anywhere". For a routed endpoint the second half is wrong, and
|
||||||
|
[ADR 0045](0045-a-machine-firewall-is-the-sum-of-what-it-listens-on.md) already said so before this
|
||||||
|
record was written: *a public service is exposed through the proxy, not by opening its own port* — it
|
||||||
|
listens `from: mesh`, only the proxy reaches it, and it is exposed by name.
|
||||||
|
|
||||||
|
Found by trying to express one real module, not by review. Its routed name must be public, because
|
||||||
|
browsers post to it; its machine-side port must not be, because that port serves the dashboard in
|
||||||
|
cleartext. Under one value driving both, saying "public" would have reopened a port an operator had
|
||||||
|
just closed. Measured the same evening: that module's routed name answered from the internet over TLS
|
||||||
|
while its machine-side port was refused from the same place. The port is not the path.
|
||||||
|
|
||||||
|
So the reach of a **routed** endpoint asks for names, and its port keeps what the manifest said. The
|
||||||
|
reach of an **unrouted** endpoint — git over ssh, a mail port, the bus — governs the port, because
|
||||||
|
there is no name and the port is the only way in. That is the same split this record already draws in
|
||||||
|
*an endpoint that is not routed is reached but never named*; what it got wrong was carrying the filter
|
||||||
|
across it.
|
||||||
|
|
||||||
|
This corrects a fact, not the decision: one statement per endpoint, three things derived from it and
|
||||||
|
none of them deciding on its own, all stand. The table in the decision should be read with the filter
|
||||||
|
column applying to an unrouted endpoint.
|
||||||
|
|
||||||
## Consequences
|
## Consequences
|
||||||
|
|
||||||
- **A manifest gains endpoint names, and a route contribution names an endpoint instead of a port.**
|
- **A manifest gains endpoint names, and a route contribution names an endpoint instead of a port.**
|
||||||
|
|||||||
@@ -0,0 +1,134 @@
|
|||||||
|
---
|
||||||
|
topic: what runs on it
|
||||||
|
status: accepted
|
||||||
|
date: 2026-09-29
|
||||||
|
deciders: jochen
|
||||||
|
reconstructed: false
|
||||||
|
extends: 02-DECISIONS/0010-delivery.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# 143. A consumer verifies the grant it is given
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
A **grant** is what the mesh writes on a consumer's machine so it can reach a provider. The real one
|
||||||
|
the forge receives for its database, as it arrives:
|
||||||
|
|
||||||
|
```
|
||||||
|
provision postgres-database
|
||||||
|
at <the provider's machine, by name>
|
||||||
|
port the machine port the provider is published on
|
||||||
|
as the role the provider created for this consumer
|
||||||
|
```
|
||||||
|
|
||||||
|
with the credential sealed in a separate file. Four facts and a password, and they are the whole
|
||||||
|
mechanism by which anything in the mesh reaches anything else.
|
||||||
|
|
||||||
|
**The mesh asserts that claim and never finds out whether it is true.**
|
||||||
|
[Issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md):
|
||||||
|
converging a machine dropped the path from a container to a port on its own machine, and for eleven
|
||||||
|
hours the mesh answered *all doing what they were told, all heard from, every module current with its
|
||||||
|
source* while a web application logged, six thousand times:
|
||||||
|
|
||||||
|
```
|
||||||
|
connection to server at "<the machine>" (10.10.0.1), port 6852 failed: timeout expired
|
||||||
|
```
|
||||||
|
|
||||||
|
Every check the mesh makes passed, because every check it makes is about the relationship between the
|
||||||
|
mesh and a machine: the declaration was applied, the digest matched, every container named was running.
|
||||||
|
None of them asks whether a consumer can reach what it requires — though the mesh composed the grant
|
||||||
|
and therefore knows the consumer, the machine, the address, the port and the credential.
|
||||||
|
|
||||||
|
**And where the check runs decides whether it catches anything.** The rule in force admitted the
|
||||||
|
machines' own addresses on the private network. A dial from the *machine* to its own address carries
|
||||||
|
exactly such a source address, so a check run by the host on its own behalf would have matched that rule
|
||||||
|
and passed — while every container on the machine was refused. This is inference from the rule that was
|
||||||
|
loaded, not a measurement: the fault was found and fixed before anyone thought to dial from the host.
|
||||||
|
It is enough to decide the question, because a check whose position differs from the consumer's is
|
||||||
|
testing something nobody asked about.
|
||||||
|
|
||||||
|
## Considered Options
|
||||||
|
|
||||||
|
1. **The control plane dials each provision.** Rejected, and it is the tempting one because the control
|
||||||
|
plane holds every fact. It sits on the provider's machine for most provisions here and reaches the
|
||||||
|
address by a path no consumer uses; in the measured outage it would have passed throughout.
|
||||||
|
2. **The host dials on the consumer's behalf, from the machine.** Rejected for the reason above: the
|
||||||
|
machine's network position is not the consumer's, and the one outage this exists to catch is exactly
|
||||||
|
a difference between them.
|
||||||
|
3. **Ask the module.** Rejected: a module is arbitrary software that the mesh does not write. Some could
|
||||||
|
report on their provisions and most cannot, and a check that covers the modules that opted in tells
|
||||||
|
nobody anything about the rest.
|
||||||
|
4. **Read the module's logs.** Rejected: the failure was in a log the whole time, and reading a module's
|
||||||
|
logs makes the mesh depend on the wording of software it does not control.
|
||||||
|
5. **The consumer verifies it, from its own network position.** Adopted.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
**A consumer verifies each grant it is given, from its own network position.** After a reconcile has
|
||||||
|
applied a grant, the machine opens a connection to the address and port that grant names, from inside
|
||||||
|
the consumer's own network namespace — the same position the consumer's software dials from, which is
|
||||||
|
the only position that answers the question the grant asks.
|
||||||
|
|
||||||
|
**It is a connection, not a conversation.** Whether the port accepts a connection is what a grant
|
||||||
|
claims; whether the credential is right, the role exists or the schema is current is the provider's to
|
||||||
|
answer and the consumer's to discover. A check that spoke each provision's protocol would be a second
|
||||||
|
implementation of every provision, and would fail for reasons that are not the mesh's.
|
||||||
|
|
||||||
|
**One failure is not news.** A provider restarting is ordinary, and so is a consumer between containers.
|
||||||
|
A grant is reported unreachable only after it has failed on **consecutive** reconciles, and the count is
|
||||||
|
what the machine reports rather than the last attempt — so a reader can tell "it was briefly away" from
|
||||||
|
"it has never worked".
|
||||||
|
|
||||||
|
**A grant that cannot be checked is said to be unchecked, never assumed good.** A consumer that is not
|
||||||
|
running has no network position to dial from; that is not a broken grant and must not read as one. It is
|
||||||
|
also not a verified grant, and the two are different sentences.
|
||||||
|
|
||||||
|
**What it costs to be wrong is the constraint on all of it.** A check that reports a working provision
|
||||||
|
broken trains a reader to ignore the report, which is worse than having none — the fault this
|
||||||
|
repository keeps finding, one level up. So the threshold is consecutive failures, the check is the
|
||||||
|
cheapest thing that answers the question, and an unknown is reported as unknown.
|
||||||
|
|
||||||
|
**The mesh says it where it says everything else.** A machine's report carries its unreachable grants,
|
||||||
|
and `status` names them beside what is out of date — so "every module current with its source" stops
|
||||||
|
being the whole of what the mesh will tell you about a machine whose modules cannot reach each other.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- **The mesh can be wrong out loud.** It has been able to assert a grant and not check it; now a grant
|
||||||
|
that does not work is a thing the mesh says, and the eleven hours of issue 145 become minutes.
|
||||||
|
- **The host gains the ability to act from a container's network position**, which it has not needed
|
||||||
|
before. That is a real capability and the only one this needs.
|
||||||
|
- **A machine reports something that is not about the declaration.** Everything it reports today is
|
||||||
|
what it applied and what it holds; this is the first thing it says about whether what it applied
|
||||||
|
works.
|
||||||
|
- **A provision with no port is not checked**, because there is nothing to dial. Several are files and
|
||||||
|
secrets, and saying "checked" about those would be the appearance of verification that this record
|
||||||
|
exists to remove.
|
||||||
|
- **What got harder:** a reconcile does more than apply. Every grant adds a connection attempt on a
|
||||||
|
cadence, which is cheap individually and worth naming: a machine with many consumers dials once per
|
||||||
|
grant per reconcile.
|
||||||
|
|
||||||
|
## How it is checked
|
||||||
|
|
||||||
|
- **The outage is caught.** A bed drops the path from a consumer's network position to a provider's
|
||||||
|
port while leaving the machine's own path to it open — the exact shape of issue 145 — and the grant
|
||||||
|
reads unreachable. This fails against the previous behaviour, where nothing reported anything, and
|
||||||
|
against a check run from the machine, which passes while the consumer cannot reach it.
|
||||||
|
- **A restarting provider is not an outage.** One failed reconcile reports nothing; the count rises and
|
||||||
|
falls, and the grant reads reachable again without anybody acting.
|
||||||
|
- **A consumer that is not running reads unchecked, not broken**, asserted separately from the
|
||||||
|
unreachable case because they are different sentences.
|
||||||
|
- **A provision with no port is not claimed to be checked.**
|
||||||
|
- **The report carries the count, not the last attempt**, so "briefly away" and "never worked" are
|
||||||
|
distinguishable by a reader who sees only the report.
|
||||||
|
- **`status` names an unreachable grant**, asserted on the output, since a check nothing surfaces is
|
||||||
|
the same as no check.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [ADR 0010](0010-delivery.md) — the declaration is owned resources; a grant is one of them
|
||||||
|
- [ADR 0009](0009-modules-and-the-graph.md) — what a provision and a consumer are
|
||||||
|
- [issue 145](../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md)
|
||||||
|
— the eleven hours
|
||||||
|
- [issue 136](../04-ISSUES/136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md) — the
|
||||||
|
same distance between a declaration and a machine, one level down
|
||||||
@@ -223,6 +223,7 @@ python3 00-META/checks/index.py fail if stale
|
|||||||
- **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md) *(superseded)*
|
- **0139** — [A network is forwarded because a module declared it](0139-a-network-is-forwarded-because-a-module-declared-it.md) *(superseded)*
|
||||||
- **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md)
|
- **0140** — [The filter constrains what arrives from outside, and says nothing about a machine's own guests](0140-the-filter-constrains-what-arrives-from-outside.md)
|
||||||
- **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)
|
- **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md)
|
||||||
|
- **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md)
|
||||||
|
|
||||||
### How it is built
|
### How it is built
|
||||||
|
|
||||||
|
|||||||
@@ -5,8 +5,9 @@ code:
|
|||||||
- mesh-controller internal/builder
|
- mesh-controller internal/builder
|
||||||
- mesh-controller cmd/mesh-controller (build, build --behind, push, status)
|
- mesh-controller cmd/mesh-controller (build, build --behind, push, status)
|
||||||
- mesh-controller internal/inventory/builds.go
|
- mesh-controller internal/inventory/builds.go
|
||||||
updated: 2026-09-21
|
updated: 2026-09-29
|
||||||
decisions:
|
decisions:
|
||||||
|
- 02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md
|
||||||
- 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
|
- 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
|
||||||
- 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
|
- 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
|
||||||
- 02-DECISIONS/0010-delivery.md
|
- 02-DECISIONS/0010-delivery.md
|
||||||
@@ -226,3 +227,38 @@ when the current failure began and how many reports in a row have said it — th
|
|||||||
id, whatever the words; three make the machine stuck, and `status` says so beside the failure. The host keeps trying — stuck is what the mesh
|
id, whatever the words; three make the machine stuck, and `status` says so beside the failure. The host keeps trying — stuck is what the mesh
|
||||||
knows, not what the machine is told. *How it is checked:* an inventory test counts three identical
|
knows, not what the machine is told. *How it is checked:* an inventory test counts three identical
|
||||||
reports, a different one, and a clean apply; the status test asserts the word appears.
|
reports, a different one, and a clean apply; the status test asserts the word appears.
|
||||||
|
|
||||||
|
## A consumer verifies the grant it is given
|
||||||
|
|
||||||
|
*2026-09-29, from an outage that ran eleven hours —
|
||||||
|
[issue 145](../../04-ISSUES/145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
|
||||||
|
settled by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md).*
|
||||||
|
|
||||||
|
A grant is four facts and a credential: the provision, the machine, the port, and who the consumer is
|
||||||
|
when it connects. It is the whole mechanism by which anything in the mesh reaches anything else, and the
|
||||||
|
mesh asserted it without ever finding out whether it was true.
|
||||||
|
|
||||||
|
What that cost: a machine was converged, the path from a container to a port on its own machine closed,
|
||||||
|
and for eleven hours the mesh answered *all heard from, every module current with its source* while a
|
||||||
|
module logged a connection timeout to its database six thousand times. Every check the mesh makes passed,
|
||||||
|
because every one is about the relationship between the mesh and a machine — applied, current, running.
|
||||||
|
None asks whether a consumer can reach what it requires.
|
||||||
|
|
||||||
|
**So a consumer verifies its own grants, from its own network position.** Not the control plane, which
|
||||||
|
reaches the address by a path no consumer uses; not the machine, whose own packets carried a source
|
||||||
|
address the filter admitted while every container's was refused. The position is the point: a check
|
||||||
|
somewhere else is testing something nobody asked about.
|
||||||
|
|
||||||
|
It opens a connection and nothing more. Whether the credential is right or the schema current is the
|
||||||
|
provider's to answer and the consumer's to discover; a check that spoke each provision's protocol would
|
||||||
|
be a second implementation of every provision.
|
||||||
|
|
||||||
|
One failure is not news — a provider restarting is ordinary — so a grant reads unreachable only after
|
||||||
|
consecutive reconciles, and the machine reports the count rather than the last attempt, which is what
|
||||||
|
separates "briefly away" from "never worked". A consumer that is not running has no position to dial
|
||||||
|
from: that grant reads unchecked, which is a different sentence from broken and must not be written as
|
||||||
|
one.
|
||||||
|
|
||||||
|
*How it is checked* is stated with the decision, and the first of them is the outage itself: a bed drops
|
||||||
|
the path from a consumer's position while leaving the machine's own open, and the grant must read
|
||||||
|
unreachable — which fails both against reporting nothing and against a check run from the machine.
|
||||||
|
|||||||
@@ -0,0 +1,102 @@
|
|||||||
|
---
|
||||||
|
status: located
|
||||||
|
opened: 2026-09-29
|
||||||
|
located-in:
|
||||||
|
- mesh-host internal/apply/opening.go (retireFirewall)
|
||||||
|
- mesh-host internal/apply/apply.go (the condition it is called under)
|
||||||
|
fixed-by:
|
||||||
|
amended-design:
|
||||||
|
---
|
||||||
|
|
||||||
|
# 143 — Converging a machine does not retire the firewall it found, and says it does
|
||||||
|
|
||||||
|
## What was observed
|
||||||
|
|
||||||
|
The control-node was converged on 2026-09-29, the first machine with a found firewall to be flipped —
|
||||||
|
the two converged before it had none.
|
||||||
|
|
||||||
|
The preview said, and the flip repeated:
|
||||||
|
|
||||||
|
```
|
||||||
|
the found firewall (ufw) is disabled, never flushed: its configuration stays on disk
|
||||||
|
...
|
||||||
|
sent: the host loads the mesh's filter and disables the firewall it found
|
||||||
|
```
|
||||||
|
|
||||||
|
The mesh then reported the node `converged`, 372 resources applied, nothing failed. Afterwards, on the
|
||||||
|
machine:
|
||||||
|
|
||||||
|
```
|
||||||
|
systemctl is-enabled ufw -> enabled
|
||||||
|
systemctl is-active ufw -> active
|
||||||
|
```
|
||||||
|
|
||||||
|
*Corrected 2026-09-29, an hour later, from reading the host rather than the declaration.* **The first
|
||||||
|
account of this was wrong.** It said the declaration carries no resource that would disable the found
|
||||||
|
firewall, and that the sentence was printed by the command with nothing implementing it. The
|
||||||
|
declaration indeed carries no such resource — but the mechanism was never meant to be one. It is a
|
||||||
|
step in the host's own apply, `retireFirewall`, and it exists, is careful, and is strict: it refuses to
|
||||||
|
retire anything until it has read back from the machine that the mesh's own table is loaded, it records
|
||||||
|
the forward policies first so a half-done retirement can be retried, and it verifies ufw reports
|
||||||
|
inactive afterwards.
|
||||||
|
|
||||||
|
What is established is narrower and stranger than "nothing implements it":
|
||||||
|
|
||||||
|
- ufw was **active and enabled two minutes after the flip**, and the flip had reported the node
|
||||||
|
converged with 372 resources applied and nothing failed.
|
||||||
|
- The machine's own record now reads `disabled_by_mesh: true` — but it was written by a reconcile
|
||||||
|
*after* an operator disabled ufw by hand, roughly fifty minutes later. A reconcile found ufw already
|
||||||
|
inactive, asked it to be inactive, read that back, and recorded that the mesh had done it.
|
||||||
|
- So the step did not take effect at the flip, and the machine's record now says it did.
|
||||||
|
|
||||||
|
The candidates are named rather than chosen, because the evidence does not separate them: the step is
|
||||||
|
called only when the apply had no failures, and a skipped step is silent; the mesh's table is loaded by
|
||||||
|
a service in the same apply, so whether it was loaded *at the moment the step asked* is an ordering
|
||||||
|
question; and the host's own detail lines do not reach the journal, so what it decided is not
|
||||||
|
recoverable after the fact.
|
||||||
|
|
||||||
|
## Why it matters beyond this instance
|
||||||
|
|
||||||
|
**It is a stated behaviour that does not happen, reported as success** — the fault this repository
|
||||||
|
exists to catch, and
|
||||||
|
[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) states it as
|
||||||
|
part of what the flip *is*: "loads the mesh's derived filter in place of its refusal-only table, and
|
||||||
|
retires the found firewall by disabling it, never by flushing".
|
||||||
|
|
||||||
|
**It could only be found on the first machine that had one.** The two machines converged before this
|
||||||
|
had no firewall to retire, so the step had never run, and nothing reported that it had not. That is
|
||||||
|
the same shape as [issue 136](../136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md):
|
||||||
|
a step that is silent when it does nothing.
|
||||||
|
|
||||||
|
**The machine is left doubly filtered, which is not what either firewall describes.** Every base chain
|
||||||
|
at a hook runs and a drop in any is final, so the machine now enforces the *intersection* of the mesh's
|
||||||
|
derived filter and a rule set left by the system being replaced. Nothing is broken by that today —
|
||||||
|
measured from outside, mail, the proxy and git-over-ssh answer and the databases and admin interfaces
|
||||||
|
are refused — but the machine's behaviour is described by neither of the two things claiming to
|
||||||
|
describe it, and the stale set includes a rule for a broker that no longer exists.
|
||||||
|
|
||||||
|
**And returning the node to adopted would be wrong in the other direction.** ADR 0100 says that
|
||||||
|
restores the found firewall by enabling it again; enabling something that was never disabled is
|
||||||
|
harmless, but the mesh's belief about which firewall is in force has been wrong in both modes.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- Which side owns retiring it — a resource in the declaration, so it is applied and reported like
|
||||||
|
everything else, or the flip as an act? A resource seems right: the flip is otherwise entirely
|
||||||
|
expressed as one, and an act that only the command performs cannot be re-checked on a later
|
||||||
|
reconcile.
|
||||||
|
- What should a reconcile do if the found firewall is enabled again by hand, or by a package update?
|
||||||
|
Convergence is a state, so presumably re-disable it and say so.
|
||||||
|
- Should the preview say what it *will* do rather than what it does, until a step exists that does it?
|
||||||
|
The wording was read as evidence twice in one session.
|
||||||
|
- Is there a check that a sentence the mesh prints corresponds to something that happened? This is the
|
||||||
|
second time in one session that a printed claim and the machine disagreed.
|
||||||
|
- **Why did the step not take effect?** It is called only when the apply had no failures, and being
|
||||||
|
skipped is silent. The mesh's table is loaded by a service in the same apply, so whether it was
|
||||||
|
loaded when the step asked is an ordering question — and ADR 0100 makes loading it first a
|
||||||
|
precondition rather than an expectation.
|
||||||
|
- **A step that records the mesh as having done what an operator did is worse than the omission.** The
|
||||||
|
record now says the mesh disabled ufw. Nothing distinguishes "we did this" from "we found it already
|
||||||
|
so". Should it?
|
||||||
|
- Why do the host's own detail lines not reach the journal? Everything it decided during the flip is
|
||||||
|
unrecoverable, which is why this account has candidates instead of a cause.
|
||||||
+84
@@ -0,0 +1,84 @@
|
|||||||
|
---
|
||||||
|
status: located
|
||||||
|
opened: 2026-09-29
|
||||||
|
located-in:
|
||||||
|
- mesh-host internal/apply/opening.go
|
||||||
|
- mesh-controller cmd/mesh-controller (the converge preview)
|
||||||
|
fixed-by:
|
||||||
|
amended-design:
|
||||||
|
---
|
||||||
|
|
||||||
|
# 144 — A predecessor's rules outlive the firewall the mesh found, and the mesh cannot see them
|
||||||
|
|
||||||
|
## What was observed
|
||||||
|
|
||||||
|
The mesh reports one thing about a machine's existing filtering: `firewall found: ufw`. On the
|
||||||
|
control-node, ufw was never what filtered the traffic that mattered.
|
||||||
|
|
||||||
|
Measured on 2026-09-29, before the machine was converged:
|
||||||
|
|
||||||
|
- ufw filters connections *to the machine*. It does not filter connections to a container's published
|
||||||
|
port, which arrive on the forwarded path where the container runtime accepts them before ufw's
|
||||||
|
forward chains are reached. Around thirty ports were published that way.
|
||||||
|
- Every one of the mesh's own forwarded openings, converged through ufw, had matched **zero packets** —
|
||||||
|
fifty rules in that chain, none ever matched, while the chain itself had passed 1.6 million
|
||||||
|
established packets. The restrictions read as applied and were inert.
|
||||||
|
- What actually kept those ports off the internet was a chain the predecessor installed in the
|
||||||
|
container runtime's own pre-accept hook, allowing the deliberately public ports and the private
|
||||||
|
ranges and dropping the rest on the outward link. Confirmed from outside: the proxy answered, the
|
||||||
|
container manager did not.
|
||||||
|
- That chain exists only in the running kernel. The persisted rule file is the distribution's empty
|
||||||
|
default, and nothing on disk recreates the chain.
|
||||||
|
|
||||||
|
After the flip, the mesh's own filter is loaded and does cover the forwarded path, so the machine no
|
||||||
|
longer depends on that chain. But **the chain is still there**, and it is now the only thing refusing
|
||||||
|
two ports the mesh believes are open: the bus and the registry, which
|
||||||
|
[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) requires be
|
||||||
|
reachable from anywhere so a machine can enrol and pull before it has a private-network address. The
|
||||||
|
mesh's rendered filter accepts both from anywhere. From outside, both are refused.
|
||||||
|
|
||||||
|
## Why it matters beyond this instance
|
||||||
|
|
||||||
|
**"The firewall found" is a kind, and filtering is not all in one place.** The host identifies one
|
||||||
|
front-end and reports it. A machine can carry rules from several sources — the front-end's own, the
|
||||||
|
container runtime's, an intrusion-prevention chain, and whatever a predecessor installed directly —
|
||||||
|
and the mesh's account of what filters the machine names exactly one of them.
|
||||||
|
|
||||||
|
**So adoption's central promise was half-true in both directions.** What the mesh converged through
|
||||||
|
the found firewall on the forwarded path did nothing at all, and what did the work was invisible to it.
|
||||||
|
A machine was reported as filtered by a mechanism that was not filtering.
|
||||||
|
|
||||||
|
**And convergence cannot retire what it cannot see.** Even once
|
||||||
|
[issue 143](../143-converging-does-not-retire-the-firewall-it-found/00-report.md) is fixed and the found
|
||||||
|
firewall is disabled, this chain remains, silently narrowing the machine below what the mesh's own
|
||||||
|
filter says. A rule the mesh did not write, cannot list, and will not remove — which today breaks the
|
||||||
|
enrolment path the design guarantees.
|
||||||
|
|
||||||
|
**The safe direction is not the same as the correct one.** Being more closed than intended broke nothing
|
||||||
|
visible, which is exactly why it went unnoticed for as long as the mesh has been on this machine.
|
||||||
|
|
||||||
|
## What it cost, measured later the same day
|
||||||
|
|
||||||
|
*2026-09-29.* The predecessor's chain was removed, and something it had been carrying went with it. It
|
||||||
|
admitted the private ranges wholesale, which is how a container on the machine reached a port declared
|
||||||
|
for the private network — the mesh's own filter admits the machines' overlay addresses, and a container
|
||||||
|
comes from a bridge. Every module that reached another by the machine's own name had been relying on the
|
||||||
|
predecessor's rule without anybody knowing.
|
||||||
|
|
||||||
|
That is [issue 145](../145-a-machine-reads-healthy-while-its-modules-cannot-reach-each-other/00-report.md),
|
||||||
|
and it ran for eleven hours while the mesh reported the machine healthy. The filter is fixed. What this
|
||||||
|
adds to the account here is that "the machine is more closed than the mesh believes" was not the
|
||||||
|
harmless direction after all — it was harmless for everything reached from outside, and an outage for
|
||||||
|
everything reached from within.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- Should the host report every place the machine filters from, rather than one kind — the front-end,
|
||||||
|
the runtime's hooks, and any chain it does not recognise, named so a person can look?
|
||||||
|
- What should the mesh do about rules it did not write and does not understand? Reporting them seems
|
||||||
|
right; removing them cannot be, and leaving them silent is what produced this.
|
||||||
|
- Does an opening converged through a found firewall need a check that it can actually take effect?
|
||||||
|
Fifty rules matching nothing would have been visible from the counters at any point.
|
||||||
|
- Is the bus and the registry being reachable from anywhere still what the mesh wants on a machine that
|
||||||
|
faces the internet? The design says yes, for enrolment. It deserves asking on its own rather than
|
||||||
|
being answered by a leftover.
|
||||||
+101
@@ -0,0 +1,101 @@
|
|||||||
|
---
|
||||||
|
status: located
|
||||||
|
opened: 2026-09-29
|
||||||
|
located-in:
|
||||||
|
- mesh-controller internal/catalogue/filtering.go (fixed for this instance)
|
||||||
|
- mesh-controller (what status reports, and what it does not ask)
|
||||||
|
fixed-by:
|
||||||
|
amended-design: 03-DESIGN/01-to-be/10-delivery.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# 145 — A machine reads healthy while its modules cannot reach each other
|
||||||
|
|
||||||
|
## What was observed
|
||||||
|
|
||||||
|
Converging the control-node closed every path by which a module on that machine reached another module
|
||||||
|
by the machine's own name. It ran for **eleven hours**. Throughout, the mesh answered:
|
||||||
|
|
||||||
|
```
|
||||||
|
4 machine(s), all doing what they were told, all heard from,
|
||||||
|
running what the mesh would send them, and every module current with its source
|
||||||
|
```
|
||||||
|
|
||||||
|
What was actually happening, from one affected module's own log:
|
||||||
|
|
||||||
|
```
|
||||||
|
Doctrine\DBAL\Exception: Failed to connect to the database:
|
||||||
|
SQLSTATE[08006] connection to server at "novox.internal" (10.10.0.1), port 6852 failed: timeout expired
|
||||||
|
```
|
||||||
|
|
||||||
|
6,154 of them, beginning at the minute of the flip. The web application accepted TCP connections and
|
||||||
|
never answered an HTTP request; a client waited 35 seconds and gave up. Confirmed from a throwaway
|
||||||
|
container on the machine: neither the store nor the forge was reachable on the machine's own address.
|
||||||
|
|
||||||
|
The cause is [issue 144](../144-the-predecessors-rules-outlive-the-firewall-it-was-found-as/00-report.md)'s
|
||||||
|
sibling and is fixed: a port declared reachable from the private network admitted the machines' own
|
||||||
|
overlay addresses, and a container on the machine comes from a bridge address, matching none of them.
|
||||||
|
What this issue is about is the eleven hours.
|
||||||
|
|
||||||
|
**Nothing the mesh reports would have shown it.** Every check the mesh makes passed, because every
|
||||||
|
check the mesh makes is about the relationship between the mesh and a machine:
|
||||||
|
|
||||||
|
- the machine applied what it was sent, and said so;
|
||||||
|
- its declaration digest matches what the mesh would send;
|
||||||
|
- every module's source commit matches what the mesh holds;
|
||||||
|
- every container the declaration names is running.
|
||||||
|
|
||||||
|
None of those asks whether a module can reach what it requires. The mesh knows precisely who requires
|
||||||
|
what — it composes the grants — and never checks that the grant works.
|
||||||
|
|
||||||
|
**Nor would an operator's usual look.** The ports were probed from outside and behaved correctly; the
|
||||||
|
routed services answered; a container's egress to the internet worked. Those are the paths a person
|
||||||
|
checks after changing a firewall, and all three were fine. The broken path was module-to-module over
|
||||||
|
the machine's own name, which nothing routine exercises.
|
||||||
|
|
||||||
|
## Why it matters beyond this instance
|
||||||
|
|
||||||
|
**A mesh that composes a dependency and never tests it can only report on itself.** Every provision the
|
||||||
|
mesh grants is a claim that a consumer can reach a provider. The mesh asserts that claim, delivers
|
||||||
|
credentials for it, and has no mechanism that ever finds out. "Every module current with its source"
|
||||||
|
is a statement about bytes, not about whether anything works.
|
||||||
|
|
||||||
|
**The failure was silent in the direction that hides longest.** A service that will not start is
|
||||||
|
noticed. A service that starts, accepts connections and then cannot reach its database serves errors
|
||||||
|
under a healthy-looking process, and the machine's own report says the container is running — which it
|
||||||
|
is.
|
||||||
|
|
||||||
|
**It is the same shape as [issue 136](../136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md),
|
||||||
|
one level up.** There, a module named a program the machine lacked and everything reported success.
|
||||||
|
Here, the mesh granted a provision the filter refused and everything reported success. Both are the
|
||||||
|
distance between a declaration and the machine, and in both cases the report was about the declaration.
|
||||||
|
|
||||||
|
**And the eleven hours are the measurement, not the bug.** The filter fault was one line and is fixed.
|
||||||
|
What is not fixed is that nothing in the mesh would have told anybody.
|
||||||
|
|
||||||
|
## What was decided
|
||||||
|
|
||||||
|
*2026-09-29, the same day.* The first open question below — should a grant be checked, and from where —
|
||||||
|
is answered by [ADR 0143](../../02-DECISIONS/0143-a-consumer-verifies-the-grant-it-is-given.md): the
|
||||||
|
consumer verifies it, from its own network position, because that is the only position that answers what
|
||||||
|
a grant claims. A check run by the control plane or by the machine would have passed throughout this
|
||||||
|
outage, since the rule in force admitted the machines' own addresses and it was the containers that were
|
||||||
|
refused.
|
||||||
|
|
||||||
|
The remaining questions below stand, and the record answers two of them: one failure is not news, only
|
||||||
|
consecutive ones, and a grant that cannot be checked is reported unchecked rather than assumed good.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- Should a grant be checked? The mesh knows the consumer, the provider, the address and the port, so a
|
||||||
|
reachability check is expressible — but from where: the consumer's machine, as part of a reconcile,
|
||||||
|
or the provider's?
|
||||||
|
- What would it cost to be wrong in the other direction? A check that reports a provision broken while
|
||||||
|
it works is worse than none, because it trains a reader to ignore the report. A provider restarting is
|
||||||
|
ordinary; a consumer between containers is ordinary.
|
||||||
|
- What should `status` say about a machine whose modules cannot reach each other? It currently has one
|
||||||
|
vocabulary for "heard from and current", and that sentence was true the whole time.
|
||||||
|
- Is there a cheaper signal than a probe? The affected module was logging the failure 6,154 times. The
|
||||||
|
mesh reads no module's logs and arguably should not — but something a module could *say* about its
|
||||||
|
own provisions would have surfaced this in minutes.
|
||||||
|
- Does the same blindness apply to the other direction — a provider that lost a consumer's grant and
|
||||||
|
is refusing it? Nothing checks that either.
|
||||||
Reference in New Issue
Block a user