Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ddb3980f09 | ||
|
|
75a8f6abc7 | ||
|
|
862253518f | ||
|
|
51ef3eb7e2 |
@@ -0,0 +1,102 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- mesh-host internal/apply/opening.go (retireFirewall)
|
||||
- mesh-host internal/apply/apply.go (the condition it is called under)
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 143 — Converging a machine does not retire the firewall it found, and says it does
|
||||
|
||||
## What was observed
|
||||
|
||||
The control-node was converged on 2026-09-29, the first machine with a found firewall to be flipped —
|
||||
the two converged before it had none.
|
||||
|
||||
The preview said, and the flip repeated:
|
||||
|
||||
```
|
||||
the found firewall (ufw) is disabled, never flushed: its configuration stays on disk
|
||||
...
|
||||
sent: the host loads the mesh's filter and disables the firewall it found
|
||||
```
|
||||
|
||||
The mesh then reported the node `converged`, 372 resources applied, nothing failed. Afterwards, on the
|
||||
machine:
|
||||
|
||||
```
|
||||
systemctl is-enabled ufw -> enabled
|
||||
systemctl is-active ufw -> active
|
||||
```
|
||||
|
||||
*Corrected 2026-09-29, an hour later, from reading the host rather than the declaration.* **The first
|
||||
account of this was wrong.** It said the declaration carries no resource that would disable the found
|
||||
firewall, and that the sentence was printed by the command with nothing implementing it. The
|
||||
declaration indeed carries no such resource — but the mechanism was never meant to be one. It is a
|
||||
step in the host's own apply, `retireFirewall`, and it exists, is careful, and is strict: it refuses to
|
||||
retire anything until it has read back from the machine that the mesh's own table is loaded, it records
|
||||
the forward policies first so a half-done retirement can be retried, and it verifies ufw reports
|
||||
inactive afterwards.
|
||||
|
||||
What is established is narrower and stranger than "nothing implements it":
|
||||
|
||||
- ufw was **active and enabled two minutes after the flip**, and the flip had reported the node
|
||||
converged with 372 resources applied and nothing failed.
|
||||
- The machine's own record now reads `disabled_by_mesh: true` — but it was written by a reconcile
|
||||
*after* an operator disabled ufw by hand, roughly fifty minutes later. A reconcile found ufw already
|
||||
inactive, asked it to be inactive, read that back, and recorded that the mesh had done it.
|
||||
- So the step did not take effect at the flip, and the machine's record now says it did.
|
||||
|
||||
The candidates are named rather than chosen, because the evidence does not separate them: the step is
|
||||
called only when the apply had no failures, and a skipped step is silent; the mesh's table is loaded by
|
||||
a service in the same apply, so whether it was loaded *at the moment the step asked* is an ordering
|
||||
question; and the host's own detail lines do not reach the journal, so what it decided is not
|
||||
recoverable after the fact.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**It is a stated behaviour that does not happen, reported as success** — the fault this repository
|
||||
exists to catch, and
|
||||
[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) states it as
|
||||
part of what the flip *is*: "loads the mesh's derived filter in place of its refusal-only table, and
|
||||
retires the found firewall by disabling it, never by flushing".
|
||||
|
||||
**It could only be found on the first machine that had one.** The two machines converged before this
|
||||
had no firewall to retire, so the step had never run, and nothing reported that it had not. That is
|
||||
the same shape as [issue 136](../136-a-module-may-name-a-program-the-machine-does-not-have/00-report.md):
|
||||
a step that is silent when it does nothing.
|
||||
|
||||
**The machine is left doubly filtered, which is not what either firewall describes.** Every base chain
|
||||
at a hook runs and a drop in any is final, so the machine now enforces the *intersection* of the mesh's
|
||||
derived filter and a rule set left by the system being replaced. Nothing is broken by that today —
|
||||
measured from outside, mail, the proxy and git-over-ssh answer and the databases and admin interfaces
|
||||
are refused — but the machine's behaviour is described by neither of the two things claiming to
|
||||
describe it, and the stale set includes a rule for a broker that no longer exists.
|
||||
|
||||
**And returning the node to adopted would be wrong in the other direction.** ADR 0100 says that
|
||||
restores the found firewall by enabling it again; enabling something that was never disabled is
|
||||
harmless, but the mesh's belief about which firewall is in force has been wrong in both modes.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Which side owns retiring it — a resource in the declaration, so it is applied and reported like
|
||||
everything else, or the flip as an act? A resource seems right: the flip is otherwise entirely
|
||||
expressed as one, and an act that only the command performs cannot be re-checked on a later
|
||||
reconcile.
|
||||
- What should a reconcile do if the found firewall is enabled again by hand, or by a package update?
|
||||
Convergence is a state, so presumably re-disable it and say so.
|
||||
- Should the preview say what it *will* do rather than what it does, until a step exists that does it?
|
||||
The wording was read as evidence twice in one session.
|
||||
- Is there a check that a sentence the mesh prints corresponds to something that happened? This is the
|
||||
second time in one session that a printed claim and the machine disagreed.
|
||||
- **Why did the step not take effect?** It is called only when the apply had no failures, and being
|
||||
skipped is silent. The mesh's table is loaded by a service in the same apply, so whether it was
|
||||
loaded when the step asked is an ordering question — and ADR 0100 makes loading it first a
|
||||
precondition rather than an expectation.
|
||||
- **A step that records the mesh as having done what an operator did is worse than the omission.** The
|
||||
record now says the mesh disabled ufw. Nothing distinguishes "we did this" from "we found it already
|
||||
so". Should it?
|
||||
- Why do the host's own detail lines not reach the journal? Everything it decided during the flip is
|
||||
unrecoverable, which is why this account has candidates instead of a cause.
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-09-29
|
||||
located-in:
|
||||
- mesh-host internal/apply/opening.go
|
||||
- mesh-controller cmd/mesh-controller (the converge preview)
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 144 — A predecessor's rules outlive the firewall the mesh found, and the mesh cannot see them
|
||||
|
||||
## What was observed
|
||||
|
||||
The mesh reports one thing about a machine's existing filtering: `firewall found: ufw`. On the
|
||||
control-node, ufw was never what filtered the traffic that mattered.
|
||||
|
||||
Measured on 2026-09-29, before the machine was converged:
|
||||
|
||||
- ufw filters connections *to the machine*. It does not filter connections to a container's published
|
||||
port, which arrive on the forwarded path where the container runtime accepts them before ufw's
|
||||
forward chains are reached. Around thirty ports were published that way.
|
||||
- Every one of the mesh's own forwarded openings, converged through ufw, had matched **zero packets** —
|
||||
fifty rules in that chain, none ever matched, while the chain itself had passed 1.6 million
|
||||
established packets. The restrictions read as applied and were inert.
|
||||
- What actually kept those ports off the internet was a chain the predecessor installed in the
|
||||
container runtime's own pre-accept hook, allowing the deliberately public ports and the private
|
||||
ranges and dropping the rest on the outward link. Confirmed from outside: the proxy answered, the
|
||||
container manager did not.
|
||||
- That chain exists only in the running kernel. The persisted rule file is the distribution's empty
|
||||
default, and nothing on disk recreates the chain.
|
||||
|
||||
After the flip, the mesh's own filter is loaded and does cover the forwarded path, so the machine no
|
||||
longer depends on that chain. But **the chain is still there**, and it is now the only thing refusing
|
||||
two ports the mesh believes are open: the bus and the registry, which
|
||||
[ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) requires be
|
||||
reachable from anywhere so a machine can enrol and pull before it has a private-network address. The
|
||||
mesh's rendered filter accepts both from anywhere. From outside, both are refused.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
**"The firewall found" is a kind, and filtering is not all in one place.** The host identifies one
|
||||
front-end and reports it. A machine can carry rules from several sources — the front-end's own, the
|
||||
container runtime's, an intrusion-prevention chain, and whatever a predecessor installed directly —
|
||||
and the mesh's account of what filters the machine names exactly one of them.
|
||||
|
||||
**So adoption's central promise was half-true in both directions.** What the mesh converged through
|
||||
the found firewall on the forwarded path did nothing at all, and what did the work was invisible to it.
|
||||
A machine was reported as filtered by a mechanism that was not filtering.
|
||||
|
||||
**And convergence cannot retire what it cannot see.** Even once
|
||||
[issue 143](../143-converging-does-not-retire-the-firewall-it-found/00-report.md) is fixed and the found
|
||||
firewall is disabled, this chain remains, silently narrowing the machine below what the mesh's own
|
||||
filter says. A rule the mesh did not write, cannot list, and will not remove — which today breaks the
|
||||
enrolment path the design guarantees.
|
||||
|
||||
**The safe direction is not the same as the correct one.** Being more closed than intended broke nothing
|
||||
visible, which is exactly why it went unnoticed for as long as the mesh has been on this machine.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should the host report every place the machine filters from, rather than one kind — the front-end,
|
||||
the runtime's hooks, and any chain it does not recognise, named so a person can look?
|
||||
- What should the mesh do about rules it did not write and does not understand? Reporting them seems
|
||||
right; removing them cannot be, and leaving them silent is what produced this.
|
||||
- Does an opening converged through a found firewall need a check that it can actually take effect?
|
||||
Fifty rules matching nothing would have been visible from the counters at any point.
|
||||
- Is the bus and the registry being reachable from anywhere still what the mesh wants on a machine that
|
||||
faces the internet? The design says yes, for enrolment. It deserves asking on its own rather than
|
||||
being answered by a leftover.
|
||||
Reference in New Issue
Block a user