jschoubben
492ac7be18
Converging the control node closed every path by which a module reached another by the machine's own name, and it ran for eleven hours while the mesh answered 'all doing what they were told, all heard from, running what the mesh would send them, and every module current with its source'. 6,154 database failures in one module's log, beginning at the minute of the flip. The service accepted TCP and never answered HTTP. Every check the mesh makes passed, because every check the mesh makes is about the relationship between the mesh and a machine — applied, current, containers running. None asks whether a module can reach what it requires, though the mesh composes every grant and so knows exactly who requires what. The filter fault is fixed. The eleven hours are the measurement, not the bug. Also records on 144 that 'more closed than the mesh believes' was harmless only for what is reached from outside, and an outage for what is reached from within.
5.1 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design | ||
|---|---|---|---|---|---|---|
| located | 2026-09-29 |
|
144 — A predecessor's rules outlive the firewall the mesh found, and the mesh cannot see them
What was observed
The mesh reports one thing about a machine's existing filtering: firewall found: ufw. On the
control-node, ufw was never what filtered the traffic that mattered.
Measured on 2026-09-29, before the machine was converged:
- ufw filters connections to the machine. It does not filter connections to a container's published port, which arrive on the forwarded path where the container runtime accepts them before ufw's forward chains are reached. Around thirty ports were published that way.
- Every one of the mesh's own forwarded openings, converged through ufw, had matched zero packets — fifty rules in that chain, none ever matched, while the chain itself had passed 1.6 million established packets. The restrictions read as applied and were inert.
- What actually kept those ports off the internet was a chain the predecessor installed in the container runtime's own pre-accept hook, allowing the deliberately public ports and the private ranges and dropping the rest on the outward link. Confirmed from outside: the proxy answered, the container manager did not.
- That chain exists only in the running kernel. The persisted rule file is the distribution's empty default, and nothing on disk recreates the chain.
After the flip, the mesh's own filter is loaded and does cover the forwarded path, so the machine no longer depends on that chain. But the chain is still there, and it is now the only thing refusing two ports the mesh believes are open: the bus and the registry, which ADR 0100 requires be reachable from anywhere so a machine can enrol and pull before it has a private-network address. The mesh's rendered filter accepts both from anywhere. From outside, both are refused.
Why it matters beyond this instance
"The firewall found" is a kind, and filtering is not all in one place. The host identifies one front-end and reports it. A machine can carry rules from several sources — the front-end's own, the container runtime's, an intrusion-prevention chain, and whatever a predecessor installed directly — and the mesh's account of what filters the machine names exactly one of them.
So adoption's central promise was half-true in both directions. What the mesh converged through the found firewall on the forwarded path did nothing at all, and what did the work was invisible to it. A machine was reported as filtered by a mechanism that was not filtering.
And convergence cannot retire what it cannot see. Even once issue 143 is fixed and the found firewall is disabled, this chain remains, silently narrowing the machine below what the mesh's own filter says. A rule the mesh did not write, cannot list, and will not remove — which today breaks the enrolment path the design guarantees.
The safe direction is not the same as the correct one. Being more closed than intended broke nothing visible, which is exactly why it went unnoticed for as long as the mesh has been on this machine.
What it cost, measured later the same day
2026-09-29. The predecessor's chain was removed, and something it had been carrying went with it. It admitted the private ranges wholesale, which is how a container on the machine reached a port declared for the private network — the mesh's own filter admits the machines' overlay addresses, and a container comes from a bridge. Every module that reached another by the machine's own name had been relying on the predecessor's rule without anybody knowing.
That is issue 145, and it ran for eleven hours while the mesh reported the machine healthy. The filter is fixed. What this adds to the account here is that "the machine is more closed than the mesh believes" was not the harmless direction after all — it was harmless for everything reached from outside, and an outage for everything reached from within.
Open questions
- Should the host report every place the machine filters from, rather than one kind — the front-end, the runtime's hooks, and any chain it does not recognise, named so a person can look?
- What should the mesh do about rules it did not write and does not understand? Reporting them seems right; removing them cannot be, and leaving them silent is what produced this.
- Does an opening converged through a found firewall need a check that it can actually take effect? Fifty rules matching nothing would have been visible from the counters at any point.
- Is the bus and the registry being reachable from anywhere still what the mesh wants on a machine that faces the internet? The design says yes, for enrolment. It deserves asking on its own rather than being answered by a leftover.