jschoubben
1572d74a18
The broker's port was accepted on input but not forwarded, so a joined node reached a DNAT'd broker only until it restarted. Fixed in mesh-controller (25e42b3), proven by the built-store-cross-node bed adopting the foundation's broker (which restarts it) and the joined node still receiving declarations.
2.5 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design | |
|---|---|---|---|---|---|
| resolved | 2026-09-18 |
|
mesh-controller fix/one-push-is-enough (25e42b3); gated by the built-store-cross-node bed |
063 — The foundation's ports are accepted on input but not forwarded
Symptom
A joined node reaches the mesh's broker once, then — after the broker restarts — can never reach it again, and so never receives another declaration. Every check between them passes: the port is open, the broker is up, the overlay is established.
Observed on the built-store-cross-node bed, intermittently. A joined node's host connected to the
broker, received its first declaration, and worked. When the foundation's own broker was adopted —
which restarts it — the node's link dropped with CONNECTION_FORCED - Broker shutdown, then
connection refused, then i/o timeout for ever. The node was left with no declaration on disk
at all, and every downstream assertion (here, the registry trust) failed as a consequence of a
node that had received nothing.
The bed passed on the runs where the broker did not happen to restart after the joined node first connected, which is why three green runs preceded the red one on identical code.
Why it matters beyond the instance
The mesh firewall opens the foundation's ports — the broker above all — in the input chain,
from anywhere, so a node can enrol before it has an overlay address. But the broker is a published
container port: a cross-node packet to it is redirected by the runtime and handled in the
forward chain, which never carried a rule for the foundation. The first connection survived
only on its conntrack established entry; a broker restart dropped the entry, and the next dial
hit the forward chain's default drop.
This is issue 047's lesson — a firewall with no forward coverage says nothing about container ports — reappearing for the one port the whole mesh depends on, because the foundation is not a module and was added to input alone. A rule that works only until the thing it governs restarts is worse than none: it passes every test written before the restart.
Open questions
- Is the broker the only foundation port that resolves to a container, or should every foundation port be assumed to be published and forwarded on principle? (The fix forwards all of them, on that principle.)
- Should a bed assert reachability after a deliberate broker restart, so "reachable until it bounces" can never again read as "reachable"?