Issue 063 — the foundation's ports are forwarded, resolved

The broker's port was accepted on input but not forwarded, so a joined node reached a DNAT'd broker only until it restarted. Fixed in mesh-controller (25e42b3), proven by the built-store-cross-node bed adopting the foundation's broker (which restarts it) and the joined node still receiving declarations.
2026-09-18 02:20:11 +02:00
parent 9fd7e6c458
commit 1572d74a18
1 changed files with 50 additions and 0 deletions
@@ -0,0 +1,50 @@
+---
+status: resolved
+opened: 2026-09-18
+located-in:
+  - mesh-controller
+fixed-by: mesh-controller fix/one-push-is-enough (25e42b3); gated by the built-store-cross-node bed
+amended-design:
+
+---
+
+# 063 — The foundation's ports are accepted on input but not forwarded
+
+## Symptom
+
+A joined node reaches the mesh's broker once, then — after the broker restarts — can never reach
+it again, and so never receives another declaration. Every check between them passes: the port is
+open, the broker is up, the overlay is established.
+
+Observed on the built-store-cross-node bed, intermittently. A joined node's host connected to the
+broker, received its first declaration, and worked. When the foundation's own broker was adopted —
+which restarts it — the node's link dropped with `CONNECTION_FORCED - Broker shutdown`, then
+`connection refused`, then `i/o timeout` for ever. The node was left with no declaration on disk
+at all, and every downstream assertion (here, the registry trust) failed as a consequence of a
+node that had received nothing.
+
+The bed passed on the runs where the broker did not happen to restart after the joined node first
+connected, which is why three green runs preceded the red one on identical code.
+
+## Why it matters beyond the instance
+
+The mesh firewall opens the foundation's ports — the broker above all — in the **input** chain,
+from anywhere, so a node can enrol before it has an overlay address. But the broker is a published
+container port: a cross-node packet to it is redirected by the runtime and handled in the
+**forward** chain, which never carried a rule for the foundation. The first connection survived
+only on its conntrack `established` entry; a broker restart dropped the entry, and the next dial
+hit the forward chain's default drop.
+
+This is [issue 047](../047-the-firewall-does-not-cover-published-container-ports/00-report.md)'s
+lesson — a firewall with no forward coverage says nothing about container ports — reappearing for
+the one port the whole mesh depends on, because the foundation is not a module and was added to
+input alone. A rule that works only until the thing it governs restarts is worse than none: it
+passes every test written before the restart.
+
+## Open questions
+
+- Is the broker the only foundation port that resolves to a container, or should every foundation
+  port be assumed to be published and forwarded on principle? (The fix forwards all of them, on
+  that principle.)
+- Should a bed assert reachability *after* a deliberate broker restart, so "reachable until it
+  bounces" can never again read as "reachable"?