From 1572d74a183926d53fa5fc98ecb5bf0cd0a2169d Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 18 Sep 2026 02:20:11 +0200 Subject: [PATCH] =?UTF-8?q?Issue=20063=20=E2=80=94=20the=20foundation's=20?= =?UTF-8?q?ports=20are=20forwarded,=20resolved?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The broker's port was accepted on input but not forwarded, so a joined node reached a DNAT'd broker only until it restarted. Fixed in mesh-controller (25e42b3), proven by the built-store-cross-node bed adopting the foundation's broker (which restarts it) and the joined node still receiving declarations. --- .../00-report.md | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 04-ISSUES/063-the-foundation-ports-are-accepted-on-input-but-not-forwarded/00-report.md diff --git a/04-ISSUES/063-the-foundation-ports-are-accepted-on-input-but-not-forwarded/00-report.md b/04-ISSUES/063-the-foundation-ports-are-accepted-on-input-but-not-forwarded/00-report.md new file mode 100644 index 0000000..7890f37 --- /dev/null +++ b/04-ISSUES/063-the-foundation-ports-are-accepted-on-input-but-not-forwarded/00-report.md @@ -0,0 +1,50 @@ +--- +status: resolved +opened: 2026-09-18 +located-in: + - mesh-controller +fixed-by: mesh-controller fix/one-push-is-enough (25e42b3); gated by the built-store-cross-node bed +amended-design: + +--- + +# 063 — The foundation's ports are accepted on input but not forwarded + +## Symptom + +A joined node reaches the mesh's broker once, then — after the broker restarts — can never reach +it again, and so never receives another declaration. Every check between them passes: the port is +open, the broker is up, the overlay is established. + +Observed on the built-store-cross-node bed, intermittently. A joined node's host connected to the +broker, received its first declaration, and worked. When the foundation's own broker was adopted — +which restarts it — the node's link dropped with `CONNECTION_FORCED - Broker shutdown`, then +`connection refused`, then `i/o timeout` for ever. The node was left with no declaration on disk +at all, and every downstream assertion (here, the registry trust) failed as a consequence of a +node that had received nothing. + +The bed passed on the runs where the broker did not happen to restart after the joined node first +connected, which is why three green runs preceded the red one on identical code. + +## Why it matters beyond the instance + +The mesh firewall opens the foundation's ports — the broker above all — in the **input** chain, +from anywhere, so a node can enrol before it has an overlay address. But the broker is a published +container port: a cross-node packet to it is redirected by the runtime and handled in the +**forward** chain, which never carried a rule for the foundation. The first connection survived +only on its conntrack `established` entry; a broker restart dropped the entry, and the next dial +hit the forward chain's default drop. + +This is [issue 047](../047-the-firewall-does-not-cover-published-container-ports/00-report.md)'s +lesson — a firewall with no forward coverage says nothing about container ports — reappearing for +the one port the whole mesh depends on, because the foundation is not a module and was added to +input alone. A rule that works only until the thing it governs restarts is worse than none: it +passes every test written before the restart. + +## Open questions + +- Is the broker the only foundation port that resolves to a container, or should every foundation + port be assumed to be published and forwarded on principle? (The fix forwards all of them, on + that principle.) +- Should a bed assert reachability *after* a deliberate broker restart, so "reachable until it + bounces" can never again read as "reachable"?