Merge pull request 'Issues 102–106 and ADR 0105: what the core migration found, and the hub adopting the predecessor's tunnel' (#88) from core/issues-102-106-and-tunnel-adr into main

2026-09-23 20:54:53 +00:00
parent ee2bdf220c cb2117f1c4
commit 8638ba3a4f
8 changed files with 365 additions and 1 deletions
@@ -0,0 +1,117 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-09-23
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
+---
+
+# 105. The mesh adopts the predecessor's tunnel in place
+
+## Context
+
+The control-node is adopted and its store and broker have been moved onto the ports the
+predecessor served them on, so the predecessor's other machines keep reaching them. They reach
+them **over the predecessor's tunnel**: a WireGuard interface on the control-node with three
+peers, an address range, and a port the hosting provider already lets through. The mesh's own
+private network runs beside it on a second interface, a second range and a second port — one the
+provider does not let through, so no other machine can join the mesh
+([research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md), the runbook's
+measurement of the upstream filter).
+
+[ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) says the mesh's range *must not
+overlap a tunnel the predecessor still runs*. That was written for coexistence. It leaves the
+migration with two tunnels for as long as any predecessor machine exists, and the second one
+unreachable.
+
+The mesh already adopts in place where the thing found is the thing it would have raised: the
+store and the broker were taken over as modules, keyed on the container that was running
+([ADR 0078](0078-the-store-and-broker-are-modules.md)). A WireGuard interface is the same shape: a
+private key, a listening port, a list of peers by public key, an address. The mesh's is not
+different in kind from the predecessor's; it is a second one.
+
+The operator's instruction: take over the tunnel interface, same range — everything stays the same.
+
+## Considered Options
+
+1. **Two tunnels until the last predecessor machine is gone.** Rejected: the mesh's stays
+   unreachable from outside, so no machine can join, so the last predecessor machine is never gone.
+2. **Move the mesh's tunnel onto the predecessor's port with the mesh's own key and range.** The
+   peers' packets arrive and are dropped — WireGuard authenticates by key, and the mesh's key is not
+   the one they know. Every other machine loses its tunnel until it enrols, and it enrols over a bus
+   it reaches through that tunnel. Rejected.
+3. **Adopt the predecessor's tunnel in place: its private key, its peers, its range, its port.**
+   Adopted.
+
+## Decision
+
+**On an adopted node that is the hub, the private network takes over the tunnel it finds.** The
+mesh's interface is raised with the found interface's **private key**, on its **port**, with its
+**address and range**, and every **peer** the found interface had — public key, allowed address —
+carried into the mesh's peer list as a peer not yet enrolled. The found interface is stopped, never
+flushed; its configuration stays on disk, kept like any held file.
+
+**Nothing a peer knows changes.** A predecessor machine keeps the same server key, the same
+endpoint, the same address and the same route; it cannot tell the tunnel changed hands. When that
+machine enrols, it keeps its address: the mesh assigns an enrolling node the address the tunnel
+already had for its key, and only a node with no such address is given a fresh one from the range.
+
+**The mesh's own addresses are the range's.** The controller composes every node's private address
+from the tunnel it holds, so adopting the predecessor's range moves the mesh's addresses with it —
+the hub's, and every binding, hosts-file entry and endpoint derived from it. Those are readers of
+the setting; they follow it, per ADR 0100's rule for ports. A reader that does not follow is
+[issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md).
+
+**This narrows ADR 0100.** Its rule that the range must not overlap a tunnel the predecessor still
+runs applies to a node that is *not* adopting the tunnel: where the found tunnel is left running
+beside the mesh's, the ranges must differ. Where it is adopted, there is one tunnel and one range.
+
+**The guard's question answers itself.** [ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
+admits the mesh's own ports only from the private network's interface. With one tunnel, the
+predecessor's peers arrive on it, and nothing needs to be admitted from an interface the mesh does
+not own.
+
+## Consequences
+
+- **The order of a migration changes.** The hub's tunnel can be taken as soon as the node is
+  adopted, before any service — and should be, because it is what lets other machines join. The
+  runbook's step order is amended.
+- **The hub takes the provider's open port for free.** The port the predecessor's tunnel used is
+  by definition one the provider passes.
+- **A peer's identity precedes its enrolment.** The mesh holds public keys and addresses for
+  machines it has no record of. They are peers of the tunnel, not nodes of the mesh, until they
+  enrol; the registry must be able to say both.
+- **What got harder:** the private key of the found interface is read from the machine and becomes
+  the mesh's — the one case where the mesh takes a credential it did not mint. It is sealed like
+  any own secret from then on, and the found configuration file is kept, not copied further.
+- Two documents currently say the opposite: ADR 0100's non-overlap rule (narrowed above) and the
+  runbook's port plan, which is amended with this record.
+
+## How it is checked
+
+A lab bed prepares a hub the way the predecessor leaves one: a WireGuard interface with a key, a
+port, a range and two peers, each peer a second machine that reaches a service on the hub through
+the tunnel. Then:
+
+- **Adopted, the tunnel changes hands and the peers notice nothing**: the found interface is down
+  and its file is on disk; the mesh's interface is up with the found key, port and address; each
+  peer's service call succeeds before, during and after, with no reconfiguration on the peer.
+- **A peer enrols and keeps its address**: the machine joins the mesh over the tunnel it already
+  has, and its node address is the one the tunnel held for it.
+- **A new machine gets a fresh address from the same range**, and reaches both the hub and the
+  enrolled peer.
+- **Nothing derived from the address is stale**: every binding, hosts entry and endpoint the
+  controller composes says the adopted range, before and after a push.
+
+Unit tests hold the controller to reading the hub's address and range from the adopted tunnel,
+assigning an enrolling node the address its key already had, and refusing to hand out an address
+the tunnel already holds; and the host to raising the mesh's interface with the found key and
+peers and stopping the found interface without flushing it.
+
+## References
+
+- [ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md),
+  [ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
+- [issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)
+- [research 012 — the minimum viable node](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md)
@@ -93,6 +93,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0102** — [The mesh writes into a shared file, never over it](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)
 - **0103** — [What an adopted node holds, and what its guard refuses](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
 - **0104** — [A provision may be answered by an adapter to the predecessor](0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md)
+- **0105** — [The mesh adopts the predecessor's tunnel in place](0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md)

 ### Its tiers, from the bottom up

@@ -7,9 +7,10 @@ code:
  - mesh-controller internal/identity/authority.go
  - mesh-host internal/identity/serving.go
  - mesh-host internal/apply (the service that reflects a rule set)
-updated: 2026-09-22
+updated: 2026-09-23
 decisions:
  - 02-DECISIONS/0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md
+  - 02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md
  - 02-DECISIONS/0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md
  - 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
  - 02-DECISIONS/0099-a-step-that-runs-once-names-what-it-reads.md
@@ -715,3 +716,19 @@ The list is worth having in one place, because it is most of the argument:
  resolvable inside the mesh, not only routable from outside it; the mechanism that writes
  `<node>.internal` into containers does not yet also write the routed names, which is why an
  internal issuer cannot currently validate one without a hand-placed entry.
+
+## The hub adopts the predecessor's tunnel
+
+*2026-09-23, [ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md).*
+
+On an adopted node that is the hub, the private network is not raised beside the tunnel it finds;
+it **takes it over**: the found interface's private key, its port, its address and range, and every
+peer it had, carried as peers not yet enrolled. The found interface is stopped, its configuration
+kept on disk. A predecessor machine sees the same server key at the same endpoint and cannot tell
+the tunnel changed hands; when it enrols, it keeps the address the tunnel already held for its key.
+
+The mesh's own addresses are the adopted range's — every binding, hosts entry and endpoint the
+controller composes follows it, as readers of a setting. ADR 0100's non-overlap rule applies only
+where a found tunnel is left running beside the mesh's; where it is adopted there is one tunnel.
+The guard admits the mesh's ports from that one interface, and the predecessor's peers arrive on
+it.
@@ -0,0 +1,66 @@
+---
+status: located
+opened: 2026-09-23
+located-in: [mesh-controller cmd/mesh-bootstrap, mesh-controller internal/builder, mesh-controller internal/catalogue]
+fixed-by:
+amended-design:
+---
+
+# 102 — An address recorded at genesis or at build does not follow the node's ports
+
+## What was observed
+
+The control-node, 2026-09-23, migrating the foundation's store and broker onto the ports the
+predecessor served them on — the move [ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)
+describes: *the ports given become that node's settings, and every place that uses them reads them
+from there.*
+
+Most places did. When the store was given 6852, the bindings handed to the forge, the analytics
+service and the catalogue all said 6852. When the broker was given 5679, its consumers reconnected.
+
+Three places did not, and each took the mesh down in a different way:
+
+1. **The controller's own store address.** A secret written at genesis says `127.0.0.1:5432`. The
+   store moved; the controller could not reach its inventory; the mesh was headless.
+2. **The controller's own broker address.** The same, for `127.0.0.1:5672`. The controller
+   crash-looped and its control queue filled.
+3. **Every image reference the mesh has built.** Each recorded build reads
+   `<registry-address>:5100/<module>/<artifact>@sha256:…`, and declarations carry that literal. Move the
+   registry and every fresh pull of a mesh image fails — a new node, a recreate after eviction.
+   Found by reading before the move; the first two were found by making it.
+
+All three are the same fact: an address was **written down** when the port was decided, rather than
+**read** from the node's settings when it is used. The first two live in genesis secrets; the third
+lives in stored data, which is worse — it is baked into every build ever recorded.
+
+Both outages were closed by hand with a forwarder on the old address to the new one. Two such
+forwarders are holding the control-node's mesh together while this is open. They are not the fix;
+they are the shape of the bug, made visible.
+
+## Why it matters beyond this instance
+
+A port that is a setting in nine places and a constant in three is a constant. The migration's
+whole method — take the predecessor's ports one service at a time — depends on every reader
+following the setting, and the readers that do not are exactly the control plane's own, which is
+the worst place for them: the failure is headlessness, and headlessness cannot be repaired through
+the mesh.
+
+The image reference case will bite any mesh that changes its registry's port, or moves its registry
+to another node, or ever has two registries. It also means an image is recorded by *where it was
+pushed* rather than *what it is*: the digest is the identity, the address is a route to it, and the
+mesh stores them as one string.
+
+This is the same hole [issue 085](../085-the-packages-port-given-at-genesis-is-not-a-setting/00-report.md)
+found for the packages port, fixed there for that one reader. It is not one reader; it is a
+category.
+
+## Open questions
+
+- Should the controller read its own store and broker addresses from the node's settings at start,
+  the way it composes them for every other module — and re-read them when they change, since it is
+  the thing that changes them?
+- Should a recorded build store the artifact's **digest and path** only, with the registry address
+  composed into the declaration from the node's current settings — so a reference is assembled
+  where it is used, never stored?
+- Is there a way to find the remaining constants mechanically — every place a foundation port
+  number appears as a literal — rather than one outage at a time?
@@ -0,0 +1,45 @@
+---
+status: located
+opened: 2026-09-23
+located-in: [mesh-host internal/apply]
+fixed-by:
+amended-design:
+---
+
+# 103 — A container is not recreated when a file it reads changes
+
+## What was observed
+
+The control-node, 2026-09-23. The store was given the predecessor's port. The host rewrote the
+forge's and the analytics service's environment files with the new port — correctly — and left
+both containers running with the old one in their environment. Both lost their database. Both
+stayed "up" and healthy-looking for the twenty minutes it took to notice, then answered 502.
+
+A restart did not help: a container reads its `env-file` when it is **created**, not when it
+starts, so `docker restart` handed both containers the same stale environment. Only removing them
+and letting the host recreate them fixed it.
+
+The host decides whether a container needs recreating by comparing a hash of its declared spec.
+The spec names the env file's *path*; the file's *content* is not part of it. So a change that
+alters everything the process will see alters nothing the host compares.
+
+## Why it matters beyond this instance
+
+Every module with an `env-file` — which is most of them — has a configuration the host writes and a
+container that reads it once. Any change to that configuration that the host applies without
+recreating the container is applied to the disk and not to the service. The mesh then reports the
+node as running what it was told, because the file is right; only the process is wrong.
+
+The ports move is the obvious trigger, and it is the migration's whole method. But a rotated
+credential, a re-provisioned database, a changed binding — anything the host substitutes into a
+file a container reads — has the same shape.
+
+## Open questions
+
+- Should the spec hash cover the **content** of every file the container mounts or reads, so a
+  changed file recreates it — accepting that every such change is a restart of the service?
+- Or should the host recreate on content change only for `env-file` and mounted secrets, and leave
+  bind-mounted data alone — a file the service reads at start versus a directory it reads while
+  running?
+- How does the host report the difference between "the file is right" and "the process has read
+  it"? Today it cannot, and that is what made this silent.
@@ -0,0 +1,51 @@
+---
+status: located
+opened: 2026-09-23
+located-in: [mesh-host cmd/mesh-host]
+fixed-by:
+amended-design:
+---
+
+# 104 — `reconcile` applies the declaration the host was installed with, and refuses nothing
+
+## What was observed
+
+The control-node, 2026-09-23, adopted, twelve modules assigned, two services already cut over.
+Chasing why an assignment had not finished, an operator ran the host's own `reconcile` command by
+hand.
+
+It did not reconcile the node against the controller's declaration. It applied **the declaration
+the host carries** — the genesis bundle: foundation only, *converged*. In order: it recreated the
+store, tried to recreate the broker and failed on a held port, wrote the converged base filter to
+disk, enabled and started its service, and stopped at the first failing action with "nothing after
+it was attempted".
+
+The base filter is the ruleset [ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)
+exists to keep off an adopted node: input policy drop, three ports allowed. It closed the machine
+to the internet for about forty-five minutes — every site, the forge's path to its database — and
+was removed by hand.
+
+Nothing about the command said any of this would happen. It printed what it did after it did it.
+The node's mode was known to the host — it reports "adopted" in every report — and the
+declaration it applied said "converged", and no comparison was made.
+
+## Why it matters beyond this instance
+
+The host has two declarations and one command that does not say which it means. The bundle is
+right at genesis and stale a minute later; on an adopted node it is actively dangerous, because it
+is a converged declaration for a machine that is not converged. A command an operator would
+reasonably reach for under pressure — "make the machine match" — is the one that must not be run.
+
+The operator error here was real and is recorded as such. But a design that turns "I ran the
+obvious command" into a closed machine has a hole of its own, and the runbook's line "never bypass
+the controller" was standing in for a refusal the host should make itself.
+
+## Open questions
+
+- Should `reconcile` refuse outright when the declaration it carries is older than the one the
+  controller last sent, or says a different mode than the node reports — naming both?
+- Should a converged declaration be refused on an adopted node at the point of application,
+  whatever command delivered it, since the mode is a fact the host already knows?
+- Should the host preview before applying from a file — the way `converge` previews — and stop at
+  the first action *before* running it rather than after?
+- Does the bundle need to remain applicable after genesis at all, or should genesis consume it?
@@ -0,0 +1,34 @@
+---
+status: open
+opened: 2026-09-23
+located-in: []
+fixed-by:
+amended-design:
+---
+
+# 105 — The hub of the private network is a placement, not a seat
+
+## What was observed
+
+Reading the registry of a one-node mesh, 2026-09-23. The private network claims a seat,
+`the-private-network`, scoped to the **node** — because every node has an interface and a
+mesh-scoped seat would refuse the second machine. The fact that matters, *which node is the hub the
+others rendezvous at*, is a placement record (`overlay place <node> hub`) and no seat at all.
+
+## Why it matters beyond this instance
+
+The four foundation seats all say the same thing: *there is exactly one of me in this mesh*
+([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)). "There is
+exactly one hub" is that shape. As a placement it is refused by nothing: two nodes can be placed as
+hub, and the mesh would compute a graph with two rendezvous points and say nothing.
+
+It also confuses the reading. Asked "who provides the private network", the registry answers with
+a node-scoped seat held by every node, which is true and not what was asked.
+
+## Open questions
+
+- Should the hub be a mesh-scoped seat — `the-hub`, or the network's own name — claimed by the
+  node that is placed there, refused elsewhere?
+- Does the per-node seat still say anything once the hub is a seat, or is it the interface's
+  presence restated?
+- What else in the mesh is "exactly one" and recorded as a placement rather than a seat?
@@ -0,0 +1,33 @@
+---
+status: open
+opened: 2026-09-23
+located-in: []
+fixed-by:
+amended-design:
+---
+
+# 106 — The vault claims no seat, so nothing refuses a second one
+
+## What was observed
+
+Reading the registry, 2026-09-23. The vault provides `secret` to the whole mesh and **claims no
+seat**. The store, the broker, the controller and the catalogue each claim one, named after
+themselves, so that a second claimant is refused at resolution
+([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)). The vault
+was added after that record and did not inherit the rule.
+
+## Why it matters beyond this instance
+
+The vault holds every credential the mesh mints. A second vault, assigned by mistake or by a module
+that provides `secret` itself, would answer requirements the first was answering, and nothing in
+resolution would object. Of all the components to allow two of silently, this is the one to allow
+least.
+
+The rule is already written; this is an instance it was not applied to. Worth asking whether
+others were missed the same way — every provider added after 0079.
+
+## Open questions
+
+- A mesh-scoped seat `mesh-vault`, by the 0079 convention — is there any reason not to?
+- Should a provider of a mesh-scoped provision be required to claim a seat, or say explicitly that
+  more than one is allowed, so the omission cannot recur?