2026-09-23 20:54:54 +00:00
8 changed files with 365 additions and 1 deletions
@@ -0,0 +1,117 @@
 ---
 topic: the mesh
 status: accepted
 date: 2026-09-23
 deciders: jochen
 reconstructed: false
 extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
 ---
 # 105. The mesh adopts the predecessor's tunnel in place
 ## Context
 The control-node is adopted and its store and broker have been moved onto the ports the
 predecessor served them on, so the predecessor's other machines keep reaching them. They reach
 them **over the predecessor's tunnel**: a WireGuard interface on the control-node with three
 peers, an address range, and a port the hosting provider already lets through. The mesh's own
 private network runs beside it on a second interface, a second range and a second port — one the
 provider does not let through, so no other machine can join the mesh
 ([research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md), the runbook's
 measurement of the upstream filter).
 [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) says the mesh's range *must not
 overlap a tunnel the predecessor still runs*. That was written for coexistence. It leaves the
 migration with two tunnels for as long as any predecessor machine exists, and the second one
 unreachable.
 The mesh already adopts in place where the thing found is the thing it would have raised: the
 store and the broker were taken over as modules, keyed on the container that was running
 ([ADR 0078](0078-the-store-and-broker-are-modules.md)). A WireGuard interface is the same shape: a
 private key, a listening port, a list of peers by public key, an address. The mesh's is not
 different in kind from the predecessor's; it is a second one.
 The operator's instruction: take over the tunnel interface, same range — everything stays the same.
 ## Considered Options
 1. **Two tunnels until the last predecessor machine is gone.** Rejected: the mesh's stays
   unreachable from outside, so no machine can join, so the last predecessor machine is never gone.
 2. **Move the mesh's tunnel onto the predecessor's port with the mesh's own key and range.** The
   peers' packets arrive and are dropped — WireGuard authenticates by key, and the mesh's key is not
   the one they know. Every other machine loses its tunnel until it enrols, and it enrols over a bus
   it reaches through that tunnel. Rejected.
 3. **Adopt the predecessor's tunnel in place: its private key, its peers, its range, its port.**
   Adopted.
 ## Decision
 **On an adopted node that is the hub, the private network takes over the tunnel it finds.** The
 mesh's interface is raised with the found interface's **private key**, on its **port**, with its
 **address and range**, and every **peer** the found interface had — public key, allowed address —
 carried into the mesh's peer list as a peer not yet enrolled. The found interface is stopped, never
 flushed; its configuration stays on disk, kept like any held file.
 **Nothing a peer knows changes.** A predecessor machine keeps the same server key, the same
 endpoint, the same address and the same route; it cannot tell the tunnel changed hands. When that
 machine enrols, it keeps its address: the mesh assigns an enrolling node the address the tunnel
 already had for its key, and only a node with no such address is given a fresh one from the range.
 **The mesh's own addresses are the range's.** The controller composes every node's private address
 from the tunnel it holds, so adopting the predecessor's range moves the mesh's addresses with it —
 the hub's, and every binding, hosts-file entry and endpoint derived from it. Those are readers of
 the setting; they follow it, per ADR 0100's rule for ports. A reader that does not follow is
 [issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md).
 **This narrows ADR 0100.** Its rule that the range must not overlap a tunnel the predecessor still
 runs applies to a node that is *not* adopting the tunnel: where the found tunnel is left running
 beside the mesh's, the ranges must differ. Where it is adopted, there is one tunnel and one range.
 **The guard's question answers itself.** [ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
 admits the mesh's own ports only from the private network's interface. With one tunnel, the
 predecessor's peers arrive on it, and nothing needs to be admitted from an interface the mesh does
 not own.
 ## Consequences
 - **The order of a migration changes.** The hub's tunnel can be taken as soon as the node is
  adopted, before any service — and should be, because it is what lets other machines join. The
  runbook's step order is amended.
 - **The hub takes the provider's open port for free.** The port the predecessor's tunnel used is
  by definition one the provider passes.
 - **A peer's identity precedes its enrolment.** The mesh holds public keys and addresses for
  machines it has no record of. They are peers of the tunnel, not nodes of the mesh, until they
  enrol; the registry must be able to say both.
 - **What got harder:** the private key of the found interface is read from the machine and becomes
  the mesh's — the one case where the mesh takes a credential it did not mint. It is sealed like
  any own secret from then on, and the found configuration file is kept, not copied further.
 - Two documents currently say the opposite: ADR 0100's non-overlap rule (narrowed above) and the
  runbook's port plan, which is amended with this record.
 ## How it is checked
 A lab bed prepares a hub the way the predecessor leaves one: a WireGuard interface with a key, a
 port, a range and two peers, each peer a second machine that reaches a service on the hub through
 the tunnel. Then:
 - **Adopted, the tunnel changes hands and the peers notice nothing**: the found interface is down
  and its file is on disk; the mesh's interface is up with the found key, port and address; each
  peer's service call succeeds before, during and after, with no reconfiguration on the peer.
 - **A peer enrols and keeps its address**: the machine joins the mesh over the tunnel it already
  has, and its node address is the one the tunnel held for it.
 - **A new machine gets a fresh address from the same range**, and reaches both the hub and the
  enrolled peer.
 - **Nothing derived from the address is stale**: every binding, hosts entry and endpoint the
  controller composes says the adopted range, before and after a push.
 Unit tests hold the controller to reading the hub's address and range from the adopted tunnel,
 assigning an enrolling node the address its key already had, and refusing to hand out an address
 the tunnel already holds; and the host to raising the mesh's interface with the found key and
 peers and stopping the found interface without flushing it.
 ## References
 - [ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md),
  [ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
 - [issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md)
 - [research 012 — the minimum viable node](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md)
@@ -93,6 +93,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0102** — [The mesh writes into a shared file, never over it](0102-the-mesh-writes-into-a-shared-file-never-over-it.md)
 - **0103** — [What an adopted node holds, and what its guard refuses](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md)
 - **0104** — [A provision may be answered by an adapter to the predecessor](0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md)
 - **0105** — [The mesh adopts the predecessor's tunnel in place](0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md)
 ### Its tiers, from the bottom up
@@ -7,9 +7,10 @@ code:
  - mesh-controller internal/identity/authority.go
  - mesh-host internal/identity/serving.go
  - mesh-host internal/apply (the service that reflects a rule set)
-updated: 2026-09-22
+updated: 2026-09-23
 decisions:
  - 02-DECISIONS/0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md
  - 02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md
  - 02-DECISIONS/0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md
  - 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md
  - 02-DECISIONS/0099-a-step-that-runs-once-names-what-it-reads.md
@@ -715,3 +716,19 @@ The list is worth having in one place, because it is most of the argument:
  resolvable inside the mesh, not only routable from outside it; the mechanism that writes
  `<node>.internal` into containers does not yet also write the routed names, which is why an
  internal issuer cannot currently validate one without a hand-placed entry.
 ## The hub adopts the predecessor's tunnel
 *2026-09-23, [ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md).*
 On an adopted node that is the hub, the private network is not raised beside the tunnel it finds;
 it **takes it over**: the found interface's private key, its port, its address and range, and every
 peer it had, carried as peers not yet enrolled. The found interface is stopped, its configuration
 kept on disk. A predecessor machine sees the same server key at the same endpoint and cannot tell
 the tunnel changed hands; when it enrols, it keeps the address the tunnel already held for its key.
 The mesh's own addresses are the adopted range's — every binding, hosts entry and endpoint the
 controller composes follows it, as readers of a setting. ADR 0100's non-overlap rule applies only
 where a found tunnel is left running beside the mesh's; where it is adopted there is one tunnel.
 The guard admits the mesh's ports from that one interface, and the predecessor's peers arrive on
 it.
@@ -0,0 +1,66 @@
 ---
 status: located
 opened: 2026-09-23
 located-in: [mesh-controller cmd/mesh-bootstrap, mesh-controller internal/builder, mesh-controller internal/catalogue]
 fixed-by:
 amended-design:
 ---
 # 102 — An address recorded at genesis or at build does not follow the node's ports
 ## What was observed
 The control-node, 2026-09-23, migrating the foundation's store and broker onto the ports the
 predecessor served them on — the move [ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)
 describes: *the ports given become that node's settings, and every place that uses them reads them
 from there.*
 Most places did. When the store was given 6852, the bindings handed to the forge, the analytics
 service and the catalogue all said 6852. When the broker was given 5679, its consumers reconnected.
 Three places did not, and each took the mesh down in a different way:
 1. **The controller's own store address.** A secret written at genesis says `127.0.0.1:5432`. The
   store moved; the controller could not reach its inventory; the mesh was headless.
 2. **The controller's own broker address.** The same, for `127.0.0.1:5672`. The controller
   crash-looped and its control queue filled.
 3. **Every image reference the mesh has built.** Each recorded build reads
   `<registry-address>:5100/<module>/<artifact>@sha256:…`, and declarations carry that literal. Move the
   registry and every fresh pull of a mesh image fails — a new node, a recreate after eviction.
   Found by reading before the move; the first two were found by making it.
 All three are the same fact: an address was **written down** when the port was decided, rather than
 **read** from the node's settings when it is used. The first two live in genesis secrets; the third
 lives in stored data, which is worse — it is baked into every build ever recorded.
 Both outages were closed by hand with a forwarder on the old address to the new one. Two such
 forwarders are holding the control-node's mesh together while this is open. They are not the fix;
 they are the shape of the bug, made visible.
 ## Why it matters beyond this instance
 A port that is a setting in nine places and a constant in three is a constant. The migration's
 whole method — take the predecessor's ports one service at a time — depends on every reader
 following the setting, and the readers that do not are exactly the control plane's own, which is
 the worst place for them: the failure is headlessness, and headlessness cannot be repaired through
 the mesh.
 The image reference case will bite any mesh that changes its registry's port, or moves its registry
 to another node, or ever has two registries. It also means an image is recorded by *where it was
 pushed* rather than *what it is*: the digest is the identity, the address is a route to it, and the
 mesh stores them as one string.
 This is the same hole [issue 085](../085-the-packages-port-given-at-genesis-is-not-a-setting/00-report.md)
 found for the packages port, fixed there for that one reader. It is not one reader; it is a
 category.
 ## Open questions
 - Should the controller read its own store and broker addresses from the node's settings at start,
  the way it composes them for every other module — and re-read them when they change, since it is
  the thing that changes them?
 - Should a recorded build store the artifact's **digest and path** only, with the registry address
  composed into the declaration from the node's current settings — so a reference is assembled
  where it is used, never stored?
 - Is there a way to find the remaining constants mechanically — every place a foundation port
  number appears as a literal — rather than one outage at a time?
@@ -0,0 +1,45 @@
 ---
 status: located
 opened: 2026-09-23
 located-in: [mesh-host internal/apply]
 fixed-by:
 amended-design:
 ---
 # 103 — A container is not recreated when a file it reads changes
 ## What was observed
 The control-node, 2026-09-23. The store was given the predecessor's port. The host rewrote the
 forge's and the analytics service's environment files with the new port — correctly — and left
 both containers running with the old one in their environment. Both lost their database. Both
 stayed "up" and healthy-looking for the twenty minutes it took to notice, then answered 502.
 A restart did not help: a container reads its `env-file` when it is **created**, not when it
 starts, so `docker restart` handed both containers the same stale environment. Only removing them
 and letting the host recreate them fixed it.
 The host decides whether a container needs recreating by comparing a hash of its declared spec.
 The spec names the env file's *path*; the file's *content* is not part of it. So a change that
 alters everything the process will see alters nothing the host compares.
 ## Why it matters beyond this instance
 Every module with an `env-file` — which is most of them — has a configuration the host writes and a
 container that reads it once. Any change to that configuration that the host applies without
 recreating the container is applied to the disk and not to the service. The mesh then reports the
 node as running what it was told, because the file is right; only the process is wrong.
 The ports move is the obvious trigger, and it is the migration's whole method. But a rotated
 credential, a re-provisioned database, a changed binding — anything the host substitutes into a
 file a container reads — has the same shape.
 ## Open questions
 - Should the spec hash cover the **content** of every file the container mounts or reads, so a
  changed file recreates it — accepting that every such change is a restart of the service?
 - Or should the host recreate on content change only for `env-file` and mounted secrets, and leave
  bind-mounted data alone — a file the service reads at start versus a directory it reads while
  running?
 - How does the host report the difference between "the file is right" and "the process has read
  it"? Today it cannot, and that is what made this silent.
@@ -0,0 +1,51 @@
 ---
 status: located
 opened: 2026-09-23
 located-in: [mesh-host cmd/mesh-host]
 fixed-by:
 amended-design:
 ---
 # 104 — `reconcile` applies the declaration the host was installed with, and refuses nothing
 ## What was observed
 The control-node, 2026-09-23, adopted, twelve modules assigned, two services already cut over.
 Chasing why an assignment had not finished, an operator ran the host's own `reconcile` command by
 hand.
 It did not reconcile the node against the controller's declaration. It applied **the declaration
 the host carries** — the genesis bundle: foundation only, *converged*. In order: it recreated the
 store, tried to recreate the broker and failed on a held port, wrote the converged base filter to
 disk, enabled and started its service, and stopped at the first failing action with "nothing after
 it was attempted".
 The base filter is the ruleset [ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md)
 exists to keep off an adopted node: input policy drop, three ports allowed. It closed the machine
 to the internet for about forty-five minutes — every site, the forge's path to its database — and
 was removed by hand.
 Nothing about the command said any of this would happen. It printed what it did after it did it.
 The node's mode was known to the host — it reports "adopted" in every report — and the
 declaration it applied said "converged", and no comparison was made.
 ## Why it matters beyond this instance
 The host has two declarations and one command that does not say which it means. The bundle is
 right at genesis and stale a minute later; on an adopted node it is actively dangerous, because it
 is a converged declaration for a machine that is not converged. A command an operator would
 reasonably reach for under pressure — "make the machine match" — is the one that must not be run.
 The operator error here was real and is recorded as such. But a design that turns "I ran the
 obvious command" into a closed machine has a hole of its own, and the runbook's line "never bypass
 the controller" was standing in for a refusal the host should make itself.
 ## Open questions
 - Should `reconcile` refuse outright when the declaration it carries is older than the one the
  controller last sent, or says a different mode than the node reports — naming both?
 - Should a converged declaration be refused on an adopted node at the point of application,
  whatever command delivered it, since the mode is a fact the host already knows?
 - Should the host preview before applying from a file — the way `converge` previews — and stop at
  the first action *before* running it rather than after?
 - Does the bundle need to remain applicable after genesis at all, or should genesis consume it?
@@ -0,0 +1,34 @@
 ---
 status: open
 opened: 2026-09-23
 located-in: []
 fixed-by:
 amended-design:
 ---
 # 105 — The hub of the private network is a placement, not a seat
 ## What was observed
 Reading the registry of a one-node mesh, 2026-09-23. The private network claims a seat,
 `the-private-network`, scoped to the **node** — because every node has an interface and a
 mesh-scoped seat would refuse the second machine. The fact that matters, *which node is the hub the
 others rendezvous at*, is a placement record (`overlay place <node> hub`) and no seat at all.
 ## Why it matters beyond this instance
 The four foundation seats all say the same thing: *there is exactly one of me in this mesh*
 ([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)). "There is
 exactly one hub" is that shape. As a placement it is refused by nothing: two nodes can be placed as
 hub, and the mesh would compute a graph with two rendezvous points and say nothing.
 It also confuses the reading. Asked "who provides the private network", the registry answers with
 a node-scoped seat held by every node, which is true and not what was asked.
 ## Open questions
 - Should the hub be a mesh-scoped seat — `the-hub`, or the network's own name — claimed by the
  node that is placed there, refused elsewhere?
 - Does the per-node seat still say anything once the hub is a seat, or is it the interface's
  presence restated?
 - What else in the mesh is "exactly one" and recorded as a placement rather than a seat?
@@ -0,0 +1,33 @@
 ---
 status: open
 opened: 2026-09-23
 located-in: []
 fixed-by:
 amended-design:
 ---
 # 106 — The vault claims no seat, so nothing refuses a second one
 ## What was observed
 Reading the registry, 2026-09-23. The vault provides `secret` to the whole mesh and **claims no
 seat**. The store, the broker, the controller and the catalogue each claim one, named after
 themselves, so that a second claimant is refused at resolution
 ([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)). The vault
 was added after that record and did not inherit the rule.
 ## Why it matters beyond this instance
 The vault holds every credential the mesh mints. A second vault, assigned by mistake or by a module
 that provides `secret` itself, would answer requirements the first was answering, and nothing in
 resolution would object. Of all the components to allow two of silently, this is the one to allow
 least.
 The rule is already written; this is an instance it was not applied to. Worth asking whether
 others were missed the same way — every provider added after 0079.
 ## Open questions
 - A mesh-scoped seat `mesh-vault`, by the 0079 convention — is there any reason not to?
 - Should a provider of a mesh-scoped provision be required to claim a seat, or say explicitly that
  more than one is allowed, so the omission cannot recur?