From cb2117f1c449534d9f3e25275573c7d5cd82da65 Mon Sep 17 00:00:00 2001 From: jochen Date: Wed, 23 Sep 2026 22:50:10 +0200 Subject: [PATCH] =?UTF-8?q?Issues=20102=E2=80=93106=20and=20ADR=200105=20f?= =?UTF-8?q?rom=20the=20core=20migration?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two birth-address outages and a registry that would have been the third; a container that keeps a stale environment after its file changes; a host command that applied a converged declaration to an adopted node; the hub and the vault without seats. And the decision the operator made under it all: the hub adopts the predecessor's tunnel in place, key and peers and range and port. --- ...adopts-the-predecessors-tunnel-in-place.md | 117 ++++++++++++++++++ 02-DECISIONS/README.md | 1 + 03-DESIGN/01-to-be/08-connectivity.md | 19 ++- .../00-report.md | 66 ++++++++++ .../00-report.md | 45 +++++++ .../00-report.md | 51 ++++++++ .../00-report.md | 34 +++++ .../106-the-vault-claims-no-seat/00-report.md | 33 +++++ 8 files changed, 365 insertions(+), 1 deletion(-) create mode 100644 02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md create mode 100644 04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md create mode 100644 04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md create mode 100644 04-ISSUES/104-reconcile-applies-a-stale-declaration-and-refuses-nothing/00-report.md create mode 100644 04-ISSUES/105-the-hub-of-the-private-network-is-not-a-seat/00-report.md create mode 100644 04-ISSUES/106-the-vault-claims-no-seat/00-report.md diff --git a/02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md b/02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md new file mode 100644 index 0000000..d99a476 --- /dev/null +++ b/02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md @@ -0,0 +1,117 @@ +--- +topic: the mesh +status: accepted +date: 2026-09-23 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md +--- + +# 105. The mesh adopts the predecessor's tunnel in place + +## Context + +The control-node is adopted and its store and broker have been moved onto the ports the +predecessor served them on, so the predecessor's other machines keep reaching them. They reach +them **over the predecessor's tunnel**: a WireGuard interface on the control-node with three +peers, an address range, and a port the hosting provider already lets through. The mesh's own +private network runs beside it on a second interface, a second range and a second port — one the +provider does not let through, so no other machine can join the mesh +([research 012](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md), the runbook's +measurement of the upstream filter). + +[ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md) says the mesh's range *must not +overlap a tunnel the predecessor still runs*. That was written for coexistence. It leaves the +migration with two tunnels for as long as any predecessor machine exists, and the second one +unreachable. + +The mesh already adopts in place where the thing found is the thing it would have raised: the +store and the broker were taken over as modules, keyed on the container that was running +([ADR 0078](0078-the-store-and-broker-are-modules.md)). A WireGuard interface is the same shape: a +private key, a listening port, a list of peers by public key, an address. The mesh's is not +different in kind from the predecessor's; it is a second one. + +The operator's instruction: take over the tunnel interface, same range — everything stays the same. + +## Considered Options + +1. **Two tunnels until the last predecessor machine is gone.** Rejected: the mesh's stays + unreachable from outside, so no machine can join, so the last predecessor machine is never gone. +2. **Move the mesh's tunnel onto the predecessor's port with the mesh's own key and range.** The + peers' packets arrive and are dropped — WireGuard authenticates by key, and the mesh's key is not + the one they know. Every other machine loses its tunnel until it enrols, and it enrols over a bus + it reaches through that tunnel. Rejected. +3. **Adopt the predecessor's tunnel in place: its private key, its peers, its range, its port.** + Adopted. + +## Decision + +**On an adopted node that is the hub, the private network takes over the tunnel it finds.** The +mesh's interface is raised with the found interface's **private key**, on its **port**, with its +**address and range**, and every **peer** the found interface had — public key, allowed address — +carried into the mesh's peer list as a peer not yet enrolled. The found interface is stopped, never +flushed; its configuration stays on disk, kept like any held file. + +**Nothing a peer knows changes.** A predecessor machine keeps the same server key, the same +endpoint, the same address and the same route; it cannot tell the tunnel changed hands. When that +machine enrols, it keeps its address: the mesh assigns an enrolling node the address the tunnel +already had for its key, and only a node with no such address is given a fresh one from the range. + +**The mesh's own addresses are the range's.** The controller composes every node's private address +from the tunnel it holds, so adopting the predecessor's range moves the mesh's addresses with it — +the hub's, and every binding, hosts-file entry and endpoint derived from it. Those are readers of +the setting; they follow it, per ADR 0100's rule for ports. A reader that does not follow is +[issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md). + +**This narrows ADR 0100.** Its rule that the range must not overlap a tunnel the predecessor still +runs applies to a node that is *not* adopting the tunnel: where the found tunnel is left running +beside the mesh's, the ranges must differ. Where it is adopted, there is one tunnel and one range. + +**The guard's question answers itself.** [ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md) +admits the mesh's own ports only from the private network's interface. With one tunnel, the +predecessor's peers arrive on it, and nothing needs to be admitted from an interface the mesh does +not own. + +## Consequences + +- **The order of a migration changes.** The hub's tunnel can be taken as soon as the node is + adopted, before any service — and should be, because it is what lets other machines join. The + runbook's step order is amended. +- **The hub takes the provider's open port for free.** The port the predecessor's tunnel used is + by definition one the provider passes. +- **A peer's identity precedes its enrolment.** The mesh holds public keys and addresses for + machines it has no record of. They are peers of the tunnel, not nodes of the mesh, until they + enrol; the registry must be able to say both. +- **What got harder:** the private key of the found interface is read from the machine and becomes + the mesh's — the one case where the mesh takes a credential it did not mint. It is sealed like + any own secret from then on, and the found configuration file is kept, not copied further. +- Two documents currently say the opposite: ADR 0100's non-overlap rule (narrowed above) and the + runbook's port plan, which is amended with this record. + +## How it is checked + +A lab bed prepares a hub the way the predecessor leaves one: a WireGuard interface with a key, a +port, a range and two peers, each peer a second machine that reaches a service on the hub through +the tunnel. Then: + +- **Adopted, the tunnel changes hands and the peers notice nothing**: the found interface is down + and its file is on disk; the mesh's interface is up with the found key, port and address; each + peer's service call succeeds before, during and after, with no reconfiguration on the peer. +- **A peer enrols and keeps its address**: the machine joins the mesh over the tunnel it already + has, and its node address is the one the tunnel held for it. +- **A new machine gets a fresh address from the same range**, and reaches both the hub and the + enrolled peer. +- **Nothing derived from the address is stale**: every binding, hosts entry and endpoint the + controller composes says the adopted range, before and after a push. + +Unit tests hold the controller to reading the hub's address and range from the adopted tunnel, +assigning an enrolling node the address its key already had, and refusing to hand out an address +the tunnel already holds; and the host to raising the mesh's interface with the found key and +peers and stopping the found interface without flushing it. + +## References + +- [ADR 0078](0078-the-store-and-broker-are-modules.md), [ADR 0100](0100-a-node-in-use-is-adopted-before-it-is-converged.md), + [ADR 0103](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md) +- [issue 102](../04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md) +- [research 012 — the minimum viable node](../01-RESEARCH/012-the-minimum-viable-node/00-overview.md) diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 62260b6..c152888 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -93,6 +93,7 @@ python3 00-META/checks/index.py fail if stale - **0102** — [The mesh writes into a shared file, never over it](0102-the-mesh-writes-into-a-shared-file-never-over-it.md) - **0103** — [What an adopted node holds, and what its guard refuses](0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md) - **0104** — [A provision may be answered by an adapter to the predecessor](0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md) +- **0105** — [The mesh adopts the predecessor's tunnel in place](0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/08-connectivity.md b/03-DESIGN/01-to-be/08-connectivity.md index 5cccb35..28aa8c0 100644 --- a/03-DESIGN/01-to-be/08-connectivity.md +++ b/03-DESIGN/01-to-be/08-connectivity.md @@ -7,9 +7,10 @@ code: - mesh-controller internal/identity/authority.go - mesh-host internal/identity/serving.go - mesh-host internal/apply (the service that reflects a rule set) -updated: 2026-09-22 +updated: 2026-09-23 decisions: - 02-DECISIONS/0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md + - 02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md - 02-DECISIONS/0103-what-an-adopted-node-holds-and-what-its-guard-refuses.md - 02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md - 02-DECISIONS/0099-a-step-that-runs-once-names-what-it-reads.md @@ -715,3 +716,19 @@ The list is worth having in one place, because it is most of the argument: resolvable inside the mesh, not only routable from outside it; the mechanism that writes `.internal` into containers does not yet also write the routed names, which is why an internal issuer cannot currently validate one without a hand-placed entry. + +## The hub adopts the predecessor's tunnel + +*2026-09-23, [ADR 0105](../../02-DECISIONS/0105-the-mesh-adopts-the-predecessors-tunnel-in-place.md).* + +On an adopted node that is the hub, the private network is not raised beside the tunnel it finds; +it **takes it over**: the found interface's private key, its port, its address and range, and every +peer it had, carried as peers not yet enrolled. The found interface is stopped, its configuration +kept on disk. A predecessor machine sees the same server key at the same endpoint and cannot tell +the tunnel changed hands; when it enrols, it keeps the address the tunnel already held for its key. + +The mesh's own addresses are the adopted range's — every binding, hosts entry and endpoint the +controller composes follows it, as readers of a setting. ADR 0100's non-overlap rule applies only +where a found tunnel is left running beside the mesh's; where it is adopted there is one tunnel. +The guard admits the mesh's ports from that one interface, and the predecessor's peers arrive on +it. diff --git a/04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md b/04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md new file mode 100644 index 0000000..a12d166 --- /dev/null +++ b/04-ISSUES/102-an-address-recorded-at-genesis-or-build-does-not-follow-the-nodes-ports/00-report.md @@ -0,0 +1,66 @@ +--- +status: located +opened: 2026-09-23 +located-in: [mesh-controller cmd/mesh-bootstrap, mesh-controller internal/builder, mesh-controller internal/catalogue] +fixed-by: +amended-design: +--- + +# 102 — An address recorded at genesis or at build does not follow the node's ports + +## What was observed + +The control-node, 2026-09-23, migrating the foundation's store and broker onto the ports the +predecessor served them on — the move [ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) +describes: *the ports given become that node's settings, and every place that uses them reads them +from there.* + +Most places did. When the store was given 6852, the bindings handed to the forge, the analytics +service and the catalogue all said 6852. When the broker was given 5679, its consumers reconnected. + +Three places did not, and each took the mesh down in a different way: + +1. **The controller's own store address.** A secret written at genesis says `127.0.0.1:5432`. The + store moved; the controller could not reach its inventory; the mesh was headless. +2. **The controller's own broker address.** The same, for `127.0.0.1:5672`. The controller + crash-looped and its control queue filled. +3. **Every image reference the mesh has built.** Each recorded build reads + `:5100//@sha256:…`, and declarations carry that literal. Move the + registry and every fresh pull of a mesh image fails — a new node, a recreate after eviction. + Found by reading before the move; the first two were found by making it. + +All three are the same fact: an address was **written down** when the port was decided, rather than +**read** from the node's settings when it is used. The first two live in genesis secrets; the third +lives in stored data, which is worse — it is baked into every build ever recorded. + +Both outages were closed by hand with a forwarder on the old address to the new one. Two such +forwarders are holding the control-node's mesh together while this is open. They are not the fix; +they are the shape of the bug, made visible. + +## Why it matters beyond this instance + +A port that is a setting in nine places and a constant in three is a constant. The migration's +whole method — take the predecessor's ports one service at a time — depends on every reader +following the setting, and the readers that do not are exactly the control plane's own, which is +the worst place for them: the failure is headlessness, and headlessness cannot be repaired through +the mesh. + +The image reference case will bite any mesh that changes its registry's port, or moves its registry +to another node, or ever has two registries. It also means an image is recorded by *where it was +pushed* rather than *what it is*: the digest is the identity, the address is a route to it, and the +mesh stores them as one string. + +This is the same hole [issue 085](../085-the-packages-port-given-at-genesis-is-not-a-setting/00-report.md) +found for the packages port, fixed there for that one reader. It is not one reader; it is a +category. + +## Open questions + +- Should the controller read its own store and broker addresses from the node's settings at start, + the way it composes them for every other module — and re-read them when they change, since it is + the thing that changes them? +- Should a recorded build store the artifact's **digest and path** only, with the registry address + composed into the declaration from the node's current settings — so a reference is assembled + where it is used, never stored? +- Is there a way to find the remaining constants mechanically — every place a foundation port + number appears as a literal — rather than one outage at a time? diff --git a/04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md b/04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md new file mode 100644 index 0000000..f34684d --- /dev/null +++ b/04-ISSUES/103-a-container-is-not-recreated-when-a-file-it-reads-changes/00-report.md @@ -0,0 +1,45 @@ +--- +status: located +opened: 2026-09-23 +located-in: [mesh-host internal/apply] +fixed-by: +amended-design: +--- + +# 103 — A container is not recreated when a file it reads changes + +## What was observed + +The control-node, 2026-09-23. The store was given the predecessor's port. The host rewrote the +forge's and the analytics service's environment files with the new port — correctly — and left +both containers running with the old one in their environment. Both lost their database. Both +stayed "up" and healthy-looking for the twenty minutes it took to notice, then answered 502. + +A restart did not help: a container reads its `env-file` when it is **created**, not when it +starts, so `docker restart` handed both containers the same stale environment. Only removing them +and letting the host recreate them fixed it. + +The host decides whether a container needs recreating by comparing a hash of its declared spec. +The spec names the env file's *path*; the file's *content* is not part of it. So a change that +alters everything the process will see alters nothing the host compares. + +## Why it matters beyond this instance + +Every module with an `env-file` — which is most of them — has a configuration the host writes and a +container that reads it once. Any change to that configuration that the host applies without +recreating the container is applied to the disk and not to the service. The mesh then reports the +node as running what it was told, because the file is right; only the process is wrong. + +The ports move is the obvious trigger, and it is the migration's whole method. But a rotated +credential, a re-provisioned database, a changed binding — anything the host substitutes into a +file a container reads — has the same shape. + +## Open questions + +- Should the spec hash cover the **content** of every file the container mounts or reads, so a + changed file recreates it — accepting that every such change is a restart of the service? +- Or should the host recreate on content change only for `env-file` and mounted secrets, and leave + bind-mounted data alone — a file the service reads at start versus a directory it reads while + running? +- How does the host report the difference between "the file is right" and "the process has read + it"? Today it cannot, and that is what made this silent. diff --git a/04-ISSUES/104-reconcile-applies-a-stale-declaration-and-refuses-nothing/00-report.md b/04-ISSUES/104-reconcile-applies-a-stale-declaration-and-refuses-nothing/00-report.md new file mode 100644 index 0000000..085c0ff --- /dev/null +++ b/04-ISSUES/104-reconcile-applies-a-stale-declaration-and-refuses-nothing/00-report.md @@ -0,0 +1,51 @@ +--- +status: located +opened: 2026-09-23 +located-in: [mesh-host cmd/mesh-host] +fixed-by: +amended-design: +--- + +# 104 — `reconcile` applies the declaration the host was installed with, and refuses nothing + +## What was observed + +The control-node, 2026-09-23, adopted, twelve modules assigned, two services already cut over. +Chasing why an assignment had not finished, an operator ran the host's own `reconcile` command by +hand. + +It did not reconcile the node against the controller's declaration. It applied **the declaration +the host carries** — the genesis bundle: foundation only, *converged*. In order: it recreated the +store, tried to recreate the broker and failed on a held port, wrote the converged base filter to +disk, enabled and started its service, and stopped at the first failing action with "nothing after +it was attempted". + +The base filter is the ruleset [ADR 0100](../../02-DECISIONS/0100-a-node-in-use-is-adopted-before-it-is-converged.md) +exists to keep off an adopted node: input policy drop, three ports allowed. It closed the machine +to the internet for about forty-five minutes — every site, the forge's path to its database — and +was removed by hand. + +Nothing about the command said any of this would happen. It printed what it did after it did it. +The node's mode was known to the host — it reports "adopted" in every report — and the +declaration it applied said "converged", and no comparison was made. + +## Why it matters beyond this instance + +The host has two declarations and one command that does not say which it means. The bundle is +right at genesis and stale a minute later; on an adopted node it is actively dangerous, because it +is a converged declaration for a machine that is not converged. A command an operator would +reasonably reach for under pressure — "make the machine match" — is the one that must not be run. + +The operator error here was real and is recorded as such. But a design that turns "I ran the +obvious command" into a closed machine has a hole of its own, and the runbook's line "never bypass +the controller" was standing in for a refusal the host should make itself. + +## Open questions + +- Should `reconcile` refuse outright when the declaration it carries is older than the one the + controller last sent, or says a different mode than the node reports — naming both? +- Should a converged declaration be refused on an adopted node at the point of application, + whatever command delivered it, since the mode is a fact the host already knows? +- Should the host preview before applying from a file — the way `converge` previews — and stop at + the first action *before* running it rather than after? +- Does the bundle need to remain applicable after genesis at all, or should genesis consume it? diff --git a/04-ISSUES/105-the-hub-of-the-private-network-is-not-a-seat/00-report.md b/04-ISSUES/105-the-hub-of-the-private-network-is-not-a-seat/00-report.md new file mode 100644 index 0000000..9641d74 --- /dev/null +++ b/04-ISSUES/105-the-hub-of-the-private-network-is-not-a-seat/00-report.md @@ -0,0 +1,34 @@ +--- +status: open +opened: 2026-09-23 +located-in: [] +fixed-by: +amended-design: +--- + +# 105 — The hub of the private network is a placement, not a seat + +## What was observed + +Reading the registry of a one-node mesh, 2026-09-23. The private network claims a seat, +`the-private-network`, scoped to the **node** — because every node has an interface and a +mesh-scoped seat would refuse the second machine. The fact that matters, *which node is the hub the +others rendezvous at*, is a placement record (`overlay place hub`) and no seat at all. + +## Why it matters beyond this instance + +The four foundation seats all say the same thing: *there is exactly one of me in this mesh* +([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)). "There is +exactly one hub" is that shape. As a placement it is refused by nothing: two nodes can be placed as +hub, and the mesh would compute a graph with two rendezvous points and say nothing. + +It also confuses the reading. Asked "who provides the private network", the registry answers with +a node-scoped seat held by every node, which is true and not what was asked. + +## Open questions + +- Should the hub be a mesh-scoped seat — `the-hub`, or the network's own name — claimed by the + node that is placed there, refused elsewhere? +- Does the per-node seat still say anything once the hub is a seat, or is it the interface's + presence restated? +- What else in the mesh is "exactly one" and recorded as a placement rather than a seat? diff --git a/04-ISSUES/106-the-vault-claims-no-seat/00-report.md b/04-ISSUES/106-the-vault-claims-no-seat/00-report.md new file mode 100644 index 0000000..9f2f9d4 --- /dev/null +++ b/04-ISSUES/106-the-vault-claims-no-seat/00-report.md @@ -0,0 +1,33 @@ +--- +status: open +opened: 2026-09-23 +located-in: [] +fixed-by: +amended-design: +--- + +# 106 — The vault claims no seat, so nothing refuses a second one + +## What was observed + +Reading the registry, 2026-09-23. The vault provides `secret` to the whole mesh and **claims no +seat**. The store, the broker, the controller and the catalogue each claim one, named after +themselves, so that a second claimant is refused at resolution +([ADR 0079](../../02-DECISIONS/0079-the-foundation-seats-are-named-after-their-servers.md)). The vault +was added after that record and did not inherit the rule. + +## Why it matters beyond this instance + +The vault holds every credential the mesh mints. A second vault, assigned by mistake or by a module +that provides `secret` itself, would answer requirements the first was answering, and nothing in +resolution would object. Of all the components to allow two of silently, this is the one to allow +least. + +The rule is already written; this is an instance it was not applied to. Worth asking whether +others were missed the same way — every provider added after 0079. + +## Open questions + +- A mesh-scoped seat `mesh-vault`, by the 0079 convention — is there any reason not to? +- Should a provider of a mesh-scoped provision be required to claim a seat, or say explicitly that + more than one is allowed, so the omission cannot recur?