Issue 194: the host's own former archive stops every machine applying anything

Rule 5 of ADR 0163 (a former target is removed) met issue 162 (an archive has no removal) in the host's own archive, the first time a host carrying former targets replaced itself; every machine applied nothing from then on.
2026-10-02 02:42:19 +02:00
5 changed files with 65 additions and 160 deletions
@@ -124,30 +124,6 @@ This corrects a fact, not the decision: one statement per endpoint, three things
 none of them deciding on its own, all stand. The table in the decision should be read with the filter
 column applying to an unrouted endpoint.
 ## Progressive insight — 2026-10-02, from issue 191
 **For a routed endpoint, "the proxy serves the internal name" has to mean "serves it to the private
 network", and only the proxy can make it mean that.** The decision says `internal` means the proxy
 serves the internal name and not the public one. It does not say to whom, and the proxy answered
 every name it routes to any request that carried it, on the same listeners as its public names. A
 name being internal kept nobody out: a request from the internet only had to send it. While every
 routed endpoint also had a public name, nothing showed it. Once an endpoint could be internal alone
 ([issue 191](../04-ISSUES/191-a-route-with-only-an-internal-name-is-dropped/00-report.md)), serving
 its name to everyone would have published exactly what `internal` was chosen to keep private.
 The earlier insight above says the port is not the path for a routed endpoint. This is its other
 half: the proxy is the path, so the proxy is where `internal` is enforced. It serves an internal name
 only to a request from the private network — the mesh's range, the machine itself, or one of its own
 container networks ([ADR 0144](0144-anything-on-a-machine-may-call-anything-on-it.md)). To anyone else, the name is
 answered as one never routed, in the handshake and in the request, and not listed among the names it
 serves. This holds for the internal name of a `both` endpoint too, whose outsiders have its public
 name.
 The decision, the options and the consequences stand: one statement per endpoint, three things
 derived from it. Checked in the proxy's own tests: an internal-only name is served to the mesh
 range, to loopback and to a container bridge, and refused, unlisted and uncertified for a request
 from outside; with no range given, it is served to the machine alone.
 ## Consequences
 - **A manifest gains endpoint names, and a route contribution names an endpoint instead of a port.**
@@ -7,7 +7,7 @@ code:
  - mesh-controller internal/identity/authority.go
  - mesh-host internal/identity/serving.go
  - mesh-host internal/apply (the service that reflects a rule set)
-updated: 2026-10-02
+updated: 2026-09-30
 decisions:
  - 02-DECISIONS/0148-the-meshs-names-are-resolved-not-copied-into-containers.md
  - 02-DECISIONS/0151-a-routes-internal-name-is-composed-under-the-node-that-serves-it.md
@@ -842,15 +842,6 @@ One value, three readers:
 | `public` | the machine port, to anywhere | the public name | the public authority |
 | `both` | the machine port, to anywhere | both names | each name's own authority |
 *2026-10-02.* **The proxy serves an internal name to the private network only**
 ([ADR 0138](../../02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md),
 its insight of this date). It answers public and internal names on the same listeners, so the name a
 request carries is the request's own claim, not where the request came from. An internal name is
 served to the mesh's range, to the machine itself and to its own container networks; to anyone else
 it is answered as a name never routed, in the handshake as well as the request. Without this, an
 endpoint with reach `internal` would be public under a name that is easy to guess
 ([issue 191](../../04-ISSUES/191-a-route-with-only-an-internal-name-is-dropped/00-report.md)).
 **An endpoint that is not routed is reached and never named.** No route contribution means no name is
 composed and no certificate requested, while the filter still acts on it. That is the case the model
 could not express at all, and it is the ordinary case for anything that is not HTTP.
@@ -1,61 +0,0 @@
 ---
 status: located
 opened: 2026-10-01
 located-in: [mesh-controller examples/route-proxy/main.go (routesFrom requires a route's public `name` and treats `internal-name` only as an alias of it; the handler serves every routed name to any source), mesh-catalog modules/route-proxy/module.json (the proxy is not told the private network's range)]
 fixed-by:
 amended-design: [03-DESIGN/01-to-be/08-connectivity.md]
 ---
 # 191 — A route with only an internal name is dropped as naming nothing
 ## What was observed
 A module whose endpoint reaches only the private network could not be reached by its internal name.
 The module ran and answered on its own port. Its route's internal name resolved to the serving node.
 The request failed during the TLS handshake:
 ```
 http: TLS handshake error from …: no public route for "unifi.home-server.internal" in this mesh,
  so no certificate is asked for
 ```
 The proxy's own log said why, every time it re-read its routes:
 ```
 unifi on home-server asked for a route and named nothing; skipped
 ```
 The route it skipped was not empty. The mesh had given it an endpoint, a port, a scheme and an internal
 name, and no public name:
 | route | `name` | `internal-name` | served |
 |---|---|---|---|
 | home-assistant | a public name | `home-assistant.home-server.internal` | under both |
 | unifi | — | `unifi.home-server.internal` | under neither |
 Three other modules on the same node were skipped with the same line on the same pass.
 ## Why it matters
 **Reach is decided in one place, and the proxy reads the old shape of the decision.**
 [ADR 0138](../../02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md)
 made an endpoint's reach decide which names exist. The controller composes the public name, the
 internal name, or both, and composes no name that nobody asked for
 ([issue 140](../140-an-endpoints-reach-is-not-declared/01-resolution.md)). An endpoint that reaches
 only the private network is the ordinary case for anything that should not face the internet. It is
 exactly the case the proxy drops.
 **The failure is quiet and points the wrong way.** Nothing marks the module unhealthy. The handshake
 error says *no public route*, which reads as a certificate fault on the reader's side. The line that
 gives the real cause is one of four identical lines repeated every few seconds in a log nobody reads
 until they already suspect the proxy.
 **The opposite move is not a workaround.** Giving the endpoint public reach makes the proxy serve it.
 It also publishes an administration interface to the internet to get a name on the private network.
 ## Open questions
 - Should the proxy refuse a route it cannot serve in a way the controller or an operator sees, not
  only in its own log? The same silent skip covers a route with no usable port or an unknown scheme.
 - What checks that what the controller composes and what the proxy serves stay the same shape? ADR
  0138 changed one side and nothing failed on the other.
@@ -1,65 +0,0 @@
 # Diagnosis
 *2026-10-01.*
 **Ruled out first: the module itself.** Its container was up and had not restarted. The controller's
 status endpoint answered on its own port with `"up": true`. The tool wrapper beside it was serving
 all its tools.
 **Ruled out: name resolution.** The internal name resolved to the serving node's private-network
 address, which is where the proxy listens. Plain HTTP to the name reached the proxy and got a 404.
 HTTPS failed in the handshake, and the proxy logged that it had no route for the name.
 **The route as the proxy received it.** The mesh-written route file held a complete contribution
 for the module: endpoint `web`, port, scheme `https`, `insecure`, a label, and `internal-name`. It had
 no `name`. That is what the controller composes for an endpoint whose reach stops at the private
 network (ADR 0138, `composeName`). The contribution was correct.
 **Located: `routesFrom` in the proxy.** It reads `name` first and skips the contribution if `name`
 is empty. It reads `internal-name` only at the end, as a second host for a rule that already has a
 public one. So the proxy can serve an internal name only next to a public one. That matched the
 mesh before ADR 0138, when both names were always composed. It has been wrong since then.
 The other half of the proxy already handles the case. Certificates for a host are split by whether it
 is in the public set: hosts outside it go to the internal authority, and only hosts inside it are
 eligible for ACME. A host that is only ever an internal name falls on the correct side of both checks
 without change. For certificates, only reading the route was wrong; who may reach the route is the next section.
 **The fix.** `routesFrom` takes a route that names either host, serves each name it carries, and
 marks only the public one as public. It still skips a route that names neither, with the same log line.
 A test proves an internal-only route is served, certified by the internal authority, and refused by
 the public one. That test fails against the code before the change.
 ## The first fix would have made the name public — 2026-10-02, from review
 Serving the dropped route was not enough. The proxy picks a route from the name a request carries
 and never from where the request came from, and it answers public and internal names on the same
 listeners. Its public names resolve to an address the internet reaches. So once the internal-only
 route was served, any request from the internet carrying `unifi.home-server.internal` — a name of a
 fixed, guessable shape — would have reached an administration interface that reach `internal` was
 chosen to keep private. Before the fix the route was unreachable from everywhere. After it, it would
 have been reachable from everywhere. Two more leaks came with it: the proxy's answer for an unrouted
 name listed every name it serves, internal ones included, and the handshake handed a certificate
 naming the internal host to any client.
 Nothing showed this while every routed endpoint also had a public name: its internal name exposed
 nothing the public one did not. It is a gap in the decision's wording, not only in the proxy — ADR
 0138 says the proxy *serves* the internal name without saying to whom — so it is recorded there as a
 progressive insight and in the to-be connectivity design.
 **Where "inside" is decided.** The mesh's guard recognises the private network by the interface a
 packet arrives on, never by source address, because a source can be claimed. The proxy cannot see the
 interface, so it reads the source: the mesh's range, loopback, and the ranges of the machine's own
 container bridges, which are the same interfaces the guard names. A claimed source does not carry
 here, as a connection needs its replies, and replies to those addresses leave by the tunnel or a
 local bridge. The range reaches the proxy from the catalog as the machine's `mesh-range`, the way the
 intrusion filter already receives it. With none given, the proxy serves internal names to the machine
 alone: refused, not opened.
 **What changed with it.** The internal name of a route that also has a public one is now served to
 the private network only, like any other internal name. Outsiders have the public name, so nothing
 they could reach is lost.
 **Order of release.** The catalog change comes first. A proxy built from the change but started
 without the range would serve internal names to its own machine alone, and every other member of the
 mesh would lose them until the range arrived.
@@ -0,0 +1,64 @@
 ---
 status: located
 opened: 2026-10-02
 located-in: [mesh-host internal/apply (removeOrphan: a former target of a kind with no removal was fatal), mesh-host internal/store (Record keeps a former target for every kind, the host's own archive included)]
 fixed-by:
 amended-design:
 ---
 # 194 — The host's own former archive stops every machine applying anything
 ## What was observed
 2026-10-02, 00:34Z, on all four machines of this mesh, the first time a host carrying former
 targets ([ADR 0163](../../02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md), rule 5,
 built in mesh-host 63) replaced itself with a newer host (mesh-host 64).
 The host delivers its own successor as an archive whose target is a versioned directory
 ([ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md)): every new version
 is the same resource with a new target. Since mesh-host 63 the record keeps a resource's former
 target so the next apply removes what the host wrote under it
 ([issue 097](../097-a-resource-that-changes-target-leaves-the-old-one-behind/00-report.md)). So
 the new host's first apply found the previous version's directory as a former target of its own
 archive, and asked the removal for an archive — which does not exist
 ([issue 162](../162-an-archive-cannot-be-undeclared/00-report.md)):
 ```
 applying "mesh-host.next@former:/usr/lib/nox-mesh-host/versions/3c906749ad27": no way to remove a "archive"
 0 resource(s) were applied and remain
 ```
 Orphans are removed before any resource is applied on a converged machine, so the refusal ended
 every apply at its first step. Every machine reported `failed`, applied nothing, and would have
 gone on doing so: a host fix is itself an archive the same apply would have to write, and the apply
 never reached it. The machines kept running what they had; nothing new from the mesh could land.
 ## Why it matters beyond this instance
 Two rules that are each right met in the one resource the host cannot afford to stop on. Rule 5
 says a former target is removed and said; issue 162 says an archive has no removal, deliberately,
 so an unassignment nothing can undo is never reported as done. Neither rule was wrong; their
 meeting was never tested, because the bed that would have found it is a host replacing itself
 under the new rule, and the first such replacement was the live one. The fix is narrow: a former
 target of a kind the host cannot remove is left in place, said, and forgotten — never fatal,
 because nobody dropped it. An archive the declaration dropped still refuses, as 162 has it.
 ## What it took to recover
 The broken host cannot apply its own fix: the fix is delivered as an archive, and the apply fails
 before writing anything. On each machine the host's record (`/var/lib/mesh-host/state.json`) had to
 lose the one `@former:` entry by hand, once, so that the next push could write the fixed archive and
 stand aside for it. A manual edit of the host's record is otherwise never done; it is written here
 because the alternative was four machines that could apply nothing.
 ## Open questions
 - Should `Record` keep a former target for a kind the host cannot remove at all? The trace is
  useful; the removal it implies is not. Keeping it and letting the apply forget it is what the fix
  does; not recording it would be quieter.
 - Should the host's own versions directory be cleaned by the launcher rather than by the apply —
  the one archive whose former targets are genuinely removable, by the thing that knows which one
  runs?
 - Is there a bed that replaces a host under the current rules before the live mesh does
  (the proof row of [ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md) was
  a single crossover, before former targets existed)?