Issue 194: the host's own former archive stops every machine applying anything

Rule 5 of ADR 0163 (a former target is removed) met issue 162 (an archive has no removal) in the host's own archive, the first time a host carrying former targets replaced itself; every machine applied nothing from then on.
2026-10-02 02:42:19 +02:00
2 changed files with 64 additions and 62 deletions
@@ -0,0 +1,64 @@
 ---
 status: located
 opened: 2026-10-02
 located-in: [mesh-host internal/apply (removeOrphan: a former target of a kind with no removal was fatal), mesh-host internal/store (Record keeps a former target for every kind, the host's own archive included)]
 fixed-by:
 amended-design:
 ---
 # 194 — The host's own former archive stops every machine applying anything
 ## What was observed
 2026-10-02, 00:34Z, on all four machines of this mesh, the first time a host carrying former
 targets ([ADR 0163](../../02-DECISIONS/0163-taking-a-module-over-is-a-comparison.md), rule 5,
 built in mesh-host 63) replaced itself with a newer host (mesh-host 64).
 The host delivers its own successor as an archive whose target is a versioned directory
 ([ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md)): every new version
 is the same resource with a new target. Since mesh-host 63 the record keeps a resource's former
 target so the next apply removes what the host wrote under it
 ([issue 097](../097-a-resource-that-changes-target-leaves-the-old-one-behind/00-report.md)). So
 the new host's first apply found the previous version's directory as a former target of its own
 archive, and asked the removal for an archive — which does not exist
 ([issue 162](../162-an-archive-cannot-be-undeclared/00-report.md)):
 ```
 applying "mesh-host.next@former:/usr/lib/nox-mesh-host/versions/3c906749ad27": no way to remove a "archive"
 0 resource(s) were applied and remain
 ```
 Orphans are removed before any resource is applied on a converged machine, so the refusal ended
 every apply at its first step. Every machine reported `failed`, applied nothing, and would have
 gone on doing so: a host fix is itself an archive the same apply would have to write, and the apply
 never reached it. The machines kept running what they had; nothing new from the mesh could land.
 ## Why it matters beyond this instance
 Two rules that are each right met in the one resource the host cannot afford to stop on. Rule 5
 says a former target is removed and said; issue 162 says an archive has no removal, deliberately,
 so an unassignment nothing can undo is never reported as done. Neither rule was wrong; their
 meeting was never tested, because the bed that would have found it is a host replacing itself
 under the new rule, and the first such replacement was the live one. The fix is narrow: a former
 target of a kind the host cannot remove is left in place, said, and forgotten — never fatal,
 because nobody dropped it. An archive the declaration dropped still refuses, as 162 has it.
 ## What it took to recover
 The broken host cannot apply its own fix: the fix is delivered as an archive, and the apply fails
 before writing anything. On each machine the host's record (`/var/lib/mesh-host/state.json`) had to
 lose the one `@former:` entry by hand, once, so that the next push could write the fixed archive and
 stand aside for it. A manual edit of the host's record is otherwise never done; it is written here
 because the alternative was four machines that could apply nothing.
 ## Open questions
 - Should `Record` keep a former target for a kind the host cannot remove at all? The trace is
  useful; the removal it implies is not. Keeping it and letting the apply forget it is what the fix
  does; not recording it would be quieter.
 - Should the host's own versions directory be cleaned by the launcher rather than by the apply —
  the one archive whose former targets are genuinely removable, by the thing that knows which one
  runs?
 - Is there a bed that replaces a host under the current rules before the live mesh does
  (the proof row of [ADR 0141](../../02-DECISIONS/0141-the-host-delivers-its-own-successor.md) was
  a single crossover, before former targets existed)?
@@ -1,62 +0,0 @@
 ---
 status: open
 opened: 2026-10-02
 located-in: []
 fixed-by:
 amended-design:
 ---
 # 195 — Every assigned module is counted as a bus user without a credential, and the real gaps are lost in the count
 ## What was observed
 `status`, and `plan` for any machine, open with one line before anything else:
 ```
 the bus's user list leaves out 49 user(s) the mesh has minted no credential for: <node>.<module>, …
 Each is a user that cannot connect until one is issued
 ```
 The 49 are spread over four machines and name 26 distinct modules. Checked against the catalogue on
 2026-10-02:
 | what the module's definition says | modules |
 |---|---|
 | declares an own secret named `broker` | 1 — the route proxy, which needed a bus account for issue 191 |
 | declares no `broker` secret, and emits, consumes and serves nothing on the bus | 17 — the packet filter, the intrusion filter, the ssh daemon, the resolver configuration, the certificate authority, the broker itself and others |
 | declares no `broker` secret, and **emits events** | 1 |
 | not in this catalogue, so not checked | 7 |
 So the line counts every module assigned anywhere as a bus user. For almost all of them that is not a
 missing credential. A module with no `broker` secret has nowhere to receive one, and the mesh already
 says an account nothing reads is an orphan ([issue 078](../078-a-delivered-secret-is-accepted-under-any-name/00-report.md)).
 Two real gaps sit inside the count and cannot be told from the noise:
 - **A declared `broker` secret was filled with a value that is not an account.** Before its account was
  issued, the route proxy's plan on both machines already carried a sealed `broker` file, while the same
  status line said no credential had been minted for it. A push had made the declared secret the way it
  makes any own secret. The module would have started with a credential the bus does not know, and
  nothing would have said why. It was found only because the account was being issued by hand.
 - **A module that emits events declares no way to reach the bus.** Its events can go nowhere, and no
  check refuses that.
 ## Why it matters
 **A warning that is always on is read as never on.** The line names 49 users on every `status` and every
 `plan`. An operator, or an agent, learns to scroll past it. The one entry that was a real fault looked
 exactly like the 48 that were not.
 **The fault that was real is the silent kind.** A module whose broker credential is a generated value
 starts, fails to authenticate, and reports that three layers away from the cause. That is the failure
 the composition already refuses for a secret that was never made at all ("declared and not made"). Here
 a value was made, so the refusal never fired.
 ## Open questions
 - Should a bus user be composed for a module that declares no `broker` secret at all? If not, the line
  shrinks to the modules that can actually use an account.
 - Is a `broker` secret ever correctly made by the generic generator? If not, should composition refuse
  a declared `broker` until it is issued, or should the mesh issue it as part of placing the module?
 - Should a module that emits, consumes or serves on the bus be refused when it declares no `broker`
  secret?