mesh-issues: import hq's issue folders at their own numbers, and an issue with a folder is whole in the tracker (hq ADR 0275, to-be 53 slice 3)

2026-10-10 16:00:19 +02:00
parent 1fcf46fbfc
commit d05308a419
19 changed files with 2243 additions and 49 deletions
@@ -0,0 +1,119 @@
+---
+status: resolved
+opened: 2026-08-23
+located-in: [hal (the instance), mesh-host internal/bootstrap (preflight asks the daemon), mesh-controller internal/catalogue (capabilities)]
+fixed-by:
+  - "the instance only: PR #962 — incus hook. Merged, ran on one node in 6s of a 60s budget; pipeline #6832 green in 48s. The class remains open."
+  - "partly, on the mesh: mesh-host 73c010e, 9d8239a — preflight and the profile detectors ask the daemon, not the package (installed-but-broken reports absent)"
+  - "the class, on the mesh: a manifest declares capabilities, an action declares verify (ADR 0005), and a module's own assertions are the lab's verdict (design 01). The old arrangement is replaced module by module, so its count is never taken"
+amended-design: 03-DESIGN/01-to-be/01-end-to-end-testing.md
+---
+
+# 007 — An installed package is not an available capability
+
+## Symptom
+
+A module declares the virtualisation package the lab needs. The package is installed —
+version 7.3.0-1, recorded as explicitly installed. The client binary runs.
+
+The capability does not exist:
+
+| Checked | State |
+|---|---|
+| `incus.service` | disabled, inactive |
+| `incus.socket` | disabled, inactive |
+| `incus-user.socket` | disabled, inactive |
+| the operator's group membership | not a member of any incus group |
+| the client | reports **`Server version: unreachable`** |
+
+Nothing failed. Nothing reported anything. The declaration was satisfied exactly as written,
+and the thing it was declared for cannot be used.
+
+## Why this is not issue 001 again
+
+[Issue 001](../001-failed-package-install-reports-success/00-report.md) is *the install failed
+and the job reported success*. This is the opposite and arguably worse: **the install
+succeeded, and success was not the point.**
+
+A package is a set of files. A capability is a running service, an enabled socket, and an
+identity permitted to reach it. The module model declares the first and has no vocabulary for
+the second, so the gap between them is invisible — there is no state in which the mesh believes
+this node has virtualisation and is wrong, because the mesh was never asked to believe it.
+
+The distance between the two is the same one the delivery layer already has a name for:
+**transport versus effect.** A package install reports that files arrived, which is transport.
+
+## Why it matters now
+
+This is the first requirement of the lab
+([ADR 0016](../../02-DECISIONS/0016-the-lab.md)), which is
+phase 0 of the entire migration. The first capability the new work depends on is present,
+declared, and unusable — and would have stayed unusable silently.
+
+It also generalises. Every module that declares a package needing a unit enabled, a group
+joined, a kernel module loaded, or a socket activated has this gap. Post-install work lives in
+hooks, and hooks have their own recorded failure mode: thirteen were found that had never run.
+
+## Evidence
+
+- Package recorded as installed 2026-08-23 00:02, explicitly.
+- Both units disabled and inactive; the operator in no incus group; client reports the server
+  unreachable.
+
+## Open questions
+
+- Should a module be able to declare a **capability** — a unit that must be enabled, a group
+  the operator must be in — rather than only the package that provides it?
+- If that is what hooks are for, why is the gap invisible when a hook does not run? A hook that
+  never fires and a hook that fires and does nothing are indistinguishable today.
+- Is this what the verify stage should be asserting? It exists, and a module's own assertions
+  are meant to test outcomes rather than steps — "the socket accepts a connection" is exactly
+  that shape.
+- How many other declared packages are in this state? Nothing currently reports it, which means
+  the answer is unknown rather than zero.
+
+## The instance is fixed; the class is what this issue is now about
+
+*Updated 2026-08-23.* A hook now does the post-install work — group membership, subordinate id
+ranges, enabling both units, creating the storage pool, the bridge, and the default profile.
+Verified independently afterwards: the group exists with the operator in it, both id files
+carry the range, the service is active, and the storage pool reports `CREATED`.
+
+**So the mechanism was never missing.** Hooks are exactly the right place for this, and used
+properly they work. The gap is narrower and worse than "there is no way to do it":
+
+> The hook did six things. **Six checks were then performed by a human, by hand.** Nothing in
+> the pipeline asserted any of them, and a pipeline that dispatched a hook which silently never
+> fired would have been green in the same 48 seconds.
+
+That is not hypothetical — a hook named for a feature its module does not carry is skipped
+without complaint, and thirteen such hooks were found at once in the past. The distance between
+*the hook ran and did six things* and *the hook was dispatched* is invisible from the outside,
+and it is the whole of this issue.
+
+### The verification that was done by hand is the assertion set
+
+The six checks performed after the merge are, almost word for word, what the module's own
+verification should assert — outcomes, not steps, exactly as the lab design requires:
+
+| Asserted by hand | As a module assertion |
+|---|---|
+| operator in the admin group | the group exists and contains the operator |
+| subordinate id range in both files | both files carry the range |
+| both units enabled, service active | the service is active, and still active shortly after |
+| storage pool present | the pool exists and reports created |
+| bridge present with an address | the bridge exists and holds its address |
+| default profile wired to both | the profile references the bridge and the pool |
+
+They currently live in a chat message. Moved into the module, they would run on every delivery
+to every node, and the difference between a hook that worked and a hook that was merely
+dispatched would stop being something a person has to notice.
+
+### What this issue now asks
+
+- Does a module gain a way to declare a **capability** — the outcome — separately from the
+  package that provides it, or is "write assertions" the whole answer?
+- The verify stage exists and asserts almost nothing. Is this simply the first module that
+  should use it properly, making the issue a coverage problem rather than a design gap?
+- **How many other declared packages are in the state this one was in?** Still unknown, still
+  unreported by anything, and now demonstrably worth asking.
@@ -0,0 +1,53 @@
+---
+status: resolved
+opened: 2026-09-12
+resolved: 2026-09-12
+located-in: [mesh-controller (the verb existed)]
+fixed-by: nothing — the capability already existed and the wrong verb was used
+amended-design:
+---
+
+# 043 — A module cannot be given an account for the mesh's own queues
+
+**Withdrawn the day it was opened. The premise was wrong.** Kept rather than deleted, because the
+mistake is repeatable and the reason is worth reading.
+
+## What was claimed
+
+That a build machine could not be given access to the mesh's build queue, because a broker account
+is scoped to what a module `emits` and `consumes`, and the build queue is not a module event. The
+evidence was a builder that authenticated and was then refused:
+
+```
+ACCESS_REFUSED - User 'anchor-builder' doesn't have permissions to queue 'builds'
+```
+
+## Why it was wrong
+
+**The capability exists and is reachable from the command line.** There are two verbs, and they
+create different things:
+
+| Verb | Creates | Scoped to |
+|---|---|---|
+| `module issue <module> --node <n>` | a module's account | what that module emits and consumes |
+| `builder issue <name> [--node <n>]` | a build machine's account | the build queue, and the mesh exchange |
+
+The refusal was produced by using the first for a job the second exists to do. Running
+`builder issue` and pushing produced a builder that starts and takes work — no change to any
+manifest, any code, or the account mechanism.
+
+## The part worth keeping
+
+**The wrong verb succeeds, and says so.** `module issue` reported *"broker account created, scoped
+to what it emits and consumes"* for a module that emits and consumes nothing, producing an account
+that authenticates and can do nothing. The failure then appears one layer away, in the module's own
+log, as a permissions error against a queue — which reads like a missing capability rather than a
+misused command.
+
+That is a small, real sharp edge, and it is the whole of what this issue found. Whether it is worth
+anything — a refusal when a module with no events asks for an account, or a note in the builder
+module pointing at the verb that fits it — is a judgement, not a defect.
+
+**And a lesson that is not about the mesh:** the capability was three lines away in the same file as
+the code being read, under a name that says exactly what it does. The issue was written before
+looking for it.
@@ -0,0 +1,29 @@
+---
+status: resolved
+opened: 2026-09-22
+located-in: [mesh-catalog modules/redis (the provisioner's ACL)]
+fixed-by: mesh-catalog multiple-fixes (the consumer's ACL user loses the dangerous command category); proven by the grant end-to-end bed, which now asserts a write outside the consumer's keys and FLUSHALL are refused
+---
+
+# 080 — A cache grant lets the consumer flush the server
+
+## Symptom
+
+The cache provider's provisioner creates each consumer an ACL user confined to keys under its own
+login and allowed every command. A key pattern confines only commands that name keys. `FLUSHALL`,
+`FLUSHDB`, `CONFIG`, `SHUTDOWN` and the rest of the dangerous category name none, so a consumer
+granted "its own keys" could wipe every other consumer's, or stop the server.
+
+Found by carrying the large mesh bed's retired tenancy assertions into the grant end-to-end bed:
+`FLUSHALL` as the consumer answered `OK`.
+
+## Why it matters beyond the instance
+
+A grant is the mesh's promise that a consumer gets what it asked for and nothing else. The
+promise was checked on the key pattern and never on the command set, and the one bed that had
+asked was retired before it was run against the catalogue's module.
+
+## What would close it
+
+The ACL user is allowed the ordinary command set minus the dangerous category, and the grant bed
+asserts a write outside the consumer's keys and a `FLUSHALL` are both refused.
@@ -0,0 +1,58 @@
+---
+status: wontfix
+opened: 2026-09-29
+located-in:
+  - mesh-controller internal/catalogue/declaration.go (contributions are emitted for an assigned module, taken or not)
+fixed-by:
+---
+
+# 150 — A route is contributed before its module is taken
+
+## What was observed
+
+ace is adopted and runs `route-adapter` beside the predecessor's traefik (ADR 0104). The first web
+module migrated there was searxng. `assign ace searxng` + `push ace` — the step that is supposed to
+change nothing on an adopted machine (ADR 0100, hq 125: assign holds, take replaces) — took
+`searxng.zurag.be` down:
+
+```
+https://searxng.zurag.be/  →  502
+```
+
+for about five minutes, until the module was unassigned again.
+
+## Why
+
+Assign held everything it found on the machine — the predecessor's `searxng` container, its
+directories — exactly as designed. But the module's **route contribution** is not a resource on the
+machine, so nothing held it: it reached `route-adapter` at once, which wrote
+`mesh-searxng.zurag.be.yml` into traefik's file provider pointing at the mesh's assigned machine port
+(`http://ace.internal:20000`) — where nothing listened, because the container that would was held.
+traefik's file router for the name then won over the predecessor's docker-label router for the same
+name, and the name served a dead backend.
+
+On this occasion the window was lengthened by an unrelated failure (the machine's docker address
+pools were exhausted, so the module's network could not be created), but the fault does not depend on
+it: **between assign and take, every routed module's public name points at a backend the mesh has
+deliberately not started.** The runbook's §6 step 2 ("check the preparation — HAL's service still
+serves") is false for every routed module on a node running route-adapter.
+
+## What the operator did
+
+Rolled back per runbook (unassign before take), then re-ran as assign → push → wait for the node to
+report its holds → take → push, keeping the window to the ~10 s between the two pushes. A workaround,
+not a fix: a module whose data must move between assign and take (runbook §6 step 3) cannot shrink
+its window this way.
+
+## What would be right
+
+A contribution from a module that is assigned but **not taken** on an adopted node should be held like
+the module's resources are — withheld from the provider until take — or the provider should be told
+the contributor is held and leave the predecessor's route alone. Either keeps "assign changes nothing"
+true for routed modules.
+
+## Won't fix — 2026-10-08
+
+The arrangement it is about no longer exists: every machine is converged, no machine runs the
+predecessor's proxy or the `route-adapter` beside it, and the predecessor it was adopted from is
+retired, so there is no adopted machine on which an assigned but untaken module can contribute a route.
@@ -0,0 +1,44 @@
+---
+status: resolved
+opened: 2026-10-04
+located-in: [mesh-controller cmd/mesh-controller (the merge gate)]
+fixed-by: novox/mesh-controller PR #96 (bbd442c, the merge gate, ADR 0237)
+replay: R236
+amended-design:
+---
+
+# 236 — The catalogue check passes a manifest the host refuses
+
+## What was observed
+
+2026-10-04. A login-manager module passed `mesh-controller module check`, was registered, built and
+assigned. The first push of its declaration was refused whole by the host:
+
+```
+refused a declaration: this declaration is refused, and none of it was applied:
+  - resource "lemurs.service": a service that omits state leaves the unit's lifecycle to the
+    machine, and boot and takes-over are both its lifecycle
+```
+
+The rule is the host's declaration validation. The controller's check never applies it, so a manifest
+can pass every check the catalogue has and still fail on the first machine that receives it.
+
+## Why it matters beyond this instance
+
+A refusal is whole, so one module with this fault blocks every other change for that node until the
+module is fixed, rebuilt and pushed again. The fault is static: it is in the manifest, and could be
+found before a module is merged. Today it is found by assigning the module to a live machine.
+
+## Open questions
+
+1. Should the host's declaration validation be importable, so that `module check` (and registration)
+   runs it over each resource the manifest declares?
+2. Or should the controller validate the composed declaration before it sends it, and refuse to send
+   one the host would refuse?
+3. Which other host-side rules are not visible to the catalogue check today?
+
+## Resolved — 2026-10-08
+
+The merge gate (mesh-controller PR #96) composes each machine with the change and runs the node-engine's
+own declaration validation over it, failing the pull request with the machine and the module. R236 in
+mesh-lab's replays register holds it.
@@ -0,0 +1,64 @@
+---
+status: diagnosing
+opened: 2026-10-09
+located-in: []
+fixed-by:
+amended-design:
+---
+
+# 345. A calendar reminder shows as a transparent, empty window, and the mesh had no tool to look at it
+
+## Symptom
+
+On 2026-10-09 the operator reported that on the workstation, when an Outlook (Exchange) meeting is
+due, the mail client's reminder window appears, but it is transparent and shows no content: no title,
+no time, no Dismiss or Snooze. The mail client is Betterbird (a Thunderbird fork, installed from the
+AUR), with the Exchange calendar through the Owl add-on. The workstation runs i3 and the `picom`
+compositor (glx backend, vsync, fading). No module owned Betterbird.
+
+The operator's view of done: *on the workstation, when an Outlook meeting is due, the reminder window
+shows the meeting (title, time, Dismiss/Snooze) on a normal background, not a transparent empty window,
+and the mesh has tools to see and fix Betterbird.*
+
+## Reported history
+
+The operator reports (2026-10-09) earlier trouble with picom on the workstations, never recorded: a chat
+window (Telegram) froze, not redrawn, after a change of i3 workspace. `records_search` for picom finds
+nothing about it. It is the same kind of failure as this one, a window the compositor shows without its
+content, and it makes picom the first suspect here.
+
+## Why it is an issue
+
+Two gaps, and the symptom itself.
+
+1. **A desktop application the operator uses every day ran outside the mesh.** Nothing could say which
+   window the reminder is, what the compositor makes of it, or which preferences the mail client runs
+   with. A change to it could only be made by hand in its profile, which the mesh's rules forbid and
+   which nothing would keep.
+2. **The tools that look at a screen could not look at this window.** The window manager's list
+   (`node-display-session.windows`) leaves out every window it does not tile, the reminder among them,
+   and says nothing a compositor decides by (a window's depth, type, opacity). The screenshot tool of the
+   display server failed for one window with `X Error BadColor (invalid Colormap parameter)
+   X_QueryColors`, because it dumped the window by its id.
+3. **The symptom's cause is not known yet.** A reminder that shows nothing is a missed meeting.
+
+The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
+
+## Impact
+
+- The operator cannot read or dismiss a meeting reminder: the window is there but empty.
+- Until now, the next session that looked at it had to work around the mesh to see a window.
+
+## What is done
+
+- The display server's module gains a window list that includes unmanaged windows with their depth,
+  type, state, role and own opacity, and its screenshot of one window is cut from the screen as the
+  compositor shows it (catalogue pull request, linked from this record's pull request).
+- A `betterbird` module: status, named preferences without credentials or accounts, a user.js declared
+  through the module's setting, its log, restart, check, and a reminder on demand from a throw-away
+  profile, so a fix can be proved without touching the operator's calendar.
+
+- The `picom` module's backend and damage tracking become settings of each machine (defaults unchanged:
+  `glx`, `use-damage` true), so the candidate fixes can be set and proved without a hand edit.
+
+What is not done: the cause, and so the fix. See the diagnosis.
@@ -0,0 +1,79 @@
+---
+status: resolved
+opened: 2026-10-10
+located-in: [mesh-catalog modules/mesh-delivery (holder.go deliveryWalking), mesh-controller cmd/mesh-controller (upgrades.go SourceMoved, release_plan.go laterOnTheBranch)]
+fixed-by: mesh-catalog#171, mesh-controller#197
+replay: R362
+amended-design: 03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md
+---
+
+# 362. Two catalogue merges seconds apart left their delivery waiting for a person
+
+## Symptom
+
+Shortly after midnight on 2026-10-10 (local time) two pull requests of the catalogue were merged
+18 seconds apart: the claude-code change at 00:20:52, then the dunst change at 00:21:09. Both had passed
+their checks.
+
+`mesh-delivery.deliveries` then showed every delivery of the dunst pull request `superseded`, and the
+claude-code delivery `held`, waiting for "a person: merged, and the controller opened no walk for it within
+10m0s — nothing it holds follows that branch, or the merge was not heard: a person decides". The
+controller's `plans` said of the walk for the catalogue's head, which carried both merges: "published; its
+walk waits for mesh-delivery's word — `plans go … --why …` starts it by hand".
+
+Both reasons were wrong. A walk had been opened, and it contained the held merge. Nothing gave it the word.
+The operator's change sat undelivered for 58 minutes, until the operator's hand-act `mesh-controller.plans
+go` started the walk. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours
+before `mesh-delivery.stalled` lists it, and the walk's own wait raised only a warning, at 00:51, which said
+there was nothing to do and pointed at the ended delivery (the diagnosis's second entry; this sentence said
+before that the walk's wait raised nothing).
+
+## Why it is an issue
+
+A merge that passed its checks must reach its nodes without a person, and a delivery that cannot must say
+so loudly and truly. Here both failed: two ordinary merges close together, which happens whenever the
+operator merges a batch, stopped the mesh's delivery silently, and the reason it gave pointed away from the
+cause.
+
+## Where it lives
+
+The diagnosis ([01-diagnosis.md](01-diagnosis.md)) traces it to two parts that each behave as written and
+disagree on what a walk's commit means:
+
+- The controller plans a merge it hears after a later merge of the same branch at the later merge's commit,
+  and supersedes the earlier walk (the fix for
+  [issue 349](../349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md)).
+  So the claude-code merge got a walk at the dunst merge's commit.
+- mesh-delivery finds the delivery a walk walks by its walk id or by the commit it was merged as. The new
+  walk matched the dunst delivery, which its first walk's supersession had already ended, so no live
+  delivery held the walk and nothing gave it its turn. The claude-code delivery, merged as a different
+  commit, never got a walk and was held after ten minutes.
+
+## Open
+
+- How mesh-delivery learns which merges a walk carries: the controller's walk record naming every merge it
+  answers (its commit and the commits planned into it), or mesh-delivery reading containment itself, as it
+  already does to decide *covered*.
+- A superseded walk whose successor matches only final deliveries: whether the successor should be adopted
+  as a delivery of its own, so that something always holds a waiting walk.
+- A walk waiting for mesh-delivery's word past a bound, and a delivery held because no walk was opened,
+  raise a condition to the operator instead of waiting silently (failure must be loud).
+
+## Decided
+
+The operator decided the model that removes this shape, not only its bookkeeping:
+[ADR 0276](../../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md). Merges are collected in a rolling merge window and each batch is walked once, with at most one
+commit per repository, one walk at a time. The bookkeeping that tied a walk to an ended delivery is repaired as
+well, and every wait of this shape has a bound and a condition. It is designed in to-be 47, *The merge window
+and the batch*.
+
+## Resolved
+
+2026-10-10. mesh-delivery's side is mesh-catalog#171 (merged): a walk is tied to every live delivery whose
+merge it answers, never to one that has ended, and a waiting walk no live delivery holds is listed by `stalled`
+at once. The controller's side is mesh-controller#197: the merge window and the batch, one walk per batch with
+one commit per repository, the walk naming every merge it answers, and the bounds of decision 9. The replay
+R362 is registered in mesh-lab's replays register (mesh-lab#75), proved to fail on the commit before the
+mesh-delivery fix and pass on it. The build raised one rule the record had not settled, decided during the
+build and written into ADR 0276: a merge on the controller's own path never shares a batch with a catalogue
+merge.