Issue 243: a rebuilt licence store silences every node

The manager's generation counter restarted with its store after issue 241, and nodes that wait for a higher number discarded every binding unseen.
2026-10-05 09:59:17 +02:00
parent 2a501af0eb
commit 127675e2da
2 changed files with 77 additions and 0 deletions
@@ -0,0 +1,44 @@
+---
+status: located
+opened: 2026-10-05
+located-in: [mesh-catalog modules/claude-code, mesh-catalog modules/claude-licence-manager]
+fixed-by:
+amended-design:
+---
+
+# 243 — A rebuilt licence store silences every node, and nothing says so
+
+## What was observed
+
+The licence manager's store lost its tables in the incident of
+[issue 241](../241-one-unreadable-grants-file-dropped-every-database-on-the-control-node/00-report.md). When
+they were made again, the manager adopted the remaining licence from a node's report, as
+[ADR 0206](../../02-DECISIONS/0206-a-node-reports-the-anthropic-grant-it-holds-and-the-licence-manager-adopts-a-licence-by-refreshing-it.md)
+intends. It bound every node again and rotated the licence on schedule. Its log showed nothing wrong.
+
+On the machines, the morning after:
+
+| machine | generation it last applied | generation the manager gave it | what it did |
+|---|---|---|---|
+| workstation A | 16 | lower ones, then 17 | ignored three bindings; applied the fourth |
+| home server | 11 | 10 | ignored it, kept a token issued before the loss |
+| workstation B | 12 | 9 | ignored it, kept a token issued before the loss |
+| control node | 12 | 12 | in step, by chance |
+
+- **A login waited three minutes.** The operator logged in to a second account on workstation A. The
+  manager adopted the login and moved the machine to it at once (ADR 0209). The machine went on
+  reporting its login as waiting. The manager adopted the same login again three more times, a vendor
+  refresh each time, about a minute apart. Only when the count passed 16 did the machine take it.
+- **Two machines would have lost the agent when their tokens expired.** Neither had taken the rotation
+  made since the loss. Each would have stopped working, with nothing anywhere saying why.
+
+## Why it matters
+
+The manager counted generations from one again, because its counter is a database sequence and was
+made again with the tables. A machine applies a binding only when its generation is greater than the
+one it applied last. So a store that is rebuilt, the very event a mesh must survive, silences every
+machine that applied a higher number. And the manager cannot see it: it publishes, the publish
+succeeds, and the machine discards the binding without a word.
+
+The generation guards against something that cannot happen. The bindings state keeps only the latest
+value per machine, so an older binding cannot arrive after a newer one.
@@ -0,0 +1,33 @@
+# 243 — Diagnosis
+
+## 2026-10-05
+
+**Trail.**
+
+1. The operator reported that a login on workstation A was "not going well".
+2. The machine's log showed it reporting a login as waiting, four times, while it still held the old
+   licence's binding at generation 16.
+3. The manager's log showed it adopting that login four times, and the store's tables missing from
+   02:09 until it adopted the remaining licence again at 02:22.
+4. The bindings the manager listed carried lower generations than the ones each machine reported:
+   10 against 11 on one machine, 9 against 12 on another.
+5. A refresh of the remaining licence, the manager's own verb, gave every binding a generation
+   above the old numbers. All three of its machines applied it within the second.
+
+**Located in two places.**
+
+- **The agent module, on each machine.** It applies a binding only when its generation is greater than
+  the one applied, a rule that assumes the counter never goes back. It now applies any binding that
+  differs from the one applied, in licence or generation. The state holds only the latest value per
+  machine, so nothing older can arrive. Checked by the module's tests: a lower generation after a
+  rebuild is applied, and an equal one is still skipped.
+- **The licence manager.** It numbers bindings from a database sequence, which starts again with a
+  rebuilt store. Before it considers the machines' reports, it now moves the sequence past every
+  generation a machine reports having applied, and never moves it back. Without this, a binding could
+  be given exactly the number a machine already applied, and that one would still be skipped. Checked
+  by the manager's tests, and the statement was tried on a database: a fresh sequence asked to pass
+  16 gives 17 next, and asking it to pass 5 afterwards leaves it where it was.
+
+**Ruled out.** The bus delivered every binding: each machine that took the fourth binding did so in the
+same second it was published. The licences themselves were sound. The remaining licence refreshed
+on every attempt, and the second account's login was adopted from its first report.