Issue 083 opened (other control messages lost while the store restarts); 082's diagnosis carries its review

2026-09-22 13:34:05 +02:00
parent 9a75f2b6a9
commit f6ded3102d
2 changed files with 52 additions and 0 deletions
@@ -19,3 +19,16 @@ report marked "not now" and acknowledging one refused or recorded. The two-node
 it, runs the adoption push.

 **Located in:** the control plane's report consumer and its recorder. Not a decision.
+
+*On review.* An independent review stopped, killed and restarted a real store under the recorder
+and found two gaps. A store killed rather than stopped surfaces as a bare "unexpected EOF", which
+was not recognised as an outage, so the report was still lost; and a failed connection was always
+taken for an outage, so a wrong password would have been asked again every two seconds for ever,
+holding every enrolment and report behind it. The classification now checks the server's answer
+first — a wrong password, a cancelled statement, a dropped database are answers — and counts a
+network error, a connection that ended mid-conversation, a timeout, or anything the driver marks
+safe to retry as an outage. One report holds the queue at most two minutes, then is let go with a
+line saying it was lost; shutting down does not wait out the pause. What remains: a failed report
+retried after a partial write counts its failure twice (the recorder's writes are not one
+transaction), and the other control messages have the same loss
+([issue 083](../083-other-control-messages-are-lost-while-the-store-restarts/00-report.md)).
@@ -0,0 +1,39 @@
+---
+status: open
+opened: 2026-09-22
+located-in: [mesh-controller internal/link (serve: enrolment, build results, upgrade and catch-up announcements)]
+---
+
+# 083 — Other control messages are lost while the store restarts
+
+## Symptom
+
+[Issue 082](../082-a-report-that-arrives-while-the-store-restarts-is-lost/00-report.md) kept a
+node's report for another attempt while the store is restarting. The same control queue carries
+four other kinds of message, and each is still settled on a failed write as if it had been
+handled:
+
+- **An enrolment** is answered with a refusal when the store cannot be asked, so a valid token is
+  refused. Worse, if the store goes after the token is spent and before the node's key is kept,
+  the node holds no key and needs a new token — and trying the enrolment again is not safe,
+  because spending a token cannot be done twice.
+- **A build result** is rejected on any failure to record it, and nothing announces it again.
+- **An upgrade announcement** is acknowledged whatever happens, so a module that moved while the
+  store was down is never sent to the machines that run it.
+- **A catch-up announcement** is acknowledged before it is acted on, so it is lost until the
+  catalogue restarts.
+
+Found by review of 082, by reading the consumer. Not reproduced.
+
+## Why it matters beyond the instance
+
+A migration adopts the store while nodes enrol, builds run and modules move — the window 082
+found for reports, for every other message the control plane cannot afford to drop.
+
+## What would close it
+
+Each kind decided on its own: which can be handed back and asked again (build results and
+announcements, once recording them is shown to be safe to repeat), and which needs a different
+shape (enrolment, whose token must be spent once — spent and keyed in one write, or answered "not
+now" so the node asks again with the same token). A test for each, and a bed that restarts the
+store while a node enrols and a build lands.