59 lines
4.7 KiB
Markdown
59 lines
4.7 KiB
Markdown
# Diagnosis — 2026-09-22
|
|
|
|
1. **Build results, upgrade announcements and catch-up requests can simply be asked again.**
|
|
Recording a build result ignores one already kept; acting on an upgrade pushes declarations,
|
|
which converge; replaying builds re-announces what is recorded. A catch-up request used to be
|
|
acknowledged before the work; it is acknowledged after.
|
|
2. **But asking again must not stop the queue.** Issue 082's first shape slept on a message and
|
|
handed it back to the head of the queue. The control plane takes one message at a time, so
|
|
while one report waited for the store, every enrolment waited behind it — and a host waiting
|
|
on its answer gave up after ten seconds, while its request stayed queued and would later spend
|
|
the token for keys nobody held. So a message the store cannot take is now **held**: kept
|
|
unacknowledged, set aside, and tried again on a timer, while the loop goes on to the next
|
|
message. It is held for at most two minutes, then let go with a line saying it was lost; a
|
|
control plane that stops returns everything it held to the broker, since none of it was
|
|
acknowledged. Each message is held under its subject — a node's report, a module's upgrade,
|
|
the catch-up, each build on its own — and a newer message for the same subject sets the held
|
|
one aside, so an old report recorded late can never overwrite what the node is doing now. An
|
|
upgrade is held only when reading the store failed, not when a push did, so a push that timed
|
|
out is not repeated every few seconds.
|
|
3. **An enrolment cannot simply be asked again.** It spent the token first and then wrote the
|
|
node's key and its other keys — in two databases, the token in the inventory and the key in the
|
|
identity store, so no one transaction could cover both. A store gone between the two left a
|
|
spent token and a node with no key, and the host, which generates new keys on every attempt,
|
|
could not recover without a person issuing a new token.
|
|
4. **So the token is claimed, not spent, until the enrolment is complete.** The presenting key
|
|
claims it for a short lease; every write that follows overwrites, so an attempt interrupted
|
|
part-way can be made again by the same presenter; the token is spent once everything is in the
|
|
store; and only then does the token's secret stop being the node's broker password — after the
|
|
spend, because a password replaced by an attempt that then failed would be held by nobody. A
|
|
second presenter is held off while the claim is live and may take it once the lease lapses. The
|
|
presenter that spent a token may claim it again, so an answer lost after the spend does not lock
|
|
out a machine the mesh holds as enrolled. A store that cannot be asked, or a token held by
|
|
another presenter, is answered "try again" rather than refused; the host asks again with the
|
|
same request — the same keys — for as long as the claim lasts, and then says to wait that long
|
|
before running enrol again.
|
|
|
|
**What is not closed.** An enrolment request sent to a control plane that is not running at all is
|
|
still queued; if the host gives up and the control plane starts later, the request completes for
|
|
keys nobody holds. That predates this issue and is not the store's window.
|
|
|
|
**How it is checked:** unit tests hold a report to being held while the store is away and recorded
|
|
when it is back, to being set aside by a newer report from the same node, and to being let go past
|
|
the bound; they hold build results, catch-ups and upgrades to being held on the store and settled
|
|
otherwise, and an upgrade to not being held on a push that timed out. Tests against a real store
|
|
hold the claim to one presenter, its lapse, the spend to the claimant, and a claim again by the
|
|
presenter that spent it; an enrolment met by a held token is answered "try again". A unit test holds
|
|
the host to asking again while told to and stopping when its patience runs out. The store-window
|
|
bed pushes a slow step to the control-node, stops the store while its report is on the way, and
|
|
has a second machine enrol into the gap: the report is held, the enrolment is answered "not now"
|
|
meanwhile, and when the store is back the report is recorded and the enrolment completes on its
|
|
own — with the mesh holding exactly the keys the machine generated — after which the machine applies
|
|
what it is pushed and is heard from. Reviewed twice independently; the first review's findings —
|
|
the queue stopped behind a retry, the password replaced too early, a lost answer after the spend —
|
|
are what points 2 and 4 describe.
|
|
|
|
**Located in:** the control plane's consumer and enrolment, the inventory's tokens, and the host's
|
|
enrolment. Not a decision: the token stays "useless once used" (ADR 0004); what changed is when it
|
|
is used.
|