4.0 KiB
status: resolved opened: 2026-10-01 located-in: [mesh-controller cmd/mesh-controller/upgrades.go (the merge handler builds inside the receive loop)] fixed-by: mesh-controller PR 197 (ADR 0162: the merge handler writes a plan, asks the first tier and returns; outcomes and a ticker advance it) and PR 199 amended-design: []
184 — A merge announcement blocks the controller's receive loop
What was observed
The controller restarted at 12:53:32Z on 2026-10-01 and heard, among its first messages, a merge
announcement for the catalogue that touched some forty modules. Until 13:17:16Z — twenty-four
minutes — it took nothing else in: build outcomes that the build machine had announced and that
the console's builds log showed as done sat unrecorded, so builds listed none of them and the
modules stayed at their old versions; the node heartbeats were dropped by the bus as a slow consumer
on mesh.control.*.alive, twice. When the merge handler returned, everything queued arrived at once
and was taken in within a second.
Why this is here
The receive loop acts on one message at a time, which is the right discipline for a store the controller must write in order. Acting on a merge means asking builds and waiting for each outcome, minutes of work, and that wait happens inside the loop that would otherwise be hearing the outcomes of everything else. The bus keeps the merge message alive while the work runs — the fix for the earlier repeated-merge fault — so nothing is lost and nothing is redelivered, and nothing is heard either. The design permits the controller to go deaf for as long as a merge takes to build, and no status says so: the mesh reads as quiet, builds read as missing, and heartbeats read as a slow machine.
Also seen, 2026-10-01 evening
The handler's work is lost when the controller is replaced while it runs. The runtime image's merge at 14:22Z was taken by a controller that rolled forty seconds later, after asking the image's own build and before asking the forty-two that stand on it. The announcement was redelivered to the new controller, which judged it history — the source had been looked at after the merge — and said "nothing the mesh holds reads it". The dependents were asked by hand. Whatever shape the fix takes, the work a merge implies has to be recorded as asked, not held in the handler's stack.
What a fix needs to decide
Whether a merge is work the loop dispatches and returns from — the builds asked, the outcomes taken
in by the same Built handler every other outcome uses, since the stream already delivers them —
or whether the loop runs more than one handler at a time with the store's ordering kept for the
kinds that need it. The first is smaller and keeps one ordering. Either way, a controller that is
busy should say so where status is read.
How this would be checked: a controller test where a merge announcement that asks a slow build and a build outcome for another module arrive together, and the outcome is recorded before the build finishes; live, the controller's log during the next catalogue merge shows registrations interleaved with the merge's own.
Decided, 2026-10-01
ADR 0162: a merge produces a tiered plan the store keeps; the handler asks the first tier and returns; outcomes advance the plan; a controller replaced mid-plan resumes it. The loop is never held by a build again.
Resolved, 2026-10-01 evening
Since the controller holding ADR 0162's plan rolled, a merge announcement is handled in
milliseconds: the plan is written, the first tier asked, the loop free. The builds the merge
implies are asked from the store's record, tier by tier, so a controller replaced mid-plan resumes
it rather than losing it. The first live merge under it (a controller change) is the proof the
decision's table asks for; its tiers are read with plans.
How it is checked: the plan tests in mesh-controller; live, plans after a merge and the loop's
log taking reports in while the plan builds.