53 lines
3.0 KiB
Markdown
53 lines
3.0 KiB
Markdown
---
|
|
status: open
|
|
opened: 2026-10-01
|
|
located-in: [mesh-controller internal/link/serve.go (act handles one message at a time; sourceMoved waits for every build the merge asks)]
|
|
fixed-by:
|
|
amended-design: []
|
|
---
|
|
|
|
# 184 — A merge announcement blocks the controller's receive loop
|
|
|
|
## What was observed
|
|
|
|
The controller restarted at 12:53:32Z on 2026-10-01 and heard, among its first messages, a merge
|
|
announcement for the catalogue that touched some forty modules. Until 13:17:16Z — twenty-four
|
|
minutes — it took nothing else in: build outcomes that the build machine had announced and that
|
|
the console's `builds` log showed as done sat unrecorded, so `builds` listed none of them and the
|
|
modules stayed at their old versions; the node heartbeats were dropped by the bus as a slow consumer
|
|
on `mesh.control.*.alive`, twice. When the merge handler returned, everything queued arrived at once
|
|
and was taken in within a second.
|
|
|
|
## Why this is here
|
|
|
|
The receive loop acts on one message at a time, which is the right discipline for a store the
|
|
controller must write in order. Acting on a merge means asking builds and waiting for each outcome,
|
|
minutes of work, and that wait happens inside the loop that would otherwise be hearing the outcomes
|
|
of everything else. The bus keeps the merge message alive while the work runs — the fix for the
|
|
earlier repeated-merge fault — so nothing is lost and nothing is redelivered, and nothing is heard
|
|
either. The design permits the controller to go deaf for as long as a merge takes to build, and no
|
|
status says so: the mesh reads as quiet, builds read as missing, and heartbeats read as a slow
|
|
machine.
|
|
|
|
## Also seen, 2026-10-01 evening
|
|
|
|
The handler's work is lost when the controller is replaced while it runs. The runtime image's merge
|
|
at 14:22Z was taken by a controller that rolled forty seconds later, after asking the image's own
|
|
build and before asking the forty-two that stand on it. The announcement was redelivered to the new
|
|
controller, which judged it history — the source had been looked at after the merge — and said
|
|
"nothing the mesh holds reads it". The dependents were asked by hand. Whatever shape the fix takes,
|
|
the work a merge implies has to be recorded as asked, not held in the handler's stack.
|
|
|
|
## What a fix needs to decide
|
|
|
|
Whether a merge is work the loop dispatches and returns from — the builds asked, the outcomes taken
|
|
in by the same `Built` handler every other outcome uses, since the stream already delivers them —
|
|
or whether the loop runs more than one handler at a time with the store's ordering kept for the
|
|
kinds that need it. The first is smaller and keeps one ordering. Either way, a controller that is
|
|
busy should say so where `status` is read.
|
|
|
|
*How this would be checked:* a controller test where a merge announcement that asks a slow build
|
|
and a build outcome for another module arrive together, and the outcome is recorded before the
|
|
build finishes; live, the controller's log during the next catalogue merge shows registrations
|
|
interleaved with the merge's own.
|