4.0 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design | ||
|---|---|---|---|---|---|---|
| resolved | 2026-09-30 |
|
mesh-controller PR 173 (MaxAckPending 1 on the controller's events consumer); the packaging module's rebuild record and the status count remain open in the text below |
175 — An announcement queued behind a long build comes back, and the build runs again
What was observed
On the evening of 2026-09-30 five merges landed within minutes. The controller's log then showed the
same four announcements — two into this repository, one into the catalogue, one into the controller's
own — arriving again every couple of minutes, and each arrival of the controller's rebuilt the two
modules that package its source. The builder built builder and route-proxy five times over for one
merge, the control node was pushed after each, and every other message the controller handles waited
behind the builds. It looked like a slow mesh; it was a loop.
Earlier the same evening, at a lower rate, the log already carried duplicated lines — a merge seen twice, a module "moved" twice — that nobody read as a symptom.
Why
The controller acts on what it consumes in one loop, one message at a time, and a merge's handler builds every module the merge changed before it returns — minutes of work. The bus's acknowledgement window is thirty seconds. That contradiction was met once already: the message being worked on is kept alive by a heartbeat while its handler runs (issue 127's stretch, controller PR 122). The heartbeat covers one message. The consumer is a push consumer with no bound on what it may have outstanding, so the client is handed everything that is waiting at once; the messages queued behind the one being built time out unacknowledged, come back after thirty seconds, and are handled again when the loop gets to them — including the merge whose builds are already done, which builds them again. A module that only packages another repository's source has no record of which commit it was last rebuilt for, so nothing says "already done".
Why it matters beyond this instance
The design permits work to be done twice, silently, and the doubling scales with how busy the mesh is: the busier the builder, the longer the queue, the more that comes back. A push to a machine is idempotent and a rebuild produces the same digest, so nothing broke — but every merge cost several builds, the control node was pushed after each, and a person watching saw a mesh that would not settle. It was the redelivery storm of 2026-09-28 in a narrower form, one layer out.
What would have prevented it
- A consumer that is handled one at a time is delivered one at a time.
MaxAckPending: 1on the controller's events consumer: the server holds the rest, nothing times out behind a build, and the heartbeat that keeps one message alive is then keeping the message alive. - A packaging module records the commit it was last rebuilt for, so a replayed announcement is "already built from it", the answer the source-built modules already give.
- A log line that appears twice with the same commit is a symptom, and the check is cheap: the
same announcement acted on twice within its window is a count worth exposing in
status.
Resolved on the first remedy, 2026-09-30
MaxAckPending: 1 on the controller's events consumer (mesh-controller PR 173): the server hands the
controller one announcement at a time and holds the rest, so nothing times out behind a build. The
existing consumer is brought to that configuration by the assertion the controller makes at start.
The second and third remedies — a packaging module recording the commit it was last rebuilt for, and
a doubled announcement counted in status — are not built; they would make the same fault visible
and cheaper should the first ever be undone, and they are left here as what to reach for then.