The build machine takes one ask at a time, and says so while it builds #194

2026-10-01T14:16:32Z

mesh-admin commented

2026-10-01 14:16:32 +00:00

Why today's waves lost builds: the holder's worker consumer had the server default of many deliveries in flight. Every ask behind the one being built was delivered at once into the builder's channel, sat unacknowledged for the length of the build, was redelivered after the 60 s ack wait, and was dropped after the fifth delivery. Twenty-six of forty-three asks made in two minutes were never built, the queue read as empty, and the builder's log shows the survivors in redelivery order rather than ask order (hq issue 186).

HolderConsumerFor: MaxAckPending: 1 — one in flight, so a queue of asks is a queue. The controller updates the consumer at its next start.
The builder keeps a running build alive with InProgress every ten seconds (the controller's own stillWorking), so a build longer than the ack wait is neither redelivered nor counted out.

Test for the consumer; make check on the branch. Roll this before merging mesh-tools again (the runtime-image merge now rebuilds forty-two dependents through the queue).

Why today's waves lost builds: the holder's worker consumer had the server default of many deliveries in flight. Every ask behind the one being built was delivered at once into the builder's channel, sat unacknowledged for the length of the build, was redelivered after the 60 s ack wait, and was dropped after the fifth delivery. Twenty-six of forty-three asks made in two minutes were never built, the queue read as empty, and the builder's log shows the survivors in redelivery order rather than ask order (hq issue 186). - `HolderConsumerFor`: `MaxAckPending: 1` — one in flight, so a queue of asks is a queue. The controller updates the consumer at its next start. - The builder keeps a running build alive with `InProgress` every ten seconds (the controller's own `stillWorking`), so a build longer than the ack wait is neither redelivered nor counted out. Test for the consumer; `make check` on the branch. Roll this before merging mesh-tools again (the runtime-image merge now rebuilds forty-two dependents through the queue).

mesh-admin added 1 commit 2026-10-01 14:16:33 +00:00

The build machine takes one ask at a time, and says so while it builds 11b20b10ff

With the worker consumer's default of many deliveries in flight, every ask behind the one being
built was delivered at once, left unacknowledged for the length of the build, redelivered after the
ack wait and dropped after the fifth time: on 2026-10-01 twenty-six of forty-three builds asked in two
minutes were never built and the queue read as empty (hq issue 186). The holder's worker now has one
in flight, and a running build tells the bus it is still working, as the controller's long handlers
do, so a build longer than the ack wait is neither redelivered nor counted out.

mesh-admin merged commit 8d4e940866 into main

2026-10-01 14:19:18 +00:00

mesh-admin deleted branch fix/the-build-machine-takes-one-ask-at-a-time

2026-10-01 14:19:18 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#194