An idle build machine's empty fetch is asked again, not read as the end (hq ADR 0190)
A fetch on a context without a deadline waits the client's own while and reports the deadline passed — the client's, not ours — and the loop read it as "stop": every idle build agent exited clean every half minute and was restarted by its supervisor, a crash loop with nothing in the log to say why. Only our own context ending ends the machine; an empty fetch, however it is reported, is asked again.
This commit is contained in:
@@ -193,9 +193,15 @@ func (m *natsMachine) Take(ctx context.Context, do func(context.Context, Build))
|
|||||||
// machine with nothing to build, and is asked again.
|
// machine with nothing to build, and is asked again.
|
||||||
fetched, err := sub.Fetch(1, nats.Context(ctx))
|
fetched, err := sub.Fetch(1, nats.Context(ctx))
|
||||||
switch {
|
switch {
|
||||||
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
|
case ctx.Err() != nil:
|
||||||
|
// Ours ended: the machine is being stopped.
|
||||||
return nil
|
return nil
|
||||||
case errors.Is(err, nats.ErrTimeout):
|
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded), errors.Is(err, nats.ErrTimeout):
|
||||||
|
// **An empty queue, not the end.** A fetch on a context without a deadline waits the
|
||||||
|
// client's own while and then says the deadline passed — the client's, not ours. Read
|
||||||
|
// as "stop", every idle build machine exited clean every half minute and was started
|
||||||
|
// again by its supervisor, which looked like a crash loop with nothing in the log to
|
||||||
|
// say why (2026-10-03, the first build agents). Asked again.
|
||||||
continue
|
continue
|
||||||
case err != nil:
|
case err != nil:
|
||||||
if sub.IsValid() {
|
if sub.IsValid() {
|
||||||
|
|||||||
Reference in New Issue
Block a user