Keep an event alive while its bundle handles it, and hold one at a time (hq issue 387)
The consumer's ack wait is 30 seconds and the runtime gives a handler two minutes, so an event handled longer was offered again while it was handled; events waiting in the client's buffer ran out their ack wait too, so a burst reached max-deliver with nothing failed. Say in-progress every 10 seconds, as a worker's work does, and pull one event at a time.
This commit is contained in:
@@ -848,6 +848,12 @@ func ConsumerOf(node, module string) string {
|
||||
// (novox/hq issue 330).
|
||||
var NakDelay = 5 * time.Second
|
||||
|
||||
// EventHeartbeat is how often an event a bundle is still handling is said to be in progress, well inside the
|
||||
// consumer's ack wait. The client holds one event at a time (PullMaxMessages(1)): events are handed over in
|
||||
// order, one at a time, so an event waiting in the client's buffer would have its ack wait run out before it is
|
||||
// handed over, and a burst would end in max-deliver with nothing failed (novox/hq issue 387).
|
||||
var EventHeartbeat = 10 * time.Second
|
||||
|
||||
// ConsumeAs reads a module's durable consumer and hands each event to deliver (novox/hq ADR 0198):
|
||||
// acknowledged when deliver returns nil, negatively acknowledged after NakDelay when it returns an
|
||||
// error, terminated when it is not an event at all. The consumer is the controller's to create; this
|
||||
@@ -873,12 +879,31 @@ func (c *Conn) ConsumeAs(module string, deliver func(Envelope) error) (func(), e
|
||||
_ = msg.Term()
|
||||
return
|
||||
}
|
||||
if err := deliver(env); err != nil {
|
||||
// Kept alive while the bundle works on it, as a worker's work is (seat.go): the consumer's ack wait
|
||||
// (30 seconds) is for a runtime that died, not a handler that is slow — the runtime gives a handler
|
||||
// launch.EventTimeout. Without it an event handled longer than the ack wait was offered again while
|
||||
// it was handled, and counted toward max-deliver (novox/hq issue 387).
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
tick := time.NewTicker(EventHeartbeat)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
return
|
||||
case <-tick.C:
|
||||
_ = msg.InProgress()
|
||||
}
|
||||
}
|
||||
}()
|
||||
err := deliver(env)
|
||||
close(done)
|
||||
if err != nil {
|
||||
_ = msg.NakWithDelay(NakDelay)
|
||||
return
|
||||
}
|
||||
_ = msg.Ack()
|
||||
})
|
||||
}, jetstream.PullMaxMessages(1))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading %s's consumer %s: %w", module, durable, err)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user