Refresh ack timeout

[TroyKomodo]

Hi, I am unsure if this is a thing, but...
If we have a consumed queue, and then we consume a task on the queue by default the ack timeout is 30min if the task is not ack'ed within that 30min window the connection is closed and then requeued. I know I can modify the global timeout on the RMQ instance but that seems like a strange way to fix this problem. Ideally there should be a way to refresh ownership of a task as to not timeout. This is likely the wrong repo I am making this issue on but I am also unsure where I should suggest a feature.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

As documented in the Consumers guide, you can increase the limit.

Delivery of a message does not imply ownership over it, it's a concept that does not exist in any of the protocols RabbitMQ supports. "Refreshing" will complicate a lot of things, including around monitoring, and most consumers do not need anywhere close to the default 30m timeout.

If an operation takes more than 30m, consumers performing it should have a progress metric of some kind, and acknowledge delivery reasonably early. RabbitMQ queues and streams are not meant to be a general purpose data store, too.

[lukebakken]

Related to #2990

For now you will have to adjust the global consumer timeout if you need more time to complete the task.

RabbitMQ can't release resources for a queued item until the consumer acks it. We had enough experience with support cases arising from abusive consumers that the timeout was implemented.

[TroyKomodo]

I think you are correct, there should be some progress metric that would make the most sense, although might require some restructuring. The problem came up with transcoding I used RabbitMQ as a queue to issue transcoding tasks for video. The transcoder picks up the task and then acks when the transcoding job is complete. Sometimes, if the videos are several hours the job would take longer than 30min. I have since changed the timeout to 12h but I thought maybe there should be a feature to resolve this.

[minusdavid]

We've had this issue come up with the open source Koha LMS project as well.

I'm inclined to acknowledge the message after the task has been processed as that seems the most robust method when using the message queue as a work/task queue. That's the current approach taken by Koha.

However, there are currently cases where some tasks take longer than 30 minutes to complete and we're getting these consumer timeouts. I think the optimal solution is to break up those tasks into smaller batches, but that would involve significant refactoring, so not a practical short-term solution.

In theory, the easiest answer is to change the global consumer timeout, but we distribute the Koha LMS code worldwide so we don't directly control people's RabbitMQ installations.

I think that leaves Michael's suggestion of acknowledging the message early, and then using separate progress metrics for managing the task. For users, I figure they can monitor the result store (which we update as the task is progressed). For the system, I'm thinking that we implement a cronjob that fails tasks that exceed a configurable Koha application specified timeout.

Does that sound like an optimal strategy to you, @lukebakken and @michaelklishin ?

As always, very much appreciate your assistance. Hope you guys had a good holiday season!

[maxisam]

We have this issue as well, I wonder if we can at least have a way to config the timeout value per queue?

[TroyKomodo]

I ended up just using nats.

[maxisam]

well, if you do that, mind as well just set a unlimited value to consumer_timeout.

[TroyKomodo]

well, if you do that, mind as well just set a unlimited value to consumer_timeout.

Nats as a re-ack feature which lets u refresh the timeout for long living tasks.

[maxisam]

actually, I just read the doc again and realized there is a way to set different timeout per queue https://www.rabbitmq.com/docs/consumers#per-queue-configuration

but re-act feature does sounds really nice

[mkuratczyk]

AMQP 0.9.1 has no such concept. However, we do have plans to make this possible with AMQP 1.0 and in a less explicit way for AMQP 0.9.1 as well perhaps: #11603