Redis Transport with Redis Streams (XREADGROUP)

[oliverhaas]

Since I learned about Redis Stream some time ago, and that they are supposed to be an improved message queue basically with built-in ack handling, I thought I give it a shot to implement a Kombu transport with redis streams.

Preliminary result here main...oliverhaas:kombu:feat/redis-streams-transport. I haven't really tested this in a real scenario, but test coverage is decent. This was mainly just to see what readblocks I might hit...

The main annoyance with Redis streams is that XREADGROUP has slightly different behavior than BRPOP when reading from multiple streams: The COUNT refers to the number of messages per stream, and not total like for BRPOP. This means it's not directly possible to have priorities emulated like in the BRPOP redis transport (and for multiple queues this also interferes with prefetching and so on...), and also it's not easy to get closer to the full AMQP feature set either.

Even though the streams felt really nice at first, by now I think improving on a list/sorted-set based Redis transport would probably offer a better result overall. Kinda disappointing that all these approaches have some downside in some regard.

The main takeway might be that redis streams look just plain superior to pub/sub in the fanout case, so maybe the way to go here is just the take the implementation of the fanout case

[oliverhaas]

I used Redis streams for the fanout Redis queue in my package here https://pypi.org/project/celery-redis-plus/. I decided against Redis streams for the normal queue for now due to the caveats regarding XREADGROUP and priorities mentioned above.