How to perform blue/green upgrade of cluster when using ttl on messages?

[jtim08515]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.5

Erlang version used

27.2.x

Operating system (distribution) used

Ubuntu

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

N/A

Logs from node 1 (with sensitive values edited out)

N/A

Logs from node 2 (if applicable, with sensitive values edited out)

No response

Logs from node 3 (if applicable, with sensitive values edited out)

No response

rabbitmq.conf

N/A

Steps to deploy RabbitMQ cluster

N/A

Steps to reproduce the behavior in question

N/A

advanced.config

No response

Application code

No response

Kubernetes deployment file

No response

What problem are you trying to solve?

We are currently using ttl on messages so that the message can be "delayed", for say, 24 days. Is there a way to perform a blue/green cluster upgrade so that the messages that contain a ttl inside of a queue don't reset/restart when they move to the new queue on the upgraded version of the rabbitmq cluster?

[michaelklishin]

I'm afraid there is no. Whether you use queue federation or your own code to move the messages, they will be re-published as new messages.

[michaelklishin]

The closest approximation I can think of is to use your own custom app that would use message header information to adjust the expiration header on the re-published messages. That said, this will complicate moving the apps over compared to queue federation, and you need to know the original publishing date and/or the delay to compute the offset.

You can also move over the apps that do not rely on message TTL, keep queue federation around for N days so that the expired messages on the Blue side are moved to the Green by queue federation.
Therefore avoiding moving messages in the "delay" queues but hooking up all others with to the new cluster with queue federation. Then expiration will happen in the original cluster and those dead-lettered messages will be moved over quickly.

[jtim08515]

Hey @michaelklishin ,

does rabbitmq not have a header that it can add automatically to indicate when the message got created/published to a queue? I did end up finding this and it has your name on it. Is this what i'm looking for? I couldn't find much useful information on this feature and how it can be used

rabbitmq-server/deps/rabbit/src/rabbit_message_interceptor.erl

Line 29 in 6515e9a

intercept(Msg0, set_header_timestamp, Overwrite) ->

[michaelklishin]

@jtim08515 AMQP 0-9-1 message properties are documented.

Like I said, any mechanism you might use — queue federation, a custom app, a shovel — will all re-publish messages as new ones.

For AMQP 1.0, supporting absolute-expiry-time as suggested by David below is something we can add.
Maybe for AMQP 0-9-1 too but it won't help with existing messages and won't ship before at least 4.1.0.

[ansd]

RabbitMQ should parse the absolute-expiry-time property, and prioritise it over the ttl header.

[jtim08515]

Where do i find documentation in rabbitmq about 'absolute-expiry-time' and is this added to every message by default?

[ansd]

I linked above to the AMQP spec. RabbitMQ docs do currently not contain this property. Client apps have to set this property. RabbitMQ currently doesn't parse this property. However, I'm suggesting that RabbitMQ should parse this property in future versions.

[michaelklishin]

Adding support for absolute expiration time would help with this. The original AMQP 0-9-1 spec only includes support for relative expiration time.