Federation missing messages when connection broken

[PeteAUK]

Hi All,

The company I work for has been merrily running RabbitMQ in production for a few months now. Helped on from here, we have two RabbitMQ brokers - one living on AWS and one living on-prem. They connect to each other via Federated exchanges and this is done over a VPN connection we establish to keep things nice and secure. It's been fantastic if I'm honest, lightning fast and improved a number of previously tightly coupled systems. We're currently looking at moving more over to message queues.

Over the past week we've had two incidents that have meant we've lost messages and I'm wondering if either my understanding is wrong or I've not configured something up correctly.

The first instance was when we were doing server maintenance and took the on-prem RabbitMQ server down for 12 hours (we were installing new hardware). None of the messages generated at the AWS end came through when the on-prem server came back up. New messages were fine though.

The second instance was the VPN threw a hissy fit for 14 hours the other evening - typically half an hour after we closed for the day! Exactly the same scenario, we had a load of messages that were created at AWS and nothing came through. When we restarted the VPN new messages came through as expected.

Both scenarios seem to be when the two brokers can't talk to each other messages aren't held onto. The exchange is a durable one but it's largely a fairly standard set up for things like the federation (no message TTL or expiry for example, on-confirm ack mode)

[michaelklishin]

See acknowledgement mode. Publisher confirms and consumer acknowledgements can both be used by federation.

It matters a great deal what is being federated since queue federation does not replicate. Exchange federation does and uses a separate queue for outgoing messages but that queue is not replicated
and I don't see any settings that would control the persistence of messages in it. So there are limits to what federation links can do in terms of data safety.

Shovel is more bullet-proof because it is drastically simpler and publisher confirms with consumer acknoweldgements is usually enough, plus you control the properties of the source queue.

[PeteAUK]

It may be me misunderstanding how federated exchanges work then. On the federation upstream settings I thought that with the ack-mode setting set to on-confirm if the upstream broker were offline (e.g. turned off or not-contactable) the message would be unacknowledged so the underlying federation queue would attempt to resend that message? The actual publisher on the website received an acknowledgement because it successfully published to the AWS exchange (we record all messages that are unacknowledged). Completely accept that if something broke and the AWS broker fell over afterwards we'd lose anything in a "pending" state. Basically it's the diagram that's shown under "Pair of federated exchanges" on https://www.rabbitmq.com/federated-exchanges.html

I discarded shovel as an option because I want both brokers to be able to consume the same message which I didn't believe shovel was able to do. For example when I generate an order message there is an on-prem consumer that processes the actual order and we have (or at least will have) an AWS consumer that sends out a confirmation email to the customer. Shovels also seemed to want to attach a queue to an exchange rather than exchange to exchange.