Federation stops federating messages after prefetch count is hit

[eddboyer]

I've run into what looks like a problem with queue federation where messages are not being transferred after the consumer's prefetch count is reached and then cleared.

The federation links remain with a status of "running" but no messages are transferred from the upstream. This seems consistent with the expected behavior for queue federation where the link has been made but there are no consumers... in my case though there are consumers.

It seems like the federation halts if no consumers are available to consume messages (i.e. because the prefetch counts have been reached) but that it isn't continuing the federation once the consumers become available again.

Sequence of events (may require a consumer which processes events slowly):

1. Confirm federation links are working:
   1. Add 8 consumers on downstream queue, each with prefetch count of 1,000
   2. Add 100 messages to the downstream queue
   3. Add 100 messages to the upstream queue
   4. Confirm all messages are consumed from both queues
2. Replicate the issue:
   1. Add 10,000 messages to the downstream queue
   2. Prefetch count is reached on all consumers, Unacked messages shows as 8,000 on server
   3. Add 100 messages to the upstream queue
   4. Wait for ready message to drop to 0 (and hence unacked messages to drop below 8,000)
   5. Check the upstream queue: it still has all 100 messages
3. Get the federation working again:
   1. Restart one of the consumers
   2. Check the upstream queue: all messages now get federated and consumed

Here are some more details on my setup which may or may not be relevant:

• Upstream
  • RMQ version: 3.8.2
  • OS RHEL6
  • Queue type: Classic
• Downstream:
  • RMQ version: 3.9.16
  • OS RHEL7
  • Queue type: Quorum

I am still investigating this myself to try to understand what is happening better and what is and isn't relevant in the scenario. In the mean time I wanted to post this to ask...

• Is anyone aware of this as an issue?
• Am I doing something stupid or that isn't supported?
• Is anyone else facing a similar issue?

[lukebakken]

Thanks for the report. So we are certain to use the same setup as you, would you please export your definitions from both the upstream and downstream environments, and attach them to your response? That way we can use the exact same queue and federation settings.

Other questions:

• Are you running a cluster in each environment? How many nodes?
• How fast do your consumers ack messages? I'm assuming that in 2 above (replicate the issue), your consumers wait a bit before acking the 8,000 messages to give you time to add the 100 to the upstream.