Temporary shovel created when replacing nodes

[YorickH]

We have been running RabbitMQ 3.8.3 on Kubernetes through a Statefulset (moving to the Operator soonish) and are moving towards 3.10.x but we are noticing some strange behavior that is preventing us from moving forward.
Last night we did some updates on underlying k8s nodes so it recreated some of the RabbitMQ pods.

These type of pod renewals are causing messages that are in a dead-letter queue to be shoveled back to the original queue.

While reviewing the logs I noticed these logs for the dead-letter queues that had messages on them:

2022-07-12 20:21:47.141 [info] <0.9746.2786> shutting down shovel Move from worker-excel-exporter-failed, none remaining #{ack_mode => on_confirm,dest => #{current => {<0.4375.2785>,<0.11411.2786>,<<"amqp:///%2F">>},dest_queue => <<"worker-excel-exporter">>,fields_fun => #Fun<rabbit_shovel_parameters.11.9216819>,module => rabbit_amqp091_shovel,props_fun => #Fun<rabbit_shovel_parameters.12.9216819>,resource_decl => #Fun<rabbit_shovel_parameters.10.9216819>,unacked => #{},uris => ["amqp:///%2F"]},name => <<"Move from worker-excel-exporter-failed">>,reconnect_delay => 5,shovel_type => dynamic,source => #{current => {<0.4238.2786>,<0.9788.2786>,<<"amqp:///%2F">>},delete_after => 'queue-length',module => rabbit_amqp091_shovel,prefetch_count => 1000,queue => <<"worker-excel-exporter-failed">>,remaining => 77,remaining_unacked => 0,resource_decl => #Fun<rabbit_shovel_parameters.14.9216819>,source_exchange_key => <<>>,uris => ["amqp:///%2F"]}}

In this particular case, it's moving messages from worker-excel-exporter-failed back to worker-excel-exporter, which would cause some unexpected events to occur.

We have no shovels defined, and can't find any logs regarding the creation of these shovels.
We have a 7 node cluster with the following ha policy set for these type of queues:

ha-mode: exactly
ha-params: 5
ha-promote-on-failure: always
ha-sync-mode: automatic

Next to that, we have cluster_partition_handling set to pause_minority in case it would have any influence.

I'm thinking of just disabling the shovel plugin as we don't actively use it right now, but I would like to learn where this behavior is coming from so we could prevent it in the future in case we need the shovel again.

Thanks for taking a look, it would help us tremendously as I'm worried every time we touch the cluster something going to happen unexpectedly.

[michaelklishin]

A temporary Shovel is started when the "Move messages from queue A to queue B" API endpoint is used.
RabbitMQ itself does not start Shovels in any other scenario, unless told so via the configuration or an API call.