Consumers are deleted from queue

[dipaktank]

Some how consumers are deleted from queue. We are encountering following error why try to re-create connection.

WARN  [xxxxx][version:1.4.0.2][traceId:][spanId:][parentSpanId:][spanExportable:][null-consumerMonitor-1][2021-03-15 17:23:47,461]- org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.handleConsumeException:Line 739 - basicConsume failed, scheduling consumer for queue XXX.1549544931273_XXX for restart
org.springframework.amqp.AmqpIOException: java.io.IOException
	at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:113) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2116) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2062) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2042) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.core.RabbitAdmin.initialize(RabbitAdmin.java:604) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.core.RabbitAdmin.lambda$null$10(RabbitAdmin.java:532) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287) ~[spring-retry-1.2.5.RELEASE.jar!/:na]
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164) ~[spring-retry-1.2.5.RELEASE.jar!/:na]
	at org.springframework.amqp.rabbit.core.RabbitAdmin.lambda$afterPropertiesSet$11(RabbitAdmin.java:531) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.connection.CompositeConnectionListener.onCreate(CompositeConnectionListener.java:36) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:757) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:702) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:676) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:567) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$1600(CachingConnectionFactory.java:102) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:1439) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.consume(DirectMessageListenerContainer.java:702) [spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:681) [spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.restartConsumer(DirectMessageListenerContainer.java:523) [spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$startMonitor$2(DirectMessageListenerContainer.java:453) [spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
Caused by: java.io.IOException: null
	at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:147) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:968) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:46) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at sun.reflect.GeneratedMethodAccessor1423.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_102]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1184) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at com.sun.proxy.$Proxy246.queueDeclare(Unknown Source) ~[na:na]
	at org.springframework.amqp.rabbit.core.RabbitAdmin.declareQueues(RabbitAdmin.java:709) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.core.RabbitAdmin.lambda$initialize$12(RabbitAdmin.java:606) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.core.RabbitTemplate.invokeAction(RabbitTemplate.java:2151) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2110) ~[spring-rabbit-2.2.10.RELEASE.jar!/:2.2.10.RELEASE]
	... 26 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - failed to perform operation on queue 'xxx.1549544931273_xxx' in vhost '/' due to timeout, class-id=50, method-id=10)
	at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:502) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:293) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:141) ~[amqp-client-5.9.0.jar!/:5.9.0]
	... 37 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - failed to perform operation on queue 'xxx.1549544931273_xxx' in vhost '/' due to timeout, class-id=50, method-id=10)
	at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:517) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:341) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:182) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:114) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:739) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:47) ~[amqp-client-5.9.0.jar!/:5.9.0]
	at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:666) ~[amqp-client-5.9.0.jar!/:5.9.0]
	... 1 common frames omitted

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - failed to perform operation on queue 'xxx.1549544931273_xxx' in vhost '/' due to timeout

is the key line. See RabbitMQ logs for clues. We don't have enough information to suggest anything else. Please help others help you and provide as much information as possible about your environment, from RabbitMQ version to logs to what the topology looks like.

[dipaktank]

I am using RabbitMQ version 3.8.8 and Erlang 22.3.
Also we implemented clustering.

Following are some log which from the rabbitMQ server.

2021-03-16 12:28:34.012 [info] <0.16352.0> rabbit on node rabbit@MICROSVC down
2021-03-16 12:28:34.013 [warning] <0.16662.0> Management delegate query returned errors:
[{<26806.2728.0>,{exit,{nodedown,rabbit@MICROSVC},[]}}]
2021-03-16 12:28:34.047 [info] <0.16352.0> Node rabbit@MICROSVC is down, deleting its listeners
2021-03-16 12:28:34.052 [info] <0.16352.0> node rabbit@MICROSVC down: connection_closed
2021-03-16 12:28:35.056 [info] <0.16352.0> node rabbit@MICROSVC up
2021-03-16 12:28:35.058 [error] <0.16215.0> Mnesia(rabbit@MICROSVC): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@MICROSVC}
2021-03-16 12:28:35.058 [info] <0.16352.0> Autoheal request sent to rabbit@MICROSVC3DEU2
2021-03-16 12:28:35.060 [warning] <0.16352.0> Autoheal: we were selected to restart; winner is rabbit@MICROSVC
2021-03-16 12:28:35.061 [info] <0.17312.0> RabbitMQ is asked to stop...
2021-03-16 12:28:35.190 [info] <0.17312.0> Stopping RabbitMQ applications and their dependencies in the following order:
rabbitmq_federation_management
rabbitmq_management
rabbitmq_federation
amqp_client
rabbitmq_web_dispatch
cowboy
cowlib
rabbitmq_management_agent
rabbit
rabbitmq_prelaunch
rabbit_common
sysmon_handler
os_mon
ra
mnesia
2021-03-16 12:28:35.190 [info] <0.17312.0> Stopping application 'rabbitmq_federation_management'
2021-03-16 12:28:35.190 [info] <0.17312.0> Stopping application 'rabbitmq_management'
2021-03-16 12:28:35.190 [info] <0.44.0> Application rabbitmq_federation_management exited with reason: stopped