RabbitMQ: rejoin of a cluster node fail

[sturla78]

Version
rabbitmq 3.9.19, windows server 2019 standard

Summary
I have create a cluster using rabbitmqctl.bat CLI tool, with 3 nodes. On the last node (vm-multi3) i have perform a reset in order to remove the node from the cluster, following the doc at https://www.rabbitmq.com/clustering.html#removing-nodes.
Then i have perform another time the join of node 3, but the join show the following errors:

Clustering node rabbit@vm-multi3 with rabbit@vm-multi1

16:14:10.388 [warning] Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `c:/Users/Administrator/AppData/Roaming/RabbitMQ/db/rabbit@vm-multi3-feature_flags`:

16:14:10.388 [warning] Feature flags:   - list of previously enabled feature flags now marked as such: [:maintenance_mode_status]

16:14:10.733 [error] Failed to create a tracked connection table for node :"rabbit@vm-multi3": {:node_not_running, :"rabbit@vm-multi3"}

16:14:10.733 [error] Failed to create a per-vhost tracked connection table for node :"rabbit@vm-multi3": {:node_not_running, :"rabbit@vm-multi3"}

16:14:10.733 [error] Failed to create a per-user tracked connection table for node :"rabbit@vm-multi3": {:node_not_running, :"rabbit@vm-multi3"}

This problem is not present if in the meantime, stop rabbit service, and clear manually the folder of the db of mnesia (this before execute the rejoin to the cluster)

At the end of procedure seems however the the join has been completed, because the status of the nodes show correct information, but i would understand if there is something of wrong on this steps or there is a possible problem.

Thanks for help.

[sturla78]

Add a note, with rabbitmq 3.8.9 version this problem is not present, i can execute reset and join_cluster without errors.

[michaelklishin]

If you reset and remove a node, it won't rejoin the cluster unless explicitly added. We need logs from all nodes and/or an explicit set of steps in order to tell what may be going on.

[sturla78]

Yes, a have explicitly added. I try to explain the steps in order to reproduce the problem, in all machine rabbitmq has been installed for the first time:

node1
(nothing to do)

node2
rabbitmqctl.bat stop_app
rabbitmqctl.bat join_cluster rabbit@node1
rabbitmqctl.bat start_app

node3
rabbitmqctl.bat stop_app
rabbitmqctl.bat join_cluster rabbit@node1
rabbitmqctl.bat start_app

up to here all is ok , i have the cluster up and running without problem.

Now i try to remove and rejoin for example node3:

Node3
rabbitmqctl.bat stop_app
rabbitmqctl.bat reset (at this point the node is correctly remove from cluster and other machines not show up anymore)
rabbitmqctl.bat join_cluster rabbit@node1

this join_cluster command fail with the errors described on first post. This happen with rabbitmq 3.9.19 but not with 3.8.9 version. In the log of rabbit of node3 there are exactly the same errors visible on CLI output.

[michaelklishin]

What's the end goal here? Why are some nodes instantly added but others have to be reset first (they were existing cluster members)? It's very uncommon for nodes to be removed so that they later rejoin, so it would be hard to justify spending time investigating.

There is a way to explicitly remove a non-reachable or reset node from the cluster. You don't need to use it most of the time but when removing nodes, it makes sense to do that.

With modern RabbitMQ you don't have to join nodes explicitly, as there are multiple peer discovery options.

[lukebakken]

I will test these steps out with 3.8.9 and the latest version of RabbitMQ. Like @michaelklishin, this is an odd series of steps to execute. My guess is that this issue does not happen if the last sequence of steps is as follows:

rabbitmqctl.bat stop_app
rabbitmqctl.bat reset (at this point the node is correctly remove from cluster and other machines not show up anymore)
rabbitmqctl.bat start_app
rabbitmqctl.bat stop_app
rabbitmqctl.bat join_cluster rabbit@node1

[sturla78]

Hello lukebakken,
i have try also this sequence with 3.9.19, but still fail, when you say last version you mean 3.10.6 ?

Thanks

[sturla78]

Thanks michaelklishin for the reply, the reason is that in my company we have a battery of tests that run and one of this test fail for this reason, so we have discover this strange behavior that previous version of rabbit do not have and for this reason i have write this post. In my case the node that i try to remove is reachable from other nodes, and the reset work correctly because on the other nodes a can see that the node has been correctly removed from cluster. For this reason, but also for understand because this happen on 3.9.19 and not in 3.8.9 i have ask help, only for to know if there is some steps that i have wrong or if there is a change of behavior from one to another version of rabbit.

Thanks again.