Temporary “Stream Not Available” error after restarting a stream or delete replica in RabbitMQ 4.1.0 cluster (previous versions also had this problem)

[yurahaid]

Describe the bug

We are experiencing a temporary unavailability of a stream after performing a restart operation in one of our RabbitMQ clusters. 

Environment:
• RabbitMQ version: 4.1.0
• Cluster configuration: 3 nodes
• Client: Go RabbitMQ Stream client (latest version (v1.5.6)

Issue Details:

In one of our RabbitMQ clusters, when we execute a restart_stream operation on a stream, ours producers and consumers receive a Stream Not Available error for approximately 5 to 20 seconds. This issue does not occur in another cluster with an identical configuration.

Both clusters are running RabbitMQ version 4.1.0 with three nodes each. The consumers are implemented using the latest version of the Go RabbitMQ Stream client.

Reproduction steps

1.	Set up a RabbitMQ cluster with 3 nodes running version 4.1.0.
2.	Create a stream and start a consumer using the Go RabbitMQ Stream client.
3.	Perform a restart_stream or delete_replica operations on the stream - example `rabbitmq-streams restart_stream --vhost "/" "test-rabbitmq-5.stream" --preferred-leader-node="{your_node}"` (it doesn't matter whether the node leader changes or stays the same)
4.	Observe that the consumer or producer receives a Stream Not Available error for 5 to 20 seconds.

{"severity":"ALERT","ts":"2025-05-27T12:03:18.897Z","caller":"/src/cmd/test-rabbitmq/main.go:92","message":"exit with error","context":{"error":"create consumer: Stream Not Available"},"stacktrace":"main.main\n\t/src/cmd/test-rabbitmq/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283"}

Expected behavior

The stream should remain available to consumers and publishers during and after the operations with stream, without any errors.

Additional context

• The issue is reproducible consistently in one cluster but not in another with the same configuration.
• No significant differences have been identified between the two clusters in terms of configuration or client implementation.

This makes it challenging to work with RabbitMQ because we cannot add or remove a node from the cluster without affecting the system's operation.

[acogoluegnes]

Can you provide the server logs for both clusters? Setting the log level to debug would be useful, even it can be verbose.

There must be something different between the 2 clusters, whether it is in the software stack or in the infrastructure, or somewhere else. Could you provide more information about the 2 clusters?

The stream should remain available to consumers and publishers during and after the operations with stream, without any errors.

There will be some unavailability at some point. Whether you notice or not, this is unavoidable. The duration of the unavailability depends on many factors. I suspect there is some unavailability or at least disruption on the cluster without allegedly any problem, as restarting a stream changes its topology, which impacts the connections that use it.

[michaelklishin]

@yurahaid you haven't provided any evidence of a bug.

A stream restart inevitably will result in a short unavailability window. Your clients should be ready to retry the operations.

It's up to you to investigate the difference between the clusters.

[yurahaid]

Yes, I understand that, and we are prepared for it.
However, the issue is that the stream often remains unavailable for an extended period—ranging from a few seconds to several minutes—which poses a problem for us.

This is particularly noticeable when compared to another cluster with an identical configuration, where the stream becomes available almost instantly after a restart.
We expect the stream to be accessible again within 10 seconds post-restart.

Interestingly, the cluster experiencing these issues is more powerful, equipped with 8 cores.
In contrast, I tested on a less powerful cluster (2 cores per node), subjected it to a significant CPU load—making it more overloaded than the problematic one—and still observed that stream restarts were quicker and more stable, with availability restored almost immediately upon the first reconnection attempt.

[yurahaid]

I am adding logs from the clusters.

Both clusters are very similar, hosted on VMs in GCP in different projects.
CPU type n4
Disk: Hyperdisk

The problem cluster has 8 CPUs, and stable has 2 CPUs in each node.

healthy cluster with successful reconnect on the first attempt..log
problem_cluster_3_failed_reconects.log

[yurahaid]

On the problem cluster:
rabbitmq-streams coordinator_status returns only Status of stream coordinator ... without any additional data
rabbitmqctl eval "rabbit_stream_coordinator:status()." - has always had an empty output

Always, on each node, before/after the restart, the response is always empty.

On the cluster that runs normally in response to the command coordinator_status, I got a table with the nodes' states.
I see that the cluster coordinator is working correctly.

But, unfortunately, I still don't understand why we have a problem in one of the clusters, because their configuration looks the same

[acogoluegnes]

Did you try to restart the first node on the problematic cluster?

[yurahaid]

@acogoluegnes We will try to do it, but later.
This is a production cluster, and because of this problem, restarting the node will result in some of the consumers being down for several tens of minutes, and we can't allow that in production.

[kjnilsson]

@yurahaid any update one this one?

[yurahaid]

No, there are no updates yet.
As I mentioned, we can't simply restart the rabbitmq node due to this issue, as it would cause uncontrolled downtime in the production environment, resulting in the unavailability of streams, which is critical for us.

As soon as the team makes the necessary preparations to prevent this situation from becoming critical, we will revisit this issue. However, I cannot provide a specific deadline at this time. I expect it to be in the next few weeks

[michaelklishin]

@yurahaid has stopped responding => we cannot guess as to what may be going on.