Streams down after upgrade from 3.9.13 -> 3.10.5

[annieblomgren]

Hi,

We have a RabbitMQ single node instance that was upgraded from 3.9.13 to 3.10.5 and after the upgrade all Streams are down. The other classic queues are fine. There are not a lot of errors in the log related to streams, even with debug enabled. We tried downgrading RabbitMQ version back to 3.9.13, and later 3.9.14, 3.9.15 and 3.9.16 without the streams coming up.

We haven’t been able to reproduce it.

Relevant log lines while upgrading
2022-06-27 20:56:13.495092+00:00 [warn] <0.21574.434> rabbit_stream_coordinator: failed to stop member QUEUE_NAME 'rabbit@HOSTNAME-01' Error: {shutdown,{gen_server,call,[{osiris_server_sup,'rabbit@HOSTNAME-01'},{terminate_child,[103,108,XX,....]},infinity]}}

2022-06-27 20:56:13.499273+00:00 [warn] <0.16777.442> rabbit_stream_coordinator: failed to get tail of member QUEUE_NAME on rabbit@HOSTNAME-01 in 4 Error: {error,app_not_running}

2022-06-27 20:56:32.128787+00:00 [info] <0.1165.0> Recovering 1 queues of type rabbit_stream_queue took 0ms

2022-06-27 20:57:42.491319+00:00 [notice] <0.436.0> rabbit_stream_coordinator: candidate -> leader in term: 5 machine version: 2
2022-06-27 20:57:42.493180+00:00 [info] <0.436.0> Stream coordinator machine version changes from 1 to 2, applying incremental upgrade.
2022-06-27 20:57:42.493260+00:00 [info] <0.436.0> Stream coordinator machine version changes from 1 to 2, updating state.

Any input appreciated,
Annie

[kjnilsson]

It looks like the osiris application isn't running for some reason but I'd need to see the entire log really. Downgrades aren't supported and typically won't work whenever there are Ra machines like the stream coordinator involved - are the logs you are sharing from the 3.10 upgrade or one of the later downgrades?

[annieblomgren]

Thanks Karl!
Is there a way to check if osiris is running or not?

[johanrhodin]

The following osiris-named processes are running:

osiris_sup
osiris_retention
osiris_server_sup
osiris_replica_reader_sup
rabbit_osiris_metrics

[johanrhodin]

@kjnilsson any more info we can provide? We haven't been able to reproduce but can send anonymized definitions if that is of help.

I notice that in a functioning deployment there should also be osiris_writers running, which wasn't the case here.

[kjnilsson]

Hi, sorry a bit slow on this one. I've had a look at the logs and as you say there isnt really much information to explain this behaviour. We haven't seen it before. It looks like for the first attempt of the stream coordinator the osiris app wasn't running but it should retry 5s later normally. Without access to the system itself I am not sure I can do much more.

[annieblomgren]

We haven't seen it again and are still not able to reproduce it. If we see it again we can continue this and potentially give access to the system.