Reset corrupted state when upgrading stream coord machine version

[gomoripeti]

One of our customers created a stream backed queue on an earlier version of RabbitMQ, then upgraded from 3.9.8 to 3.9.14. When starting the new version the stream queue appeared to be down and the below error is visible in the logs

2022-04-07 03:59:44.363267+00:00 [info] <0.436.0> Stream coordinator machine version changes from 0 to 2, no state changes required.
2022-04-07 03:59:44.363419+00:00 [erro] <0.436.0> handle_leader err {'EXIT',
2022-04-07 03:59:44.363419+00:00 [erro] <0.436.0> {function_clause,
2022-04-07 03:59:44.363419+00:00 [erro] <0.436.0> [{rabbit_stream_coordinator,'-eval_listeners/3-fun-3-',
2022-04-07 03:59:44.363419+00:00 [erro] <0.436.0> [<0.9250.310>,<0.8256.300>,
...

Additionally when attempting further stream queue actions (creating another queue or deleting the existing) the same function_clause crash happens.
I think the reason is that the listeners format changed between machine versions and the jump from 0 -> 2 was not handled (only 1 -> 2)

My first question is whether this is already fixed on master? (I think it is hence I opened a discussion not an issue)
My other question is, as the stream queues are only for testing, is there a way to reset the stream subsystem only (reseting the coordination ra system and deleting the stream queues?) without erasing the whole cluster?

(a full stacktrace to clarify the crash: )

> rabbit_queue_type:delete(Q, false, false, <<"cli_user">>).
** exception exit: {{'EXIT',{function_clause,[{rabbit_stream_coordinator,'-eval_listeners/3-fun-3-',
                                                                         [<0.14168.320>,<0.8256.300>,
                                                                          {#{<0.14168.320> => <0.8256.300>,<0.15281.331> => <0.8256.300>,
                                                                             <0.1222.307> => <0.8256.300>,<0.13774.307> => <0.8256.300>,
                                                                             <0.8333.362> => <0.8256.300>,<0.17980.291> => <0.8256.300>,
                                                                             <0.9250.310> => <0.8256.300>,<0.1550.307> => <0.8256.300>,
                                                                             <0.13713.307> => <0.8256.300>,<0.5000.307> => <0.8256.300>,
                                                                             <0.31907.343> => <0.8256.300>,<0.20353.364> => <0.8256.300>,
                                                                             <0.13956.307> => <0.8256.300>,<0.4896.307> => <0.8256.300>,
                                                                             <0.6318.334> => <0.8256.300>,<0.4780.307> => <0.8256.300>,...},
                                                                           [{aux,{stop,"<vhost>_test_stream_1647674413614225224",
                                                                                       #{epoch => 1,index => 75,machine_version => 2,
                                                                                         node => 'rabbit@<cluster>',
                                                                                         system_time => 1649303984361},
                                                                                       #{epoch => 1,
                                                                                         event_formatter =>
                                                                                             {rabbit_stream_queue,format_osiris_event,
                                                                                                                  [{resource,<<...>>,queue,...}]},
                                                                                         leader_locator_strategy => <<"client-local">>,
                                                                                         leader_node => 'rabbit@<cluster>,
                                                                                         name => "<vhost>_test_stream_1647674413614225224",
                                                                                         nodes => ['rabbit@<cluster>'],
                                                                                         reference =>
                                                                                             {resource,<<"<vhost>">>,queue,<<"test_stream">>},
                                                                                         replica_nodes => [],...}}}]}],
                                                                         [{file,"rabbit_stream_coordinator.erl"},{line,1408}]},
                                              {maps,fold_1,3,[{file,"maps.erl"},{line,410}]},
                                              {rabbit_stream_coordinator,eval_listeners,3,
                                                                         [{file,"rabbit_stream_coordinator.erl"},{line,1408}]},
                                              {rabbit_stream_coordinator,apply,3,
                                                                         [{file,"rabbit_stream_coordinator.erl"},{line,373}]},
                                              {ra_server,apply_with,2,
                                                         [{file,"src/ra_server.erl"},{line,2149}]},
                                              {ra_server,'-apply_to/5-lists^foldl/2-0-',3,
                                                         [{file,"src/ra_server.erl"},{line,2107}]},
                                              {ra_server,apply_to,5,
                                                         [{file,"src/ra_server.erl"},{line,2107}]},
                                              {ra_server,handle_leader,2,
                                                         [{file,"src/ra_server.erl"},{line,524}]}]}},
                    {gen_statem,call,
                                [{rabbit_stream_coordinator,'rabbit@<cluster>'},
                                 {leader_call,{command,normal,
                                                       {'$usr',{delete_stream,"<vhost>_<qname>",
                                                                              #{}},
                                                               await_consensus}}},
                                 30000]}}
     in function  gen:do_call/4 (gen.erl, line 243)
     in call from gen_statem:call_dirty/4 (gen_statem.erl, line 684)
     in call from ra_server_proc:gen_statem_safe_call/3 (src/ra_server_proc.erl, line 1400)
     in call from ra_server_proc:statem_call/3 (src/ra_server_proc.erl, line 234)
     in call from rabbit_stream_coordinator:process_command/2 (rabbit_stream_coordinator.erl, line 283)
     in call from rabbit_stream_coordinator:delete_stream/2 (rabbit_stream_coordinator.erl, line 125)
     in call from rabbit_stream_queue:delete/4 (rabbit_stream_queue.erl, line 142)

[acogoluegnes]

Yes, that's a bug, follow-up issue: #4510. Thanks for reporting this.

[gomoripeti]

Thanks for the very fast fix.

In general, is there a way to reset the stream subsystem only (reseting the coordination ra system) without erasing the whole cluster?
I'm not familiar with ra. I tried deleting the coordination directory on disk, but then there were other issues when starting the ra cluster.

[michaelklishin]

Every Ra system should have its own on disk data directory. There isn't much else to reset. We cannot comment on "other issues" without any specifics.