RabbitMQ 3.10.7 - Windows - disk check crashing

[duncanjread]

We have an issue with the disk checker crashing/restarting RabbitMQ every day, exactly 24 hours after the service was restarted.
We saw this in previous versions with errors about the PowerShell script timing out so updated to 3.10.7 with Erlang 24.3.3 to try and stop this happening, I believe the disk checking was changed to use Handle.
It still happens, though the error message is different.
It's running on a Windows Server 2016 core VM with 12GB RAM and 12 vCores. We have four servers in total, at two datacentres, each server is in a cluster of two. They all have the same behaviour. Is there anyway to disable the disk checking completely, we have a completely separate monitoring system and these are dedicated RabbitMQ servers with plenty of disk space.

2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> ** Generic server rabbit_disk_monitor terminating
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> ** Last message in was update
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> ** When Server state == {state,"c:/RabbitMQBase/db/rabbit@EDB-INT-RMQ-01-mnesia",
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> 50000000,17185275904,100,10000,
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> #Ref<0.4175755648.1574174723.107504>,false,
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> true,10,120000}
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> ** Reason for termination ==
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> ** {{timeout,{gen_server,call,[os_mon_sysinfo,get_disk_info]}},
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> [{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> {rabbit_disk_monitor,get_disk_free,2,
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> [{file,"rabbit_disk_monitor.erl"},{line,260}]},
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> {rabbit_disk_monitor,internal_update,1,
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> [{file,"rabbit_disk_monitor.erl"},{line,217}]},
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> {rabbit_disk_monitor,handle_info,2,
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> [{file,"rabbit_disk_monitor.erl"},{line,166}]},
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0>
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> crasher:
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> initial call: rabbit_disk_monitor:init/1
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> pid: <0.430.0>
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> registered_name: rabbit_disk_monitor
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> exception exit: {timeout,{gen_server,call,[os_mon_sysinfo,get_disk_info]}}
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> in function gen_server:call/2 (gen_server.erl, line 239)
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> in call from rabbit_disk_monitor:get_disk_free/2 (rabbit_disk_monitor.erl, line 260)
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> in call from rabbit_disk_monitor:internal_update/1 (rabbit_disk_monitor.erl, line 217)
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> in call from rabbit_disk_monitor:handle_info/2 (rabbit_disk_monitor.erl, line 166)
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> in call from gen_server:try_dispatch/4 (gen_server.erl, line 695)
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> in call from gen_server:handle_msg/6 (gen_server.erl, line 771)
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.218.0>]
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> message_queue_len: 0
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> messages: []
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> links: [<0.429.0>]
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> dictionary: []
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> trap_exit: false
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> status: running
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> heap_size: 6772
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> stack_size: 29
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> reductions: 1087159
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0> neighbours:
2022-09-13 10:47:30.869000+01:00 [error] <0.430.0>
2022-09-13 10:47:30.884000+01:00 [error] <0.429.0> supervisor: {local,rabbit_disk_monitor_sup}
2022-09-13 10:47:30.884000+01:00 [error] <0.429.0> errorContext: child_terminated
2022-09-13 10:47:30.884000+01:00 [error] <0.429.0> reason: {timeout,{gen_server,call,[os_mon_sysinfo,get_disk_info]}}
2022-09-13 10:47:30.884000+01:00 [error] <0.429.0> offender: [{pid,<0.430.0>},
2022-09-13 10:47:30.884000+01:00 [error] <0.429.0> {id,rabbit_disk_monitor},
2022-09-13 10:47:30.884000+01:00 [error] <0.429.0> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
2022-09-13 10:47:30.884000+01:00 [error] <0.429.0> {restart_type,{transient,1}},
2022-09-13 10:47:30.884000+01:00 [error] <0.429.0> {shutdown,300000},
2022-09-13 10:47:30.884000+01:00 [error] <0.429.0> {child_type,worker}]
2022-09-13 10:47:30.888000+01:00 [info] <0.27972.2> Enabling free disk space monitoring
2022-09-13 10:47:30.888000+01:00 [info] <0.27972.2> Disk free limit set to 50MB
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> ** Generic server rabbit_disk_monitor terminating
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> ** Last message in was update
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> ** When Server state == {state,"c:/RabbitMQBase/db/rabbit@EDB-INT-RMQ-01-mnesia",
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> 50000000,17166462976,100,10000,
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> #Ref<0.4175755648.1574174725.34453>,false,true,
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> 10,120000}
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> ** Reason for termination ==
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> ** {{timeout,{gen_server,call,[os_mon_sysinfo,get_disk_info]}},
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> [{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> {rabbit_disk_monitor,get_disk_free,2,
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> [{file,"rabbit_disk_monitor.erl"},{line,260}]},
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> {rabbit_disk_monitor,internal_update,1,
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> [{file,"rabbit_disk_monitor.erl"},{line,217}]},
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> {rabbit_disk_monitor,handle_info,2,
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> [{file,"rabbit_disk_monitor.erl"},{line,166}]},
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2022-09-13 10:47:51.025000+01:00 [error] <0.27972.2>
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> crasher:
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> initial call: rabbit_disk_monitor:init/1
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> pid: <0.27972.2>
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> registered_name: rabbit_disk_monitor
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> exception exit: {timeout,{gen_server,call,[os_mon_sysinfo,get_disk_info]}}
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> in function gen_server:call/2 (gen_server.erl, line 239)
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> in call from rabbit_disk_monitor:get_disk_free/2 (rabbit_disk_monitor.erl, line 260)
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> in call from rabbit_disk_monitor:internal_update/1 (rabbit_disk_monitor.erl, line 217)
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> in call from rabbit_disk_monitor:handle_info/2 (rabbit_disk_monitor.erl, line 166)
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> in call from gen_server:try_dispatch/4 (gen_server.erl, line 695)
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> in call from gen_server:handle_msg/6 (gen_server.erl, line 771)
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.218.0>]
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> message_queue_len: 0
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> messages: []
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> links: [<0.429.0>]
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> dictionary: []
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> trap_exit: false
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> status: running
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> heap_size: 6772
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> stack_size: 29
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> reductions: 20233
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2> neighbours:
2022-09-13 10:47:51.026000+01:00 [error] <0.27972.2>
2022-09-13 10:47:51.026000+01:00 [error] <0.429.0> supervisor: {local,rabbit_disk_monitor_sup}
2022-09-13 10:47:51.026000+01:00 [error] <0.429.0> errorContext: child_terminated
2022-09-13 10:47:51.026000+01:00 [error] <0.429.0> reason: {timeout,{gen_server,call,[os_mon_sysinfo,get_disk_info]}}
2022-09-13 10:47:51.026000+01:00 [error] <0.429.0> offender: [{pid,<0.27972.2>},
2022-09-13 10:47:51.026000+01:00 [error] <0.429.0> {id,rabbit_disk_monitor},
2022-09-13 10:47:51.026000+01:00 [error] <0.429.0> {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
2022-09-13 10:47:51.026000+01:00 [error] <0.429.0> {restart_type,{transient,1}},
2022-09-13 10:47:51.026000+01:00 [error] <0.429.0> {shutdown,300000},
2022-09-13 10:47:51.026000+01:00 [error] <0.429.0> {child_type,worker}]
2022-09-13 10:47:51.033000+01:00 [info] <0.27994.2> Enabling free disk space monitoring
2022-09-13 10:47:51.033000+01:00 [info] <0.27994.2> Disk free limit set to 50MB

[lukebakken]

Please continue discussion here - #5768