mirror process memory increase persistently, after upgrade to 3.10.5

[ChenjiaxingXJTU]

there is a mirror process increase memory persistently. which reach 990MB memory in 4 days.

version

erlang 23.2 rabbitmq 3.10.5 amqp-client 5.9.0 ubuntu18.04 boinic
the cluster have two broker, with mirror queues policy, which is VM rather than container, Synchronising: batch size: 4096

cat /etc/rabbitmq/rabbitmq.conf

vm_memory_high_watermark.relative = 0.5

disk_free_limit.absolute = 1GB

cluster_partition_handling = autoheal

collect_statistics_interval = 30000

consumer_timeout = 900000

hipe_compile = true

log.file.level = info

background_gc_enabled = true

background_gc_target_interval = 30000

cat /etc/rabbitmq/rabbitmq-env.conf

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 1:1 +MMscs 256"

Below is the detailed information of <0,583,0>:

Runtime

OS PID: 8580
OS: Linux
Uptime (seconds): 298908
Is under maintenance?: false
RabbitMQ version: 3.10.5
Node name: rabbit@ip-10-110-80-92
Erlang configuration: Erlang/OTP 23 [erts-11.1.4] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [hipe]
Crypto library: OpenSSL 1.1.1 11 Sep 2018
Erlang processes: 650 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

• rabbitmq_shovel_management
• rabbitmq_shovel
• amqp10_client
• rabbitmq_management
• amqp_client
• rabbitmq_web_dispatch
• cowboy
• cowlib
• rabbitmq_management_agent

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@ip-10-110-80-92
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@ip-10-110-80-92/quorum/rabbit@ip-10-110-80-92

Config files

• /etc/rabbitmq/rabbitmq.conf

Log file(s)

• /var/log/rabbitmq/[email protected]
• /var/log/rabbitmq/rabbit@ip-10-110-80-92_upgrade.log

Alarms

(none)

Memory

Total memory used: 1.2662 gb
Calculation strategy: rss
Memory high watermark setting: 0.5 of available memory, computed to: 4.0713 gb

queue_slave_procs: 0.7705 gb (51.37 %)
binary: 0.5568 gb (37.13 %)
allocated_unused: 0.1028 gb (6.85 %)
code: 0.0319 gb (2.12 %)
other_proc: 0.015 gb (1.0 %)
other_system: 0.0129 gb (0.86 %)
other_ets: 0.0035 gb (0.23 %)
plugins: 0.0017 gb (0.11 %)
atom: 0.0014 gb (0.09 %)
mgmt_db: 0.0012 gb (0.08 %)
mnesia: 0.0004 gb (0.03 %)
connection_other: 0.0004 gb (0.03 %)
connection_channels: 0.0004 gb (0.03 %)
connection_readers: 0.0004 gb (0.02 %)
queue_procs: 0.0003 gb (0.02 %)
metrics: 0.0002 gb (0.01 %)
connection_writers: 0.0001 gb (0.0 %)
msg_index: 0.0 gb (0.0 %)
quorum_ets: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 36, limit: 32671
Sockets: 16, limit: 29401

Free Disk Space

Low free disk space watermark: 1.0 gb
Free disk space: 46.9857 gb

Totals

Connection count: 16
Queue count: 38
Virtual host count: 1

---

Home(H)|Network(N)|System(S)|Ets(E)/Mnesia(M)|App(A)|Doc(D)|Plugin(P)recon:proc_count(memory, 28) Interval:5000ms | 3Days 11:6:23 |
|Erlang/OTP 23 [erts-11.1.4] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [hipe] |
|System | Count/Limit | System | Status | Stat Info | Size |
|Proc Count | 652/1048576 | Version | 23 | Active Task | 1 |
|Port Count | 27/65536 | ps -o pcpu | 9.3% | Context Switch | 335213323 |
|Atom Count | 45999/5000000 | ps -o pmem | 16.7% | Reds(Total/SinceLast | 129112541505/2989011 |
|Mem Type | Size | Mem Type | Size | IO/GC:(5000ms) | Total/Increments |
|Total | 1.7555 GB | 100.0% | Binary | 531.9927 MB | 29.59% | IO Output | 34.3232 GB/744.4844 KB |
|Process | 1.1880 GB | 67.67% | Code | 30.3846 MB | 01.69% | IO Input | 53.1026 GB/1.0813 MB |
|Atom | 1.3307 MB | 00.07% | Port Parallelism (+spp) | false | Gc Count | 125854963/2200 |
|Ets | 5.0497 MB | 00.28% | RunQueue | 0 | Gc Words Reclaimed | 298502341034/2671027 |
|No | Pid | Memory |Name or Initial Call | Reductions| MsgQueue |Current Function |
|1 |<0.583.0> | 1.1608 GB |rabbit_prequeue:init/1 | 60889307774| 0 |gen_server2:process_next_msg/1 |
|2 |<0.690.0> | 1.7341 MB |rabbit_prequeue:init/1 | 11902933478| 0 |gen_server2:process_next_msg/1 |
|3 |<0.729.0> | 1.0692 MB |rabbit_prequeue:init/1 | 7293880242| 0 |gen_server2:process_next_msg/1 |
|4 |<0.679.0> | 675.2813 KB |rabbit_prequeue:init/1 | 220147710| 0 |gen_server2:process_next_msg/1 |
|5 |<0.718.0> | 675.1719 KB |rabbit_prequeue:init/1 | 437127871| 0 |gen_server2:process_next_msg/1 |

|Home(H)|Process Info(P)|Messages(M)|Dictionary(D)|Current Stack(C)|State(S)|Interval: 5000ms | 3Days 11:6:39 |
|Meta | Value | Memory Used | Value | Garbage Collection | Value |
|registered_name | <0.583.0> | msg_queue_len | 0 | min_bin_vheap_size | 45.3340 KB |
|initial_call | proc_lib:init_p/5 | heap_size | 606.2877 MB | min_heap_size | 233 B |
|group_leader | <0.222.0> | total_heap_size | 606.2974 MB | fullsweep_after | 63.9990 KB |
|status | waiting | trap_exit | true | minor_gcs | 0 |
|Links(2) | <0.582.0><0.584.0> |
|Monitors(2) | <0.9255.2><0.9206.2> |
|MonitoredBy(40) | <24925.11021.21><24925.11034.21><24925.11266.21><24925.29228.11><24925.3451.24><24925.11016.21><24925.10932.21>< |
|Reductions: NaN->NaN->NaN->NaN->NaN->60899567216->|
|Memorys: NaN->NaN->NaN->NaN->NaN->635752728->|
|q(quit) |

|Home(H)|Process Info(P)|Messages(M)|Dictionary(D)|Current Stack(C)|State(S)|Interval: 5000ms | 3Days 11:7:20 |
erlang:process_info(<0.583.0>, current_stacktrace).
|gen_server2:process_next_msg/1 | gen_server2.erl:673 |
|proc_lib:wake_up/3 | proc_lib.erl:236

2022-08-08 22:56:07.155508+08:00 [info] <0.489.0> Mirrored queue 'ipcmsg.event.queue' in vhost '/': Adding mirror on node 'rabbit@ip-10-110-80-92': <0.583.0>

rabbit@ip-10-110-80-92)1> lists:reverse(lists:keysort(2,[{P, erlang:process_info(P, heap_size)} || P <- erlang:processes()])).
[{<0.583.0>,{heap_size,15401305}},
{<0.729.0>,{heap_size,17731}},
{<0.690.0>,{heap_size,17731}},
{<0.15054.7>,{heap_size,10958}},
{<0.985.0>,{heap_size,10958}},
{<0.956.0>,{heap_size,10958}},
{<0.765.0>,{heap_size,10958}},
{<0.44.0>,{heap_size,10958}},
{<0.9206.2>,{heap_size,6772}},
{<0.964.0>,{heap_size,6772}},
{<0.963.0>,{heap_size,6772}},
{<0.962.0>,{heap_size,6772}},
{<0.957.0>,{heap_size,6772}},
{<0.952.0>,{heap_size,6772}},
{<0.820.0>,{heap_size,6772}},
{<0.785.0>,{heap_size,6772}},
{<0.718.0>,{heap_size,6772}},
{<0.711.0>,{heap_size,6772}},
{<0.625.0>,{heap_size,6772}},
{<0.599.0>,{heap_size,6772}},
{<0.550.0>,{heap_size,6772}},
{<0.184.0>,{heap_size,6772}},
{<0.50.0>,{heap_size,6772}},
{<0.9333.2>,{heap_size,4185}},
{<0.9320.2>,{heap_size,4185}},
{<0.9219.2>,{heap_size,...}},
{<0.965.0>,{...}},
{<0.958.0>,...},
{...}|...]

(rabbit@ip-10-110-80-92)1> erlang:memory().
[{total,1405402720},
{processes,791292008},
{processes_used,791288760},
{system,614110712},
{atom,1417545},
{atom_used,1395642},
{binary,562493472},
{code,31860547},
{ets,5432064}]

(rabbit@ip-10-110-80-92)2> erlang:process_info(pid(0,583,0)).
[{current_function,{gen_server2,process_next_msg,1}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{links,[<0.582.0>,<0.584.0>]},
{dictionary,[{{credit_to,<24925.12620.21>},70},
{{credit_to,<0.9206.2>},170},
{{credit_to,<24925.11001.21>},110},
{{credit_to,<24925.10447.21>},158},
{{credit_to,<24925.3438.24>},142},
{{credit_to,<24925.11006.21>},98},
{{credit_to,<24925.3505.24>},154},
{rand_seed,{#{jump => #Fun<rand.3.47293030>,max => 288230376151711743,
next => #Fun<rand.5.47293030>,type => exsplus},
[148631015441320539|211648746277119309]}},
{{credit_to,<24925.22501.26>},44},
{{credit_to,<24925.11011.21>},8},
{{credit_to,<24925.4588.0>},140},
{{credit_to,<24925.12672.21>},174},
{{credit_to,<24925.4057.0>},70},
{{credit_to,<24925.29228.11>},72},
{{#Ref<0.303754611.2132279297.238285>,fhc_handle},
{handle,{file_descriptor,prim_file,...},
#Ref<0.303754611.2132279297.238285>,650588,...}},
{credit_flow_default_credit,{400,200}},
{{credit_to,<24925.10514.21>},46},
{{credit_to,<0.9255.2>},96},
{{credit_to,...},106},
{{...},...},
{...}|...]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.222.0>},
{total_heap_size,95362469},
{heap_size,95360811},
{stack_size,10},
{reductions,61728543407},
{garbage_collection,[{max_heap_size,#{error_logger => true,kill => true,size => 0}},
{min_bin_vheap_size,46422},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,0}]},
{suspending,[]}]

[annieblomgren]

Using dead lettering with classic queue? Then this one might be related #5312

[michaelklishin]

Also keep in mind that Erlang 23 is now out of support and 3.10.8 will require 24.2+.

Classic queue mirroring has been deprecated for some time, consider using quorum queues, streams, or a non-mirrored classic queue.