Growing .wal file without using quorum queues

[Raikatzu]

Hello together! :)

First of all, the current versions I'm using:
RabbitMQ: 3.9.12
Erlang: 24.2

Problem description:
I've four exchanges in total, every exchange is a fanout type, one of those exchanges has a durable feature, the others don't. Also two of those exchanges only get messages in, but don't deliver them to any queue currently (one of those two exchanges not delivering to any queue is the one being durable). For a better understanding, that's it kind of visualized:
Exchange 1 | fanout | durable | message in
Exchange 2 | fanout | no feature | message in & message out
Exchange 3 | fanout | no feature | message in & message out
Exchange 4 | fanout | no feature | message in

Each of those exchanges sending out messages, use non-durable (transient) classic queues with random (server-)generated queue names & have the exclusive feature.

So I'm not having any quorum queues at all (at least from my point of view & according to the RabbitMQ management UI). Yet I noticed that inside the volume (I'm using Rabbit-MQ as a docker image via docker-compose) there is a .wal file which is continuously growing, also the path of that file is inside a directory which refers to quorum queues (the path is kind of like: ".../_data/mnesia/rabbit@HOSTNAME/quorum/rabbit@HOSTNAME/...)

As far as I've understood the documentation, .wal files are being used for quorum queues, but since all of the queues I'm currently using are classic queues, I wonder why I've a growing .wal file and how I would be able to avoid it, since it kind of fills up the disk of that environment.

Could that be a side-effect of me having messages being send to exchanges which aren't passed onto a queue (which I kind of doubt)? Or is there another thing I overlooked?

What I've also found out while typing this and rechecking things: whenever I start/restart RabbitMQ, I get some interesting log messages mentioning quorum queues and the start of a RA system inside the mentioned directory, log messages (grabbed for "quorum") look like this:

---

Feature flags: [x] quorum_queue
ra: starting system quorum_queues
starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@HOSTNAME/quorum/rabbit@HOSTNAME
ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
Feature flags: [x] quorum_queue
Running boot step rabbit_quorum_memory_manager defined by app rabbit
Recovering 0 queues of type rabbit_quorum_queue took 0ms

---

So I guess that even though I'm not even using quorum queues, the feature is being activated and somehow saves a good amount of data into a .wal file?

I would really be glad if anyone could help me out finding a solution for that problem.
Thanks in advance!

Best Regards

[lukebakken]

• Are you using the rabbitmq_mqtt plugin?
• Could you please share your entire RabbitMQ log file and configuration files?

[Raikatzu]

Based on the printing result of rabbitmq-plugins list I've got this entries mentioning mqtt plugins:

[  ] rabbitmq_mqtt      3.9.12
[  ] rabbitmq_web_mqtt  3.9.12

Therefore: no, I'm not using the rabbitmq_mqtt plugin

The configuration of RabbitMQ comes down to this:

rabbit-mq:
    image: rabbitmq:3-management-alpine
    hostname: rabbit-mq
    restart: always
    volumes:
      - rabbitmqdata:/var/lib/rabbitmq
    ports:
      - "5672:5672"
      - "15672:15672"

=> everything else is on default configuration

For the log-file: Don't get me wrong but even if most of the stuff is in the log file is some default loggings (due to using the default configuration) I still don't feel comfortable sharing the entire log file in public. Therefore I stripped some smaller things out & attached it.
Removed log-file since the "bug" (which wasn't a bug but a default behaviour of RabbitMQ) is fixed

Now that I'm continuously digging deeper, I've quite a big suspicion on these lines:

Feature flags: list of feature flags found:
<0.222.0> Feature flags:   [x] implicit_default_bindings
<0.222.0> Feature flags:   [x] maintenance_mode_status
<0.222.0> Feature flags:   [x] quorum_queue
<0.222.0> Feature flags:   [x] stream_queue
<0.222.0> Feature flags:   [x] user_limits
<0.222.0> Feature flags:   [x] virtual_host_metadata
<0.222.0> Feature flags: feature flag states written to disk: yes

especially this one Feature flags: feature flag states written to disk: yes

Best Regards

[kjnilsson]

Are you confident the WAL file is actually growing and not just large but static in size? By default the WAL is pre-allocated up to it's maximum size. You can try to disable pre-allocation by setting the ra wal_sync_method to sync using the advanced config option.

[lukebakken]

Nothing looks amiss. At this point I would like to ask the following:

• Provide a full listing (ls -la) of the .wal file that keeps growing. Take the listing on a regular interval so we know how fast it's growing.
• Provide the output of rabbitmqctl report. It would be best if you didn't alter the contents at all so if you don't wish to share it here send the file to me (my email is on my GH profile) or to [email protected]

[Raikatzu]

I think both of you are on the right track, with that hints I rechecked and seems like I investigated the situation a little bit wrong - the .wal file isn't actually growing while RabbitMQ is running, it is only changing its size when RabbitMQ is getting (re-)started (f.e. when a new deployment of the whole docker environment is being made), here are the results of my recheck:

result of du -s * | sort -n & ls -la 00000003.wal (note: last time restarted was yesterday before writing my last post, so the ls -la date fits)

8       meta.dets
8       names.dets
421008  00000003.wal

431105541 Mär  9 09:55 00000003.wal

restarted RabbitMq & executed du -s * | sort -n & ls -la 00000004.wal again (.wal file incremented by one number)

8       meta.dets
8       names.dets
514604  00000004.wal

526948287 Mär 10 15:53 00000004.wal

restarted RabbitMq & executed du -s * | sort -n & ls -la 00000005.wal again (.wal file incremented by one number)

8       meta.dets
8       names.dets
356136  00000005.wal

364678489 Mär 10 15:53 00000005.wal

So apart from it not growing while RabbitMQ is running, it seems like RabbitMQ is creating a big .wal file on every startup (or exit?) - maybe the pre-allocation @kjnilsson mentioned - until it hits its size limit (probably the default size of 512 MiB configured inraft.wal_max_size_bytes) and is being flushed to a segment file (where can I find them?) - that's what I understood so far

So I guess a first idea to get closer to a solution is to disable the pre-allocation via the configuration parameter mentioned - can I also access that parameter via the docker-compose environment variables or do I need to get into the RabbitMQ configuration itself to be able to change it?

Also: thanks for the help so far! 😄

[Raikatzu]

Sorry for not coming back earlier! I've tested deactivating the pre-allocation in the advanced config settings like mentioned and it works perfectly fine! The file is still existing, but has no relevance in size. Thank you guys a lot!

Yet I somehow can't "accept" the answer from @kjnilsson, probably because its a reply, therefore I'll accept the first (and theoretically "only" answer) in this thread. But anyways, thanks to both of you! 😄