RabbitMQ does not validate configuration settings when run from release archive

[lukebakken]

RabbitMQ version used

4.1.0

Erlang version used

27.3.x

Operating system (distribution) used

Arch Linux (via WSL)

How is RabbitMQ deployed?

Community Docker image

https://github.com/lukebakken/rabbitmq/tree/main/src/discussions/1

Hello Team RabbitMQ!

I was helping a community user with LDAP, and noticed that he could start RabbitMQ (via docker) with auth_ldap settings in his rabbitmq.conf file, even though rabbitmq_auth_backend_ldap was NOT enabled. However, when I start RabbitMQ from source, and use the same rabbitmq.conf file, the validation works as expected, and RabbitMQ fails to boot.

Original issue reported via Discord:
https://discord.com/channels/1092487794984755311/1092487794984755314/1362805174208303405

User followed-up here:
lukebakken/rabbitmq#1

Notice that, because rabbitmq_auth_backend_ldap was not enabled, this user got the following stack trace when trying to authenticate a user (noproc):

[debug] <0.1021.0> User 'mqadmin-lab' failed authentication by backend rabbit_auth_backend_internal
[info] <0.1021.0> LDAP CHECK: login for mqadmin-lab
[info] <0.1021.0>     LDAP connecting to servers: ["ldap.ad.example.com"]
[debug] <0.1021.0> User 'mqadmin-lab' authentication failed with exit:{noproc,
[debug] <0.1021.0>                                                          {gen_server2,call,
[debug] <0.1021.0>                                                           [ldap_pool,
[debug] <0.1021.0>                                                            {next_free,
[debug] <0.1021.0>                                                             <0.1021.0>},
[debug] <0.1021.0>                                                            infinity]}}:

Any ideas? I'm going to continue investigating.

[lukebakken]

The plot thickens ... I added the following to my Makefile, which starts RabbitMQ using one of the generic-unix archives~

https://github.com/lukebakken/rabbitmq/blob/main/src/discussions/1/Makefile#L12-L18

Usage:

# to use latest-toolchain
make start-release

# to use another release
make RABBITMQ_ARCHIVE=rabbitmq-server-generic-unix-4.1.0.tar.xz start-release

Both of the above start successfully, even though they should have failed due to the auth_ldap settings in the configuration files:

  ##  ##      RabbitMQ 4.1.0
  ##  ##
  ##########  Copyright (c) 2007-2025 Broadcom Inc and/or its subsidiaries
  ######  ##
  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  Erlang:      27.3.3 [jit]
  TLS Library: OpenSSL - OpenSSL 3.5.0 8 Apr 2025
  Release series support status: see https://www.rabbitmq.com/release-information

  Doc guides:  https://www.rabbitmq.com/docs
  Support:     https://www.rabbitmq.com/docs/contact
  Tutorials:   https://www.rabbitmq.com/tutorials
  Monitoring:  https://www.rabbitmq.com/docs/monitoring
  Upgrading:   https://www.rabbitmq.com/docs/upgrade

  Logs: /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/var/log/rabbitmq/[email protected]
        <stdout>

  Config file(s): /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/conf.d/10-defaults.conf
                  /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/conf.d/20-rabbitmq.conf

  Starting broker...2025-04-23 16:25:22.315804-07:00 [info] <0.216.0>
2025-04-23 16:25:22.315804-07:00 [info] <0.216.0>  node           : rabbit@SEA-3LG5HVJUWJK
2025-04-23 16:25:22.315804-07:00 [info] <0.216.0>  home dir       : /home/lbakken
2025-04-23 16:25:22.315804-07:00 [info] <0.216.0>  config file(s) : /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/conf.d/10-defaults.conf
2025-04-23 16:25:22.315804-07:00 [info] <0.216.0>                 : /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/conf.d/20-rabbitmq.conf
...
...
...
2025-04-23 16:25:22.785803-07:00 [debug] <0.604.0> == Plugins (postlaunch phase) ==
2025-04-23 16:25:22.789064-07:00 [debug] <0.604.0> Starting the following plugins: []

Note that no plugins were started, either 🤔

[lukebakken]

Starting up the Windows .zip archive using invalid auth_ldap settings "works" as well - RabbitMQ boots fine. I'm going to try the official deb package next.

[lukebakken]

I followed the Debian Bookworm instructions here and can start RabbitMQ using the invalid configuration files that have auth_ldap in them.

[michaelklishin]

@lukebakken by "invalid" do you mean a syntactically invalid Erlang term file?

[lukebakken]

I think the title of this discussion could be improved 😸

I mean that the rabbitmq.conf file has an auth_ldap section, which should require the rabbitmq_auth_ldap_plugin to be enabled for RabbitMQ to boot.

What I'm observing is that RabbitMQ boots just fine when auth_ldap is present in rabbitmq.conf, but rabbitmq_auth_ldap_plugin is NOT enabled. I observe this behavior when running RabbitMQ from various release assets (as well as the community docker image, which uses the latest-toolchain release asset).

I observe the correct behavior (RabbitMQ fails to boot) only when running RabbitMQ from main.

I'm doing all testing with RabbitMQ 4.1.0.

[michaelklishin]

Understood. Yup, that sounds like a prelaunch-level bug. Thank you for looking into it.

[lukebakken]

@michaelklishin - more information about this issue.

When you start RabbitMQ from main and enable LOG=debug, this is output. Note that only the .schema files for enabled plugins are in this output:

2025-04-25 10:31:37.215501-07:00 [debug] <0.162.0> Configuration schemas found:
2025-04-25 10:31:37.215501-07:00 [debug] <0.162.0>
2025-04-25 10:31:37.215517-07:00 [debug] <0.162.0>   - /home/lbakken/development/rabbitmq/rabbitmq-server/plugins/credentials_obfuscation-3.5.0/priv/schema/credentials_obfuscation.schema
2025-04-25 10:31:37.215545-07:00 [debug] <0.162.0>   - /home/lbakken/development/rabbitmq/rabbitmq-server/plugins/rabbit-4.1.0+beta.4.211.g1806a45/priv/schema/rabbit.schema
2025-04-25 10:31:37.215572-07:00 [debug] <0.162.0>   - /home/lbakken/development/rabbitmq/rabbitmq-server/plugins/rabbitmq_management-4.1.0+beta.4.197.gd265fdd/priv/schema/rabbitmq_management.schema
2025-04-25 10:31:37.215587-07:00 [debug] <0.162.0>   - /home/lbakken/development/rabbitmq/rabbitmq-server/plugins/rabbitmq_management_agent-4.1.0+beta.4.197.gd265fdd/priv/schema/rabbitmq_management_agent.schema
2025-04-25 10:31:37.215605-07:00 [debug] <0.162.0>   - /home/lbakken/development/rabbitmq/rabbitmq-server/plugins/rabbitmq_web_dispatch-4.1.0+beta.4.197.gd265fdd/priv/schema/rabbitmq_web_dispatch.schema
2025-04-25 10:31:37.275655-07:00 [debug] <0.162.0> Loading configuration files (Cuttlefish based):
2025-04-25 10:31:37.275729-07:00 [debug] <0.162.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/conf.d/10-defaults.conf
2025-04-25 10:31:37.275767-07:00 [debug] <0.162.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/conf.d/20-rabbitmq.conf
2025-04-25 10:31:37.295521-07:00 [debug] <0.162.0> Adding Defaults
2025-04-25 10:31:37.295851-07:00 [debug] <0.162.0> Right Hand Side Substitutions
2025-04-25 10:31:37.295913-07:00 [debug] <0.162.0> Applying Datatypes
2025-04-25 10:31:37.303433-07:00 [error] <0.162.0> You've tried to set auth_ldap.servers.1, but there is no setting with that name.
2025-04-25 10:31:37.303502-07:00 [error] <0.162.0>   Did you mean one of these?
2025-04-25 10:31:37.328840-07:00 [error] <0.162.0>     auth_backends.$num
2025-04-25 10:31:37.328902-07:00 [error] <0.162.0>     auth_mechanisms.$name
2025-04-25 10:31:37.328968-07:00 [error] <0.162.0>     log.ra.level

You can see where boot stops due to errors at the bottom of that output.

If I start RabbitMQ from rabbitmq-server-generic-unix-latest-toolchain-4.1.0.tar.xz, this is output. Note that Configuration schemas found shows all schemas, even for plugins that aren't enabled:

2025-04-25 10:35:37.727234-07:00 [debug] <0.153.0> Configuration schemas found:
2025-04-25 10:35:37.727234-07:00 [debug] <0.153.0>
2025-04-25 10:35:37.727252-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/credentials_obfuscation-3.5.0/priv/schema/credentials_obfuscation.schema
2025-04-25 10:35:37.727269-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbit-4.1.0/priv/schema/rabbit.schema
2025-04-25 10:35:37.727298-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_auth_backend_cache-4.1.0/priv/schema/rabbitmq_auth_backend_cache.schema
2025-04-25 10:35:37.727315-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_auth_backend_http-4.1.0/priv/schema/rabbitmq_auth_backend_http.schema
2025-04-25 10:35:37.727340-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_auth_backend_ldap-4.1.0/priv/schema/rabbitmq_auth_backend_ldap.schema
2025-04-25 10:35:37.727355-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_auth_backend_oauth2-4.1.0/priv/schema/rabbitmq_auth_backend_oauth2.schema
2025-04-25 10:35:37.727368-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_aws-4.1.0/priv/schema/rabbitmq_aws.schema
2025-04-25 10:35:37.727409-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_event_exchange-4.1.0/priv/schema/rabbitmq_event_exchange.schema
2025-04-25 10:35:37.727422-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_management-4.1.0/priv/schema/rabbitmq_management.schema
2025-04-25 10:35:37.727435-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_management_agent-4.1.0/priv/schema/rabbitmq_management_agent.schema
2025-04-25 10:35:37.727449-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_mqtt-4.1.0/priv/schema/rabbitmq_mqtt.schema
2025-04-25 10:35:37.727471-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_peer_discovery_aws-4.1.0/priv/schema/rabbitmq_peer_discovery_aws.schema
2025-04-25 10:35:37.727486-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_peer_discovery_common-4.1.0/priv/schema/rabbitmq_peer_discovery_common.schema
2025-04-25 10:35:37.727501-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_peer_discovery_consul-4.1.0/priv/schema/rabbitmq_peer_discovery_consul.schema
2025-04-25 10:35:37.727527-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_peer_discovery_etcd-4.1.0/priv/schema/rabbitmq_peer_discovery_etcd.schema
2025-04-25 10:35:37.727542-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_peer_discovery_k8s-4.1.0/priv/schema/rabbitmq_peer_discovery_k8s.schema
2025-04-25 10:35:37.727566-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_prometheus-4.1.0/priv/schema/rabbitmq_prometheus.schema
2025-04-25 10:35:37.727584-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_shovel-4.1.0/priv/schema/rabbitmq_shovel.schema
2025-04-25 10:35:37.727594-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_stomp-4.1.0/priv/schema/rabbitmq_stomp.schema
2025-04-25 10:35:37.727604-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_stream-4.1.0/priv/schema/rabbitmq_stream.schema
2025-04-25 10:35:37.727614-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_tracing-4.1.0/priv/schema/rabbitmq_tracing.schema
2025-04-25 10:35:37.727624-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_trust_store-4.1.0/priv/schema/rabbitmq_trust_store.schema
2025-04-25 10:35:37.727634-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_web_dispatch-4.1.0/priv/schema/rabbitmq_web_dispatch.schema
2025-04-25 10:35:37.727644-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_web_mqtt-4.1.0/priv/schema/rabbitmq_web_mqtt.schema
2025-04-25 10:35:37.727655-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/rabbitmq_server-4.1.0/plugins/rabbitmq_web_stomp-4.1.0/priv/schema/rabbitmq_web_stomp.schema
2025-04-25 10:35:37.965286-07:00 [debug] <0.153.0> Loading configuration files (Cuttlefish based):
2025-04-25 10:35:37.965368-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/conf.d/10-defaults.conf
2025-04-25 10:35:37.965413-07:00 [debug] <0.153.0>   - /home/lbakken/development/lukebakken/rabbitmq-github/src/discussions/1/conf.d/20-rabbitmq.conf
2025-04-25 10:35:37.992162-07:00 [debug] <0.153.0> Adding Defaults
2025-04-25 10:35:37.992463-07:00 [debug] <0.153.0> Right Hand Side Substitutions
2025-04-25 10:35:37.992549-07:00 [debug] <0.153.0> Applying Datatypes
2025-04-25 10:35:38.003077-07:00 [debug] <0.153.0> Validation
2025-04-25 10:35:38.011174-07:00 [debug] <0.153.0> Applied 1:1 Mappings

I'll see if I can figure out why the difference exists.

[michaelklishin]

In fact, a post-launch version would be able to tell the user which plugin must be enabled. Or at least we can handle a few common cases there easily.

[lukebakken]

Sounds like a good solution. Hopefully re-scanning .schema files won't be necessary.

[dumbbell]

I can’t come up with a generic solution because it may be fine and expected to have a plugin configured but not enabled, so that it can be enabled at runtime after boot.

I think the problem is that the authentication subsystem doesn’t check its configuration on boot. I suggest we add a boot step that compares the selected auth module to the list of modules provided by plugins, and if it belongs to a plugin that is not enabled, we throw an error with the name of the plugin to enable.

It’s a bit tricky to get the list of enabled plugins, the plugins subsystem doesn’t provide such an API. Otherwise, we should have all the bits to implement that.

What do you think?

[Zerpet]

I can’t come up with a generic solution because it may be fine and expected to have a plugin configured but not enabled, so that it can be enabled at runtime after boot.

I was going to comment exactly this 🙂 I think it's valid to have a plugin configured and not enabled. An example would be during a troubleshooting session, you disable a plugin and leave RMQ configuration unchanged, and later restart the node.

IMO the current behaviour, an exception in the logs, is fine. I would change the log level, or perhaps add a dedicated log message, for the specific case of authentication backends provided by disabled plugins. I don't think RMQ should fail to boot if a plugin is configured and disabled.

[lukebakken]

specific case of authentication backends provided by disabled plugins

In the case of the LDAP plugin, the only indication that this was the case was a pretty unhelpful stack trace.

I suggest we add a boot step that compares the selected auth module to the list of modules provided by plugins, and if it belongs to a plugin that is not enabled, we throw an error with the name of the plugin to enable.

I agree that this is what should happen with authentication-related plugins.

[lukebakken]

Related issue: #2085

[dumbbell]

Hi Luke! I will think about this issue next week when I get back from holidays. Thank you for the detailed report! :-)

[lukebakken]

@dumbbell @michaelklishin this issue is related:

#13896

The user did not have the rabbitmq_auth_backend_cache plugin enabled, and was only given a very unhelpful stack trace as an indication. I didn't even realize that was the issue during reproduction until I tested it with this advanced.config file -

https://github.com/lukebakken/rabbitmq-server-13896/blob/main/advanced.config