LDAP authentication fails after Erlang upgrade to 24.3.4

[jande353]

Hi,

Today we experienced a new problem with LDAP authentication, all worked well with RabbitMQ 3.9.16 and Erlang 24.3.3 but after upgrading to Erlang 24.3.4 the LDAP authentication no longer works. In the logs we can now see,

2022-05-04 12:58:30.046747+02:00 [notice] <0.1634.0> TLS client: In state hello at ssl_handshake.erl:892 generated CLIENT ALERT: Fatal - Handshake Failure
2022-05-04 12:58:30.046747+02:00 [notice] <0.1634.0> - {unknown_or_malformed_handshake,13}
2022-05-04 12:58:30.047112+02:00 [info] <0.500.0> LDAP connect error: {error,"connect failed"}
2022-05-04 12:58:30.047262+02:00 [info] <0.1614.0> LDAP DECISION: login for [email protected]: {error,
2022-05-04 12:58:30.047262+02:00 [info] <0.1614.0> ldap_connect_error}
2022-05-04 12:58:30.047380+02:00 [warning] <0.1614.0> HTTP access denied: rabbit_auth_backend_cache failed authenticating [email protected]: ldap_connect_error

[lukebakken]

If you downgrade to 23.3.3 is the issue fixed?

TLS client: In state hello at ssl_handshake.erl:892 generated CLIENT ALERT: Fatal - Handshake Failure

There is one TLS-related fix in the 24.3.4 release notes -

erlang/otp#5835

...but that has to do with client certificate authentication, which I'm assuming you're not using for LDAP, correct?

[michaelklishin]

We don't have a lot of log information here either but it looks like an LDAP client TLS handshake failure. There were TLS client changes in Erlang 24.3.4.

After rectifying the TLS configuration to not enable TLS 1.3 and 1.2 at the same time (just stick to 1.2 for now), use the TLS Troubleshooting guide to see if openssl s_client can connect to your the LDAP servers
with the same certificate and key pair.

[michaelklishin]

From Erlang 24.3.4 TLS implementation release notes:

Client certification could fail if TLS-1.3 enabled client
negotiated TLS-1.2 connection with the server, this is
due to the wrong version being used when decoding the
certificate request message from the server.

So removing this mix of TLSv1.3 and v1.2 sounds like a good idea to me to avoid confusion.
This was an area of recent changes.

[michaelklishin]

That issue was reported on GitHub: erlang/otp#5898

[torfaxen]

@michaelklishin I created an issue on the Erlang github yesterday: erlang/otp#5961.
Just asked becuase you said earlier "@jande353 can you put together an example repo that demonstrates what your client does? We need to report this to the Erlang maintainers and the more specific details we can collect, the better."

[lukebakken]

I added a note to erlang/otp#5961 pointing to this discussion.

[michaelklishin]

24.3.4 is very new, I don't think anyone has tried it with TLS and LDAP specifically. Nothing in RabbitMQ has changed in that area.

Please post full config files instead of snippets, there may be important details missing.

[michaelklishin]

You cannot support both TLSv1.3 and TLSv1.2, they share no cipher suites. You have to pick one or the other. If you go with TLSv1.3, all clients that connect to RabbitMQ must support TLSv1.3. I would
stick to TLSv1.2 for the purpose of this investigation.

[jande353]

rabbitmq.conf.txt

[jande353]

I attached a new rabbitmq.conf file, I have changed so we are now using only LDAP without cache plugin and only using TLSv1.2. However I still get the same error message when trying to login

2022-05-05 11:41:18.232113+02:00 [notice] <0.9800.0> - {unknown_or_malformed_handshake,13}
2022-05-05 11:41:18.237886+02:00 [notice] <0.9803.0> TLS client: In state hello at ssl_handshake.erl:892 generated CLIENT ALERT: Fatal - Handshake Failure
2022-05-05 11:41:18.237886+02:00 [notice] <0.9803.0> - {unknown_or_malformed_handshake,13}
2022-05-05 11:41:18.238199+02:00 [info] <0.504.0> LDAP connect error: {error,"connect failed"}
2022-05-05 11:41:18.238309+02:00 [info] <0.9783.0> LDAP DECISION: login for zbx_monitor: {error,ldap_connect_error}
2022-05-05 11:41:18.238471+02:00 [warning] <0.9783.0> HTTP access denied: rabbit_auth_backend_ldap failed authenticating zbx_monitor: ldap_connect_error

[michaelklishin]

Thanks, that's useful. Any chance you can put together a Docker Compose example against an LDAP server of some kind?

[jande353]

I would love to help with that but I don't know how to do that, I don't have any LDAP server that you could use. In this customer sandbox environment, we are testing RabbitMQ 3.10.0 and Erlang 24.3.4 but we have also tried with RabbitMQ 3.9.16 with same result. As I also wrote we have another customer where we use OpenShift and the RabbitMQ cluster operator, we tried with the latest image with RabbitMQ 3.10.0 and Erlang 24.3.4 and we had the exact same result as in the traditional Linux Ubuntu environment. Just now I also tried in our internal OpenShift environment with RabbitMQ and same error. I don’t know if it’s possible for you but if you like we could have a virtual meeting and I can share our internal OpenShift and RabbitMQ environment and we could look at it together in case you want me to try different settings and check logs etc.

[torfaxen]

Hello!

I am also working on the same issue as jande353, the same problem shows up with rabbitmq 3.10.0 and erlang 24.3.4.

[michaelklishin]

RabbitMQ does not implement the LDAP client used, it's a part of Erlang/OTP's standard library.

[torfaxen]

ok, have other users reported similar issues with erlang 24.3.4 and rabbitmq when upgrading?

[michaelklishin]

Not that I have seen but 24.3.4 is a couple of days old.

[torfaxen]

I am going to try and downgrade Erlang to 23.3.3 and se what happens. Do you have a guide to this or is it the one you posted earlier today?

[michaelklishin]

On Linux, the entire guide is: stop RabbitMQ. Uninstall version A of Erlang. Install version B of Erlang. Start RabbitMQ.

[michaelklishin]

According to the rabbitmq.conf provided, this environment uses

• Internal and LDAP + caching authN/authZ backends
• x.509-certificate based authentication

all at the same time. That's a pretty unique combination and it makes reproduction
significantly more involved.
Our best bet here is to try to reproduce with just an Erlang LDAP client connection.

It also attempts to use TLSv1.2 and TLSv1.3:

ssl_options.versions.1 = tlsv1.3
ssl_options.versions.2 = tlsv1.2

ssl_options.ciphers.1  = TLS_AES_256_GCM_SHA384
ssl_options.ciphers.2  = TLS_AES_128_GCM_SHA256
ssl_options.ciphers.3  = TLS_CHACHA20_POLY1305_SHA256
ssl_options.ciphers.4  = TLS_AES_128_CCM_SHA256
ssl_options.ciphers.5  = TLS_AES_128_CCM_8_SHA256

ssl_options.ciphers.6  = ECDHE-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.7  = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.8  = ECDHE-ECDSA-AES256-SHA384
ssl_options.ciphers.9  = ECDHE-RSA-AES256-SHA384
ssl_options.ciphers.10  = ECDH-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.11  = ECDH-RSA-AES256-GCM-SHA384
ssl_options.ciphers.12  = ECDH-ECDSA-AES256-SHA384
ssl_options.ciphers.13  = ECDH-RSA-AES256-SHA384
ssl_options.ciphers.14  = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.15 = DHE-DSS-AES256-GCM-SHA384
ssl_options.ciphers.16 = DHE-RSA-AES256-SHA256
ssl_options.ciphers.17 = DHE-DSS-AES256-SHA256
ssl_options.ciphers.18 = ECDHE-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.19 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.20 = ECDHE-ECDSA-AES128-SHA256
ssl_options.ciphers.21 = ECDHE-RSA-AES128-SHA256
ssl_options.ciphers.22 = ECDH-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.23 = ECDH-RSA-AES128-GCM-SHA256
ssl_options.ciphers.24 = ECDH-ECDSA-AES128-SHA256
ssl_options.ciphers.25 = ECDH-RSA-AES128-SHA256
ssl_options.ciphers.26 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.27 = DHE-DSS-AES128-GCM-SHA256
ssl_options.ciphers.28 = DHE-RSA-AES128-SHA256
ssl_options.ciphers.29 = DHE-DSS-AES128-SHA256
ssl_options.ciphers.30 = ECDHE-ECDSA-AES256-SHA
ssl_options.ciphers.31 = ECDHE-RSA-AES256-SHA
ssl_options.ciphers.32 = DHE-RSA-AES256-SHA
ssl_options.ciphers.33 = DHE-DSS-AES256-SHA
ssl_options.ciphers.34 = ECDH-ECDSA-AES256-SHA
ssl_options.ciphers.35 = ECDH-RSA-AES256-SHA
ssl_options.ciphers.36 = ECDHE-ECDSA-AES128-SHA
ssl_options.ciphers.37 = ECDHE-RSA-AES128-SHA
ssl_options.ciphers.38 = DHE-RSA-AES128-SHA
ssl_options.ciphers.39 = DHE-DSS-AES128-SHA
ssl_options.ciphers.40 = ECDH-ECDSA-AES128-SHA
ssl_options.ciphers.41 = ECDH-RSA-AES128-SHA

which the docs clearly say will not be possible because
TLSv1.3 shares no cipher suites with earlier versions.

[lukebakken]

Which LDAP server is being used?

[jande353]

Microsoft Active Directory

[lukebakken]

@jande353 here is what I would like you to do on your RabbitMQ server, or another machine that has Erlang 24.3.4 installed and can communicate with your LDAP server.

First, ensure that the erl and erlc commands are in your PATH. Then do the following:

git clone https://github.com/lukebakken/rabbitmq-server-4726.git
cd rabbitmq-server-4726
make

The following will ensure that Erlang and TLS are working on this machine. In one terminal window run this:

make run_tls_server

...and in another window:

make run_tls_client

You should see successful connection and communcation.

I have set of self-generated certificates that I saved in the certs/ directory. You'll have to overwrite these with the certs you have for your RabbitMQ server:

/etc/rabbitmq/ssl/ca_certificate_sandbox01.pem
/etc/rabbitmq/ssl/server_certificate_sandbox01.pem
/etc/rabbitmq/ssl/server_key_sandbox01.pem

An example set of commands to do that:

cp -vf /etc/rabbitmq/ssl/ca_certificate_sandbox01.pem ./certs/ca_certificate.pem
cp -vf /etc/rabbitmq/ssl/server_certificate_sandbox01.pem ./certs/client_localhost_certificate.pem
cp -vf /etc/rabbitmq/ssl/server_key_sandbox01.pem ./certs/client_localhost_key.pem

After doing the above, the following will start an LDAP Erlang client and will try to connect to the specified server and port:

make HOST=ldap-server.mycompany.com PORT=636 run_eldap_client

When you run all of the above commands, capture ALL of the output into a file and ATTACH the file to your response.

[jande353]

updated_erlang_patch_log3.txt

[jande353]

[jande353]

Unfortunately, I don’t see any difference from before when trying to use usename/password, is it possible to have more logging in your test client?

[lukebakken]

Thanks for the information. I don't think there is more logging I can try out. At this point my next step is to get TLS working with my AD LDS instance to see if I can reproduce the issue and confirm the fix.

The fact that you're no longer getting the handshake error suggests that the issue is fixed. I'm probably doing something incorrect in my test application.

[lukebakken]

OK I can reproduce the error with AD LDS over TLS using Erlang 24.3.4. I can see that my test application works with 24.3.3. Using the latest erlang, it also hangs like you report:

lbakken@shostakovich ~/development/lukebakken/rabbitmq-server-4726 (lukebakken/bakkenl-z01 *%=)
$ for erlang_version in 24.3.3 24.3.4 ref:master; do echo ERLANG_VERSION: $erlang_version; git clean -xffd; asdf local erlang "$erlang_
version"; asdf current erlang; make PORT=636 HOST=bakkenl-z01 DN=lbakken PASSWD=test1234 run_eldap_client; echo; done
ERLANG_VERSION: 24.3.3
Removing .tool-versions
Removing eldap_client.beam
erlang          24.3.3          /home/lbakken/development/lukebakken/rabbitmq-server-4726/.tool-versions
erlc +debug eldap_client.erl
erl -noinput -s eldap_client start bakkenl-z01 636 lbakken test1234
[INFO] Host: bakkenl-z01
[INFO] Port: 636
[INFO] Dn: lbakken
[INFO] Passwd: test1234
[INFO] connection opened to bakkenl-z01
[INFO] simple bind succeeded, DN: "lbakken"
[INFO] closing LDAP connection...
[INFO] LDAP connection is closed.

ERLANG_VERSION: 24.3.4
Removing .tool-versions
Removing eldap_client.beam
erlang          24.3.4          /home/lbakken/development/lukebakken/rabbitmq-server-4726/.tool-versions
erlc +debug eldap_client.erl
erl -noinput -s eldap_client start bakkenl-z01 636 lbakken test1234
[INFO] Host: bakkenl-z01
[INFO] Port: 636
[INFO] Dn: lbakken
[INFO] Passwd: test1234
{"init term=NOTICE REPORT==== 19-May-2022::07:12:03.622088 ===
TLS client: In state hello at ssl_handshake.erl:892 generated CLIENT ALERT: Fatal - Handshake Failure
 - {unknown_or_malformed_handshake,13}
inating in do_boot",{{badmatch,{error,"connect failed"}},[{eldap_client,start,4,[{file,"eldap_client.erl"},{line,37}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({{badmatch,{error,connect failed}},[{eldap_client,start,4,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: erl_crash.dump...done
make: *** [Makefile:29: run_eldap_client] Error 1

ERLANG_VERSION: ref:master
Removing .tool-versions
Removing eldap_client.beam
Removing erl_crash.dump
erlang          ref:master      /home/lbakken/development/lukebakken/rabbitmq-server-4726/.tool-versions
erlc +debug eldap_client.erl
erl -noinput -s eldap_client start bakkenl-z01 636 lbakken test1234
[INFO] Host: bakkenl-z01
[INFO] Port: 636
[INFO] Dn: lbakken
[INFO] Passwd: test1234
^C
BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution
^C

I am going to follow up with the OTP team - erlang/otp#5975

[michaelklishin]

A member of the Erlang team who works on TLS may have a lead erlang/otp#5961 (comment). It can take a while to put together a test, pass QA and get a release out. But as soon as a PR is merged, it will be possible
to build Erlang from source using kerl to try it out.

[lukebakken]

I have verified that the Erlang/OTP team has fixed this issue -

erlang/otp#5961 (comment)

cc @jande353

[lukebakken]

cc @jande353

https://erlangforums.com/t/patch-package-otp-24-3-4-1-released/1524

[jande353]

Thanks we have tried it and it works fine!