TLS failure for any incomming mails from 1and1/United Internet/Ionos servics (i.e. web.de/gmx/1&1) / Cannot receive mails when enabling DANE

[jabdoa2]

Situation: I setup a Stalwart instance with Lets Encrypt certificate (RSA; tried 2048 and 4096 bit) on the server hostname. I also got DNSSEC and DANE setup. Sending mails and getting them delivered works flawless so far. Receiving emails worked fine for all senders before enabling DANE (i.e. setting up TLSA records).

Issue: After setting up DANE I have issues to receive mails from 1and1/United Internet/Ionos services (forth biggest mobile operator and one of the big ISPs in Germany - they run multiple email services such as web.de). Any other senders work fine (i.e. GMail). I tested my setup using internet.nl, checktls and DANE Validator (dane.sys4.de) - all of them report my setup as fine. I can deliver mails to web.de with working TLS. However, they fail to send mails with TLS to me. Without DANE they will fallback to unencrypted (problematic GDPR-wise in Europe) and with DANE they will generate a bounce on their side.

Sending a mail (with DANE enabled) will instantly generate a bounce on their side:

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of
its recipients. This is a permanent error.

The following address failed:

[email protected]:
TLS authentication failed for remote MX

This is how an incoming mail looks in Stalwart (without DANE it will fall back to unencrypted after this):

DEBUG SMTP connection started (smtp.connection-start) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457
TRACE Write batch operation (store.data-write) elapsed = 0ms, total = 2
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.transfer-limit", result = "Integer(262144000)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.duration", result = "Integer(600000)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.timeout", result = "Integer(300000)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "auth.spf.verify.ehlo", result = "Integer(2)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "auth.spf.verify.mail-from", result = "Integer(2)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "auth.iprev.verify", result = "Integer(2)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.ehlo.require", result = "Integer(1)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.ehlo.reject-non-fqdn", result = "Integer(1)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.auth.directory", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.auth.directory", result = ""
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.auth.require", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.auth.errors.total", result = "Integer(3)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.auth.errors.wait", result = "Integer(5000)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.expn", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.vrfy", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.connect.script", result = ""
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "server.connect.hostname", result = "String(Borrowed("xx.my-domain.net"))"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.connect.greeting", result = "String(Owned("xx.my-domain.net Stalwart ESMTP at your service"))"
TRACE Raw SMTP output sent (smtp.raw-output) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, size = 53, contents = "220 xx.my-domain.net Stalwart ESMTP at your service\r\n"
TRACE Raw SMTP input received (smtp.raw-input) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, size = 18, contents = "EHLO mout.web.de\r\n"
INFO SMTP EHLO command (smtp.ehlo) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, domain = "mout.web.de"
INFO SPF EHLO check failed (smtp.spf-ehlo-fail) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, domain = "mout.web.de", result = No SPF record (spf.none), elapsed = 18ms
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "report.spf.send", result = "Array([Integer(1), Integer(86400000)])"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.ehlo.script", result = ""
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.pipelining", result = "Integer(1)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.chunking", result = "Integer(1)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.expn", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.vrfy", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.requiretls", result = "Integer(1)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.dsn", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.auth.mechanisms", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.future-release", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.future-release", result = ""
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.deliver-by", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.deliver-by", result = ""
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.mt-priority", result = "Integer(0)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.mt-priority", result = ""
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.data.limits.size", result = "Integer(104857600)"
TRACE Expression evaluation result (eval.result) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, id = "session.extensions.no-soliciting", result = "String(Borrowed(""))"
TRACE Raw SMTP output sent (smtp.raw-output) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, size = 209, contents = "250-xx.my-domain.net you had me at EHLO\r\n250-STARTTLS\r\n250-SMTPUTF8\r\n250-SIZE 104857600\r\n250-REQUIRETLS\r\n250-PIPELINING\r\n250-NO-SOLICITING\r\n250-ENHANCEDSTATUSCODES\r\n250-CHUNKING\r\n250-BINARYMIME\r\n250 8BITMIME\r\n"
TRACE Raw SMTP input received (smtp.raw-input) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, size = 10, contents = "STARTTLS\r\n"
DEBUG SMTP STARTTLS command (smtp.start-tls) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457
TRACE Raw SMTP output sent (smtp.raw-output) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, size = 31, contents = "220 2.0.0 Ready to start TLS.\r\n"
DEBUG TLS handshake error (tls.handshake-error) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, listenerId = "smtp", reason = "received fatal alert: InsufficientSecurity"
DEBUG SMTP connection ended (smtp.connection-end) listenerId = "smtp", localPort = 25, remoteIp = 212.227.17.12, remotePort = 50457, elapsed = 119ms

They seem to be unhappy with the security parameters (ciphers, signatures?) offered by Stalwart. I did not disable any ciphers and verified that openssl can connect using TLS1.2 and TLS1.3. Other mail providers can do it too.

Question 1: Is there a way to increase tls logging to figure out which ciphers they advertise?

Interestingly, I can send mails to their mail service with TLS:

TRACE Raw SMTP input received (delivery.raw-input) queueId = 270755677836697089, queueName = "remote", from = "[email protected]", to = ["[email protected]"], size = 2575, total = 1, contents = "220 web.de (mxweb105) Nemesis ESMTP Service ready\r\n", size = 51
TRACE Raw SMTP output sent (delivery.raw-output) queueId = 270755677836697089, queueName = "remote", from = "[email protected]", to = ["[email protected]"], size = 2575, total = 1, contents = "EHLO xx.my-domain.net\r\n", size = 23
TRACE Raw SMTP input received (delivery.raw-input) queueId = 270755677836697089, queueName = "remote", from = "[email protected]", to = ["[email protected]"], size = 2575, total = 1, contents = "250-web.de Hello xx.my-domain.net [x.x.x.x]\r\n250-8BITMIME\r\n250-SIZE 157286400\r\n250 STARTTLS\r\n", size = 98
DEBUG SMTP EHLO command (delivery.ehlo) queueId = 270755677836697089, queueName = "remote", from = "[email protected]", to = ["[email protected]"], size = 2575, total = 1, domain = "web.de", hostname = "mx-ha02.web.de", details = 20971521, elapsed = 22ms
TRACE Raw SMTP output sent (delivery.raw-output) queueId = 270755677836697089, queueName = "remote", from = "[email protected]", to = ["[email protected]"], size = 2575, total = 1, contents = "STARTTLS\r\n", size = 10
TRACE Raw SMTP input received (delivery.raw-input) queueId = 270755677836697089, queueName = "remote", from = "[email protected]", to = ["[email protected]"], size = 2575, total = 1, contents = "220 OK\r\n", size = 8
INFO SMTP STARTTLS command (delivery.start-tls) queueId = 270755677836697089, queueName = "remote", from = "[email protected]", to = ["[email protected]"], size = 2575, total = 1, domain = "web.de", hostname = "mx-ha02.web.de", version = "TLSv1_3", details = "TLS13_AES_256_GCM_SHA384", elapsed = 67ms

So it looks their setup is decent and supports TLS1.3 with a cipher which would also be supported by Stalwart on the receiving side.

This is kind of a game stopper in Germany as it would force us to remove TLSA and risk violating GDPR. Any ideas how to fix this?