Error 'Sent fatal SSL alert: internal error' when attempting to connect with YubiKey

[rau98]

IMPORTANT NOTE
Bugs about OpenVPN Access Server, OpenVPN Connect or any other product by OpenVPN Inc. should be directly reported to OpenVPN Inc. at https://support.openvpn.net

Describe the bug
I have a VPN set up that I am able to connect to using a config.ovpn that points to my cacert and pkcs12 file, but when I tried using a version of the same config.ovpn file modified to use PKCS11 instead, it fails.

To Reproduce
Load a YubiKey with the pkcs12 keystore needed to connect to the VPN. Create a config.ovpn file that uses the YubiKey and then try to connect to VPN. This is what my config file looks like:

client
dev tun
proto udp
remote 192.168.1.1 1194
resolv-retry infinite
nobind
persist-key
persist-tun
comp-lzo
verb 6
auth-nocache
ca cacert.pem
cipher AES-256-GCM
tls-auth ../ta.key 1
pkcs11-providers /usr/local/lib64/libykcs11.so.2.7.2
pkcs11-id 'Yubico\x20\x28www\x2Eyubico\x2Ecom\x29/YubiKey\x20YK5/29471392/YubiKey\x20PIV\x20\x2329471392/02'

When I try to connect, I am prompted for the YubiKey PIN twice. After entering it twice, I get this error. It never asks me to touch the YubiKey.

Enter YubiKey PIV #29471392 token Password:
Enter YubiKey PIV #29471392 token Password:
2025-09-25 16:27:50 us=16025 xkey_provider: In xkey_sign_dispatch: xkey_provider: external sign op returned ret = 0 siglen = 256
2025-09-25 16:27:50 us=16231 Sent fatal SSL alert: internal error
2025-09-25 16:27:50 us=16259 xkey_provider: In signature_freectx: entry
2025-09-25 16:27:50 us=16284 xkey_provider: In keydata_free: entry
2025-09-25 16:27:50 us=16336 OpenSSL: error:0300007F:digital envelope routines::expecting an rsa key:
2025-09-25 16:27:50 us=16357 OpenSSL: error:0300007F:digital envelope routines::expecting an rsa key:
2025-09-25 16:27:50 us=16380 OpenSSL: error:0A080006:SSL routines::EVP lib:
2025-09-25 16:27:50 us=16395 TLS_ERROR: BIO read tls_read_plaintext error
2025-09-25 16:27:50 us=16414 TLS Error: TLS object -> incoming plaintext read error
2025-09-25 16:27:50 us=16429 TLS Error: TLS handshake failed
2025-09-25 16:27:50 us=16763 TCP/UDP: Closing socket
2025-09-25 16:27:50 us=16842 SIGUSR1[soft,tls-error] received, process restarting
2025-09-25 16:27:50 us=16882 Restart pause, 1 second(s)

Expected behavior
I should be able to connect to the VPN after entering my PIN and touching the security key.

Version information (please complete the following information):

• OS: Rocky 9.3
• OpenVPN version: 2.6.9
• OpenSSL version: 3.2.2
• pkcs11-helper version: 1.30
• YubiKey PKCS11 library version: 2.7.1

Additional context
I have tried various different versions of openvpn, pkcs11-helper, and libykcs but to no avail. This most recent attempt uses the latest versions of everything I've tried so far.

[schwabe]

2025-09-25 16:27:50 us=16336 OpenSSL: error:0300007F:digital envelope routines::expecting an rsa key:

Sounds like some mismatch in certificates and private or something else in the pkcs11 stack going haywire. It is hard to debug/know what is happening here with the current log.

[rau98]

I've concluded that it's an issue with my machine, since I was able to get the expected results using other machines. However, based on the "expecting an rsa key" error message, I would like to ask if OpenVPN plans on supporting EC keys any time soon. Thank you.

[schwabe]

@rau98 not supporting EC keys would something that is only in the pkcs11 stack. Both the windows certificate storage as well the external key support via management interface (e.g. OpenVPN for Android) work flawlessly with EC keys.