gh-150101: Expose OpenSSL's error queue through SSLError.error_queue

[samatjain]

See gh-150101 for background

In the associated PR, we drain the OpenSSL error queue and expose it via the error_queue attribute on the SSLError exception. With this modification to Python's SSL module, we can see what's in that queue:

With script:

#!/usr/bin/env python3
"""Test TLS 1.2 connection to a site using only stdlib."""

import pprint
import ssl
import socket

def main():
    hostname = "www.example.com"

    # Create SSL context with TLS 1.2 maximum
    context = ssl.create_default_context()
    context.maximum_version = ssl.TLSVersion.TLSv1_2

    # Connect with TLS 1.2
    try:
        with socket.create_connection((hostname, 443)) as sock:
            with context.wrap_socket(sock, server_hostname=hostname) as ssock:
                print(f"Connected to {hostname}")
                print(f"TLS version: {ssock.version()}")
                print(f"Cipher: {ssock.cipher()}")

                # Send a simple HTTP request
                request = f"GET / HTTP/1.1\r\nHost: {hostname}\r\nConnection: close\r\n\r\n".encode('ascii')
                ssock.sendall(request)

                response = ssock.recv(4096)
                status_line = response.split(b'\r\n')[0].decode('utf-8', errors='replace')
                print(f"\nResponse status: {status_line}")
    except ssl.SSLError as e:
        if hasattr(e, 'error_queue'):
            print("\nSSL error queue:")
            pprint.pprint(e.error_queue)
        raise

if __name__ == "__main__":
    main()

We get that OpenSSL error queue that's helpful in debugging these kinds of problems:

$ python3.13 test_tls1.2.py

SSL error queue:
['error:1C8000E9:Provider routines::reason(233)  '
'(providers/implementations/kdfs/tls1_prf.c:(unknown function):208)',
'error:0A0C0103:SSL routines::internal error  (ssl/t1_enc.c:tls1_PRF:79)']
Traceback (most recent call last):
  File "/opt/bigcorp/test_tls1.2.py", line 37, in <module>
    main()
    ~~~~^^
  File "/opt/bigcorp/test_tls1.2.py", line 18, in main
    with context.wrap_socket(sock, server_hostname=hostname) as ssock:
        ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/bigcorp/lib/python3.13/ssl.py", line 460, in wrap_socket
    return self.sslsocket_class._create(
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/bigcorp/lib/python3.13/ssl.py", line 1084, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
  File "/opt/bigcorp/lib/python3.13/ssl.py", line 1380, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLError: [SSL] internal error (_ssl.c:1102)

It's worth noting, that on a stock Linux distribution (e.g. Ubuntu 24.04), the above script doesn't fail. Trying to attempt to raise SSLError in the limited ways I know how, e.g.

context = ssl.create_default_context()
context.check_hostname = True
context.verify_mode = ssl.CERT_REQUIRED

# Connect to a site but verify against a completely different hostname
# that doesn't match any SAN in the certificate

try:
    with socket.create_connection(("www.python.org", 443)) as sock:
        # Try to verify as "totally-fake-hostname.example.com"
        with context.wrap_socket(
            sock, server_hostname="totally-fake-hostname.example.com"
        ) as ssock:
            print(f"Connected: {ssock.version()}")
except ssl.SSLError as e:
    print(f"SSLError from hostname mismatch: {e}, {e.error_queue=}")
    raise

results in only 1 item in the OpenSSL error queue (see first line, you may need to scroll sideways):

SSLError from hostname mismatch: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'totally-fake-hostname.example.com'. (_ssl.c:1088), e.error_queue=['error:0A000086:SSL routines::certificate verify failed  (../ssl/statem/statem_clnt.c:tls_post_process_server_certificate:1889)']
Traceback (most recent call last):
  File "/home/xjjk/src/git/cpython/test_ssl_error.py", line 279, in <module>
    test_ssl_error_with_hostname_verification()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/xjjk/src/git/cpython/test_ssl_error.py", line 31, in test_ssl_error_with_hostname_verification
    with context.wrap_socket(
        ~~~~~~~~~~~~~~~~~~~^
        sock, server_hostname="totally-fake-hostname.example.com"
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ) as ssock:
    ^
  File "/home/xjjk/src/git/cpython/install/lib/python3.13t/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
  File "/home/xjjk/src/git/cpython/install/lib/python3.13t/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
  File "/home/xjjk/src/git/cpython/install/lib/python3.13t/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'totally-fake-hostname.example.com'. (_ssl.c:1088)

i.e.

e.error_queue=['error:0A000086:SSL routines::certificate verify failed  (../ssl/statem/statem_clnt.c:tls_post_process_server_certificate:1889)']

is new.

I can't think up of a way to raise an SSLError that results in a longer OpenSSL error queue that's significantly different than what was already in the original exception. As such, the unit tests are sort of superficial (and difficult to reliably improve, as the messages in the queue may change depending on the version of OpenSSL used). Even without a better test case, outputing what OpenSSL provides directly (e.g. we add the OpenSSL filenames and functions here) may help folks because those errors may be more Google-able.

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #32764334 | 📁 Comparing 28e1f40 against main (8b31d08)

  🔍 Preview build  

2 files changed

± library/ssl.html
± whatsnew/changelog.html

[gpshead]

FYI we spell this "versionadded:: next" and the docs automation replaces it with the right version at the right time. (don't worry about it, 3.16 is correct)

[picnixz]

I would rather not expose internal error queue and let better error messages. Mnemonics are not always up to date either

[gpshead]

decref current_eq_msg after this (regardless of success or failure) to avoid a memory leak.

[gpshead]

I'd rather we did not put large things on the stack. Have it be a pointer set to NULL here. allocate space within the while loop if null before we use it. free it unconditionally when not NULL later on the ways out of the function.

[gpshead]

I think this is OpenSSL >= 3.x only - we also work with some 1.1.1-ish forks such as AWS-LC and can still build and link against 1.1.1 itself so conditional compilation is likely needed here.

[gpshead]

fail: needs to Py_XDECREF(error_queue);

[gpshead]

is this correct? oldest first in one list vs value in errcode being the last one when there are multiple.

[gpshead]

Add a relevant Py_CLEAR(state->...) for this.

[gpshead]

error_queue?

[gpshead]

assertGreaterEqual

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.