MDEV-41008: Fix X509 issuer/subject comparison for OpenSSL 3

[vaintroub]

Summary

OpenSSL 3 escapes / and + with a backslash when rendering an RDN value in X509_NAME_oneline(); OpenSSL 1.1 and WolfSSL never escape them. A REQUIRE ISSUER/REQUIRE SUBJECT grant created under one library can stop matching the same certificate after switching to, or upgrading past, OpenSSL 3.

• Comparison defaults to plain strcmp(), so a mismatch caused by differing escaping is rejected rather than silently accepted.
• New old_mode flag X509_LENIENT_COMPARE opts into ignoring the escaping backslash on either side of the comparison, restoring the old cross-library match for sites that need it.
• This is opt-in because ignoring the backslash reintroduces the ambiguity between a single RDN whose value contains a literal / or + and a multi-RDN name using it as a separator — a maliciously crafted certificate could otherwise exploit this to impersonate a different identity. See MDEV-41008 for the reported issue and a related WolfSSL-specific report discussed on the ticket.

Test plan

• New test main.mdev-41008: verifies strict rejection by default and successful connection with old_mode=X509_LENIENT_COMPARE set, for both REQUIRE SUBJECT and REQUIRE ISSUER.

[Copilot]

🟡 Changes recommended

SSL authentication can still crash on X509_NAME_oneline() returning NULL because the result is logged/compared as a C string without a NULL guard.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR addresses cross-library incompatibilities in REQUIRE ISSUER / REQUIRE SUBJECT matching when moving to OpenSSL 3, where X509_NAME_oneline() escapes / and + differently than OpenSSL 1.1 / WolfSSL. It introduces an opt-in old_mode flag to enable a lenient comparison mode that ignores those specific escaping differences.

Changes:

• Add old_mode=X509_LENIENT_COMPARE (bit 7) and expose it via --old-mode / @@old_mode.
• Switch X509 issuer/subject matching from strcmp() to a new comparator that can optionally ignore \ before / and +.
• Add a new regression test (main.mdev-41008) and update existing old_mode sysvar tests and help output baselines.

File summaries

File	Description
sql/sys_vars.cc	Adds X509_LENIENT_COMPARE to the old_mode name set so it can be parsed/printed.
sql/sql_class.h	Defines OLD_MODE_X509_LENIENT_COMPARE bit flag.
sql/sql_acl.cc	Implements lenient X509 oneline comparison and applies it to issuer/subject checks.
mysql-test/suite/sys_vars/t/old_mode_basic.test	Extends coverage to accept the new mode name and numeric value 128; shifts invalid numeric test to 256.
mysql-test/suite/sys_vars/r/old_mode_basic.result	Updates expected outputs for the extended old_mode test.
mysql-test/main/mysqld--help.result	Updates --old-mode help text baseline to include the new mode.
mysql-test/main/mdev-41008.test	New regression test for strict-vs-lenient matching behavior.
mysql-test/main/mdev-41008.result	Expected output for the new regression test.

Review details

Suppressed comments (1)

sql/sql_acl.cc:14583

• X509_NAME_oneline() can return NULL on error; ptr is then dereferenced by DBUG_PRINT/ sql_print_information and passed to the comparison, which can crash during authentication. Add a NULL check and fail the SSL attribute check cleanly before logging/comparing.

      char *ptr= X509_NAME_oneline(X509_get_subject_name(cert), 0, 0);
      DBUG_PRINT("info", ("comparing subjects: '%s' and '%s'",
                         acl_user->x509_subject, ptr));
      if (my_x509_oneline_cmp(thd, acl_user->x509_subject, ptr))
      {

• Files reviewed: 8/8 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

SSL issuer/subject checks now call the new comparator but still lack NULL handling for X509_NAME_oneline(..., 0, 0), which can lead to undefined behavior/crashes when logging/comparing.

Review details

Suppressed comments (2)

sql/sql_acl.cc:14565

• X509_NAME_oneline(..., 0, 0) can return NULL on allocation/error; DBUG_PRINT with %s and the subsequent compare/free then risk undefined behavior or a crash. Add a NULL check before using ptr and bail out cleanly (freeing cert).

      char *ptr= X509_NAME_oneline(X509_get_issuer_name(cert), 0, 0);
      DBUG_PRINT("info", ("comparing issuers: '%s' and '%s'",
                         acl_user->x509_issuer, ptr));
      if (my_x509_oneline_cmp(thd, acl_user->x509_issuer, ptr))

sql/sql_acl.cc:14582

• Same issue for the subject comparison: if X509_NAME_oneline(..., 0, 0) returns NULL, DBUG_PRINT/comparison will dereference NULL via %s/strcmp-like logic. Add a NULL check before logging/comparing, and free cert before returning.

      char *ptr= X509_NAME_oneline(X509_get_subject_name(cert), 0, 0);
      DBUG_PRINT("info", ("comparing subjects: '%s' and '%s'",
                         acl_user->x509_subject, ptr));
      if (my_x509_oneline_cmp(thd, acl_user->x509_subject, ptr))

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

[vaintroub]

🔵 Needs a closer look

SSL issuer/subject checks now call the new comparator but still lack NULL handling for X509_NAME_oneline(..., 0, 0), which can lead to undefined behavior/crashes when logging/comparing.

Review details

Suppressed comments (2)

sql/sql_acl.cc:14565

• X509_NAME_oneline(..., 0, 0) can return NULL on allocation/error; DBUG_PRINT with %s and the subsequent compare/free then risk undefined behavior or a crash. Add a NULL check before using ptr and bail out cleanly (freeing cert).

      char *ptr= X509_NAME_oneline(X509_get_issuer_name(cert), 0, 0);
      DBUG_PRINT("info", ("comparing issuers: '%s' and '%s'",
                         acl_user->x509_issuer, ptr));
      if (my_x509_oneline_cmp(thd, acl_user->x509_issuer, ptr))

sql/sql_acl.cc:14582

• Same issue for the subject comparison: if X509_NAME_oneline(..., 0, 0) returns NULL, DBUG_PRINT/comparison will dereference NULL via %s/strcmp-like logic. Add a NULL check before logging/comparing, and free cert before returning.

      char *ptr= X509_NAME_oneline(X509_get_subject_name(cert), 0, 0);
      DBUG_PRINT("info", ("comparing subjects: '%s' and '%s'",
                         acl_user->x509_subject, ptr));
      if (my_x509_oneline_cmp(thd, acl_user->x509_subject, ptr))

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

DBUG_PRINT is ok with null pointers.