Size the pre-master secret buffer by the key exchanges, and fix a DHE-PSK overflow

[Frauschi]

One of six independent branches that cut allocations and per-object memory in the TLS layer. This one also carries a memory-safety fix.

The fix

A DHE-PSK exchange stores a two byte length, the DH shared secret, another two byte length and the PSK in the pre-master secret buffer. Sized with ENCRYPT_LEN, that is two bytes short for the largest DH parameters the build accepts together with a maximum length PSK, and the last two bytes of the key were written past the allocation on both peers. It is reached before authentication completes, from parameters the peer supplies.

test_dhe_psk_max_premaster() drives the handshake that produces the largest secret: 4096 bit parameters on the server and a MAX_PSK_KEY_LEN PSK. The client asks for an ECC named group so FFDHE negotiation does not substitute a smaller group, and the negotiated key size is checked to confirm it did not. Against the old sizing the assertion fails and a sanitizer build reports a heap buffer overflow in MakePSKPreMasterSecret().

What else changes

The buffer is part of the Arrays struct. It used to be its own allocation - made eagerly for TLS 1.3 and the sniffer, lazily elsewhere, and freed by CleanPreMaster() or FreeArrays() - for a buffer whose lifetime is exactly that of the Arrays it belongs to. It is now a member, so the two are created, wiped and released together, and nothing tracks or frees it separately.

It is sized by what the build can actually produce. ENCRYPT_LEN also has to cover an RSA signature, so it follows the largest RSA key the build supports: a build with RSA 4096 certificates reserved 512 bytes for a secret ECDHE fills 32 to 98 bytes of. MAX_PREMASTER_SZ takes the largest secret any key exchange in the build can produce - the RSA pre-master, the DH parameters a peer may send, an ECDHE secret plus the 32 byte ML-KEM half of a hybrid key share, the length prefixes and key the PSK suites append, the TLS 1.3 handshake secret that reuses the buffer - with the master secret length as a floor.

A build with DH is unchanged, since the peer chooses the parameter size. A build without it, the usual TLS 1.3 configuration, holds 98 rather than 512 bytes per handshake. A build with both DH and PSK gains two bytes, which is the overflow above.

WOLFSSL_MAX_GEN_PREMASTER_SZ lets an integration that knows what its GenPreMasterCb writes opt down; it defaults to ENCRYPT_LEN so no existing callback can be overrun, and the capacity handed to the callback is unchanged.

Behaviour changes worth flagging

wolfSSL_[CTX_]SetMaxDhKey_Sz() now returns BAD_FUNC_ARG for a ceiling above what the build's math supports, rather than accepting it and silently reducing it. This is load-bearing: without it an application could raise maxDhKeySz above the buffer size and reopen the overflow. An application that defensively calls wolfSSL_CTX_SetMaxDhKey_Sz(ctx, 4096) on a build whose math tops out lower now gets an error where it previously got success. The check sits after the crypto-policy block so a request that violates both is still reported as the policy violation.

The minimum setters are deliberately unchanged: a floor above what the build supports is a tightening the caller asked for, it simply refuses every key a peer can offer, and it cannot overrun anything.

wolfSSL_dtls_import() treats the serialized DH sizes the same way: the exported ceiling is clamped, a negotiated size above what this build supports fails the import, and the floor carries over as-is, matching the minimum setters.

Memory impact

Measured on 64-bit builds. Arrays is one allocation per handshake; on master the pre-master secret was a second one beside it.

TLS 1.3 without DH or PSK (--enable-tls13 --disable-dh --disable-psk), the usual TLS 1.3 configuration:

	master	this branch
sizeof(Arrays)	208	300
separate pre-master buffer	512	none
total per handshake	720 bytes in 2 allocations	300 bytes in 1

That is 420 bytes and one allocation/free pair saved per handshake, because MAX_PREMASTER_SZ comes out at 98 where ENCRYPT_LEN was 512.

Default build (DH enabled, so the peer chooses the parameter size and the buffer cannot shrink):

	master	this branch
total per handshake	720 bytes in 2 allocations	712 bytes in 1

No meaningful size change, but still one fewer allocation and free per handshake, and the buffer is now wiped together with the arrays rather than needing its own CleanPreMaster() path.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m3

• FLASH: .text -80 B (-0.1%, 127,845 B / 262,144 B, total: 49% used)

gcc-arm-cortex-m4-dtls13

• FLASH: .text -64 B (-0.0%, 190,916 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text -128 B (-0.0%, 788,892 B / 1,048,576 B, total: 75% used)

gcc-arm-cortex-m4-pq

• FLASH: .text -64 B (-0.0%, 306,864 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text -128 B (-0.0%, 337,008 B / 1,048,576 B, total: 32% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text -64 B (-0.0%, 128,637 B / 262,144 B, total: 49% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text -128 B (-0.1%, 245,790 B / 262,144 B, total: 94% used)

gcc-arm-cortex-m7

• FLASH: .text -64 B (-0.0%, 206,572 B / 262,144 B, total: 79% used)

gcc-arm-cortex-m7-pq

• FLASH: .text -128 B (-0.0%, 307,760 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text -64 B (-0.0%, 245,854 B / 262,144 B, total: 94% used)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-sp-math
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[Frauschi]

Jenkins retest this please - history lost.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11345

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11345

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[Frauschi]

Jenkins retest this please

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11345

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11345

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Fenrir's latest completed scan found no issues; clearing the prior automated change request.

Fenrir's latest completed scan found no issues; clearing the prior automated change request.

[Frauschi]

Jenkins retest this please - flaky test

[Copilot]

Copilot review overview

🟡 Changes recommended

Embedded storage causes a use-after-free with async Intel QAT, and the DH import-floor behavior contradicts the stated contract.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• QAT DH agreement frees embedded caller-owned buffer · New
• Reject imported DH floor exceeding MAX_DHKEY_SZ · New

What changed in this PR

Resizes and embeds the TLS pre-master secret buffer to fix a DHE-PSK overflow while reducing handshake allocations.

Changes:

• Derives buffer capacity from enabled key exchanges.
• Embeds and consistently wipes the buffer with Arrays.
• Adds boundary, import-policy, and regression tests.

File	Description
wolfssl/​internal.h	Defines sizing and embeds the buffer.
src/​internal.c	Updates allocation, DH/PSK handling, and imports.
src/​keys.c	Updates derivation buffers and cleanup.
src/​ssl.c	Resets embedded secret storage.
src/​ssl_api_dtls.c	Bounds and clears multicast secrets.
src/​ssl_api_pk.c	Enforces supported DH ceilings.
src/​sniffer.c	Updates capacities and secret cleanup.
src/​tls.c	Updates key-share bounds.
src/​tls13.c	Fully wipes TLS 1.3 secrets.
tests/​api.c	Updates secret-zeroization assertions.
tests/​api/​test_dtls.c	Tests import and secret boundaries.
tests/​api/​test_dtls.h	Registers the DTLS test.
tests/​api/​test_ssl_pk.c	Tests DH ceiling validation.
tests/​api/​test_tls.c	Adds the DHE-PSK overflow regression.
tests/​api/​test_tls.h	Registers the TLS test.
tests/​api/​test_tls13.c	Updates manual array cleanup.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.