Fix memory-safety defects in the API hook handlers

[doomedraven]

What this is

Second PR in the review series. #206 was the one-to-three-line logic defects; this one is memory safety in the API hook handlers.

Every defect here is reachable from a single API call made by the monitored process. The handler runs inside that process, so a sample that knows capemon is present can pick the argument. None of these require a chain.

Are these related? (the question #206 raised)

These ten are independent of each other. Each fix is local to one handler, each can be reverted alone, and no two share a data structure. They are in one PR because they are the same class of defect and one regression run covers all of them — not because they are coupled.

That matters because the next PR in the series is the opposite case. The NDEBUG fix (Release configs never define it, so assert() is live in shipped builds) cannot be committed on its own:

• alloc.c:85 creates the guard page inside assert(VirtualProtect(...)). Define NDEBUG and the guard page silently stops being created.
• hooking_32.c:267, :378 and hooking_64.c:542, :826 use assert((p - pre_tramp) < MAX_PRETRAMP_SIZE) as the only overflow check on the 320-byte pre_tramp buffer. The x64 notail+stack stub already emits roughly 280 bytes.

So "define NDEBUG" and "convert those asserts into real runtime checks" have to land in the same commit or the build regresses. That PR will say so explicitly. This one does not have that property.

The fixes

File	Function	Mechanism
hook_misc.c	vDbgPrintExWithPrefixInternal	strcpy of caller Prefix into UCHAR Buffer[512]; sizeof(Buffer) - cb then underflows and _vsnprintf treats it as unlimited
hook_file.c	NtSetInformationFile	Rename copy bounded only by the caller's FileNameLength — not by Length, not by the destination
hook_reg.c	RegEnumKeyW, RegEnumKeyExA/W	wcscpy_s(lpName, sizeof(lpName), ...) — sizeof of a pointer
hook_tls.c	4 sites	HexEncode writes 2*Length+1 into 65- and 97-byte stack buffers with a caller-supplied Length
hook_socket.c	alloc_combined_wsabuf	DWORD sum of WSABUF lengths wraps → undersized malloc, full-size memcpy
hook_crypto.c	CryptHashMessage	Identical overflow over rgcbToBeHashed[]
hook_special.c	CDocument_write	Identical overflow over BSTR lengths, plus wcslen re-measured between the sizing and copy passes, plus buf leaked every call
hook_process.c	NtCreateUserProcess	Loop bound underflows; ValuePtr written through unchecked
hook_services.c	servicename_from_handle	Byte count reused as a character capacity (2x overstatement); 8 KB leak per call
hook_network.c	get_ip_list	Inverted guard; write offset never advanced

Worth calling out individually

hook_reg.c is a live crash, not a theoretical one. sizeof(lpName) is 4 on x86. wcscpy_s with a destination size of 4 and a 7-character source invokes the _s invalid-parameter handler, which terminates the process. Every substitution in the parent_keys/replace_subkeys table triggers it. On x86 this means any sample that enumerates those keys kills the monitored process and the analysis produces nothing.

The Ex variants needed care: lpcName is in/out. On return it holds the length written, not the capacity, so reading it after Old_RegEnumKeyEx* gives the wrong number. The capacity is captured into a local before the call.

hook_network.c get_ip_list never worked. The guard is if (!server_list || server_list->AddrCount) return NULL; — so the body executes only when AddrCount == 0, which then calloc(1, 0)s and returns a zero-byte allocation that the log layer reads as a C string. With the guard corrected the loop also needed the offset fix, since _snprintf_s was always writing at offset 0 and each address overwrote the last.

hook_tls.c: HexEncode is left alone. It is non-static and may be referenced elsewhere, and there is no local MSVC here to catch a signature break. A static HexEncodeBounded wrapper was added instead and only the call sites with caller-controlled lengths were switched. The sites passing literal 32 or 48 are untouched.

Overflow guards: the three summation fixes cap at 64 MB (hook_socket.c, hook_crypto.c) and 32 MB (hook_special.c). A buffer past that is not going to be useful in a behaviour log anyway, and the cap makes the truncation explicit rather than implicit in a wrapped DWORD. If you would rather log them, say so and I will raise or drop the cap.

Behaviour impact

Well-formed input is unaffected. Three observable differences:

1. Oversized or malformed inputs are skipped or truncated instead of corrupting memory.
2. get_ip_list starts emitting the list it was always supposed to emit — DnsQuery_A/DnsQuery_W records will gain a ServerList value that was previously always absent.
3. On x86, the hook_reg.c substitutions stop killing the process.

Not verified by build

No MSVC on this machine. CI on #206 has not run either — gh pr checks 206 reports no checks on the branch. Two candidates: fork PRs need maintainer approval before workflows run, and .github/workflows/msbuild.yml targets runs-on: windows-2019, which GitHub has retired. If the runner is the problem I can send a prerequisite PR bumping it to windows-2022 and actions/checkout@v4 / upload-artifact@v4.

Draft until it compiles and you have had a chance to run it.

Series

• Fix logic defects that silently disable checks or corrupt output #206 — logic defects (one-to-three lines each)
• this — memory safety, hook handlers
• next — memory safety, core (log.c, misc.c, config.c, capemon.c, YaraHarness.c)
• then — log-record correctness (changes behaviour-log output, needs CAPEv2-side coordination)
• then — leaks and unchecked allocations
• then — hot-path allocations (TLS scratch buffers)
• then — hot-path algorithms (inside_hook ranges, lookup buckets, single backtrace walk)
• then — build configuration (NDEBUG + assert conversion + /O2 /Oy-, the coupled one)
• then — thread safety (hook_api lock, arena free list, breakpoint lists)

[doomedraven]

Correction to the CI note in the description above: the diagnosis there was wrong.

The MSBuild workflow is disabled_manually at the repository level, so it cannot run regardless of what the PR contains:

$ gh api repos/kevoreilly/capemon/actions/workflows
28093452  MSBuild  .github/workflows/msbuild.yml  disabled_manually

On top of that it targets the retired windows-2019 image (jobs queue for 24h and get cancelled by the timeout — visible in every PR Build Test run), and all three projects declare PlatformToolset=v141, which is not installed on windows-2022 or windows-2025.

#212 fixes the workflow file for all three. Re-enabling the workflow in Settings → Actions is the part that has to be done by hand.