Define NDEBUG in Release and build it optimized

[doomedraven]

This is the build-configuration PR in the review series. Two commits, and
they are ordered deliberately.

What is broken

1. NDEBUG is never defined in Release.

capemon.vcxproj line 150, Release|Win32:

WIN32;_CRT_SECURE_NO_WARNINGS;_CRT_SECURE_NO_WARNINGSNDEBUG;_WINDOWS;...

A missing ; fused _CRT_SECURE_NO_WARNINGS and NDEBUG into one macro
named _CRT_SECURE_NO_WARNINGSNDEBUG. Line 186, Release|x64, never had
NDEBUG in the list at all.

So every assert() in capemon is live in the shipped Release DLL. A failed
assert calls _wassert(), which writes to stderr and calls abort() —
inside the process being analysed, from inside a hook.

2. Release is built with /Od.

Neither Release ItemDefinitionGroup sets <Optimization>, so cl.exe
uses its default, which is /Od. Both Debug groups set
<Optimization>Disabled explicitly, so the Release omission reads as an
oversight. capemon is in the call path of every hooked API in the analysed
process.

Are these related?

Yes, and that coupling is the reason this PR exists as its own unit rather
than being folded into the one-line-defects PR.

You cannot just add the missing semicolon. Three asserts in the tree are
load-bearing — defining NDEBUG deletes behaviour, not just checks:

Site	What NDEBUG deletes
alloc.c:85	assert(VirtualProtect(..., PAGE_NOACCESS, ...)) — the guard page is created inside the assert argument. Whole expression vanishes; the allocator silently stops emitting guard pages.
hooking_32.c:267,378, hooking_64.c:542,826	assert((ULONG_PTR)(p - h->hookdata->pre_tramp) < MAX_PRETRAMP_SIZE) — the only overflow check on a fixed 320-byte buffer. Vanishes; an over-long trampoline overruns hook_data_t silently.
hooking_32.c:597,606, hooking_64.c:135,144	assert(0); return 0; in get_near_rel_target/get_short_rel_target. The caller uses the result as a jump target, so an unhandled opcode silently retargets the trampoline to address 0.

Commit 1 therefore removes the dependency first, then defines NDEBUG
in the same commit. Fixing the typo without the assert work would be a
regression; doing the assert work without the typo fix would be dead
effort. They have to land together.

Contrast this with the other PRs in the series (#206, #207, #208, #210):
those are batches of mutually independent one-site fixes, where each hunk
stands on its own and any subset can be dropped. This one does not
decompose.

Commit 1 — make the load-bearing asserts real, then define NDEBUG

• alloc.c:85 — the VirtualProtect call is now unconditional. The result
  is deliberately not logged: this is the allocator the logger runs on, and
  calling DebugOutput from inside cm_alloc is a reentrancy hazard.
  Honest caveat: this code is currently unreachable. alloc.h:29
  defines USE_PRIVATE_HEAP and this cm_alloc is in the #else branch.
  It is a latent defect, not a live one — worth fixing because someone will
  eventually flip that define, not because it is biting today.

• The four pre_tramp bound asserts are replaced with C_ASSERT on the sum
  of the sizeofs of the byte arrays. Every byte those builders write comes
  from memcpy(p, pre_trampN, sizeof(pre_trampN)) out of function-local
  fixed-size arrays — emit_rel patches inside the arrays, not into
  pre_tramp — so the bound is a compile-time property. That makes the
  C_ASSERT exact, strictly stronger than the runtime check it replaces,
  and immune to NDEBUG.

  • Each C_ASSERT is wrapped in its own { } because it expands to
    typedef char __C_ASSERT__[...], and two in one scope is a typedef
    redefinition error.
  • hook_create_pre_tramp_notail on x64 branches on h->numargs > 4, so
    it gets one bound per branch. The larger branch currently uses 307 of
    the 320 bytes available, which is worth knowing before anyone adds
    instructions to that stub.

• The four assert(0) sites become a DebugOutput of the offending opcode
  and address. return 0 is unchanged — this does not alter control flow,
  it makes an already-silent failure visible in the log rather than only in
  a debug build.

• The remaining asserts (alloc.c:101,105,116) are pure predicates with no
  side effects, so NDEBUG only removes the check. Left alone.

Commit 2 — build Release with /O2

Split out so it can be dropped independently. Commit 1 is a correctness
fix; this one is a performance change and carries codegen risk.

/O2 is a bundle: /Og /Oi /Ot /Oy /Ob2 /GF /Gy. Two of those conflict
with assumptions in the hooking engine, so they are explicitly turned back
off:

/Gy (function-level linking) — disabled on both platforms.
hooking.c:202-211 declares

extern void start_transparent_hooks();
extern void end_transparent_hooks();

and addr_in_our_dll_range() exempts every address in
[&start_transparent_hooks, &end_transparent_hooks) from recursion
detection. Both are one-line functions in hook_special.c bracketing a run
of HOOKDEFs, so the range is a source-order-equals-address-order
assumption. /Gy gives every function its own COMDAT, and /OPT:REF +
/OPT:ICF are already enabled on both Release links — the linker would
then be free to reorder or fold functions inside that range and the
transparent hooks would stop being transparent.
<FunctionLevelLinking>false</> keeps one code section per object file.

/Oy (frame pointer omission) — disabled on Win32 only.
hooking_32.c:895-942 operate_on_backtrace walks the EBP chain, and
capemon's own frames must show up in that walk for addr_in_our_dll_range
to detect recursion. x64 does not use EBP chains, so the setting is Win32
only.

Note that /GL is already on via <WholeProgramOptimization>true</> in
both Release property groups, so LTCG can already move code across object
files. /O2 /Gy- does not make the source-order assumption weaker than it
already is. The assumption is fragile either way — a marker section or
capturing the addresses explicitly would be a better long-term fix, but
that is a separate change.

Testing

Not compiled. There is no MSVC on the machine this was written on, so
these two commits have been verified by inspection only. CI is the only
compile gate, and see the note below.

The C_ASSERTs are the one part that will fail loudly and immediately if
the reasoning above is wrong: a bad bound is a compile error, not a runtime
surprise. Hand-summing the arrays gives 36 / 73 bytes (x86) and 133 / 307 /
205 bytes (x64) against MAX_PRETRAMP_SIZE 320, so they should pass.

Behaviour worth watching after NDEBUG lands: anything that was previously
crashing the analysed process via _wassert will now continue silently, so
if a sample's behaviour changes shape, check the debugger log for the two
new get_*_rel_target messages.

CI

No checks have run on any PR in this series. Two candidates:

1. These are fork PRs from a non-collaborator, so workflows need maintainer
   approval before they will schedule.
2. .github/workflows/msbuild.yml uses runs-on: windows-2019, which
   GitHub has retired. Even once approved it may fail to schedule.

If (2) is the problem, a prerequisite PR bumping windows-2019 →
windows-2022 and actions/checkout@v3 / upload-artifact@v3 → @v4
would fix it. Say the word and I will open it.

Series

• Fix logic defects that silently disable checks or corrupt output #206 — one-to-three-line logic defects
• Fix memory-safety defects in the API hook handlers #207 — memory safety, hook handlers
• Bound the fixed-size buffers in the core #208 — memory safety, core
• Report what the API actually produced (changes behaviour-log output) #209 — log-record correctness (changes behaviour-log output)
• Free what is allocated, restore what is saved #210 — leaks, wrong frees, last-error discipline
• this PR — build configuration
• next: hot-path allocations, hot-path algorithms, thread safety

[doomedraven]

Correction to the CI note in the description above: the diagnosis there was wrong.

The MSBuild workflow is disabled_manually at the repository level, so it cannot run regardless of what the PR contains:

$ gh api repos/kevoreilly/capemon/actions/workflows
28093452  MSBuild  .github/workflows/msbuild.yml  disabled_manually

On top of that it targets the retired windows-2019 image (jobs queue for 24h and get cancelled by the timeout — visible in every PR Build Test run), and all three projects declare PlatformToolset=v141, which is not installed on windows-2022 or windows-2025.

#212 fixes the workflow file for all three. Re-enabling the workflow in Settings → Actions is the part that has to be done by hand.