lookup: return NULL on allocation failure, keep payloads pointer-aligned

[doomedraven]

lookup_add() doesn't check what calloc() returns, and on x64 every lookup payload starts only 4-byte aligned. The LOOKUP_THREAD version of #216's scratch pool suggested there would be the macro's first user, so both are fixed here as a separate change (9 lines added, 6 removed).

Changes

1. lookup_add() returns NULL when the allocation fails. Previously it wrote through the NULL result inside lookup.c, which crashed even callers that already handle NULL.
2. lookup_get_or_create() passes that NULL through. It no longer memsets the result. The memset was redundant anyway: calloc here is cm_calloc, i.e. HeapAlloc(g_heap, HEAP_ZERO_MEMORY, ...), so the payload is already zero. Dropping it also removes a write to an entry that other threads can already see.
3. entry_t.size is now ULONG_PTR instead of unsigned int. On x64, data[] sat at offset 20 (8 + 8 + 4), so every pointer in every payload (hook_info_t, file_record_t, SOFTBP) was at 4 mod 8. x64 tolerates that for plain loads and stores, although it is still undefined behaviour in C and UBSan reports it (see Testing). The Interlocked*Pointer and 64-bit Interlocked* functions, however, require natural alignment, and LOOKUP_THREAD/LOOKUP_SHARED are meant to carry exactly that kind of state. With the change, data[] moves to offset 24 on x64 and stays at 12 on x86. sizeof(entry_t) is 24 on x64 both before and after, since those four bytes were tail padding, so no allocation grows. A C_ASSERT pins the alignment.

Callers when the allocation fails

Caller	Before	After
SetSoftwareBreakpoint, __handle_duplicate, add_force_hook_thread_func	faulted inside lookup_add despite their own NULL check	their NULL check now takes effect
add_file_to_log_tracking, file_write, add_ignored_thread, hook_clr.c JIT tracking, LOOKUP_MARK_SEEN	faulted inside lookup_add	entry is simply not added
cache_file, hook_info	faulted inside lookup_add	fault in their own code; no change in effect

I left the last two alone. Deciding what hook_info() should return when it has nothing is a design decision, not a hardening fix.

Deliberately not in this PR

• LOOKUP_SHARED first-use race. lookup_get_or_create() does the lookup and the add as two separate steps. For a shared key, two threads arriving first can each add an entry, and the later one hides the earlier. A thread-id key cannot hit this, and nothing uses LOOKUP_SHARED yet, so the fix (insert-if-absent under a CAS) can wait until something does.
• file_handle_terminate() (hook_file.c:333) reads past g_files. The loop starts from (entry_t *)&g_files.root, a fake list head, and reads p->id from it. That field lies beyond the one-pointer lookup_t, so it reads the adjacent global. This change doesn't move next or id, so the behaviour is the same before and after. It is noted here only so it isn't lost.
• TSan reports lookup_get()'s plain loads racing lookup_add()'s CAS publish. This is the long-standing lock-free pattern. It is fine on x86/x64 because nothing is ever freed, and it is unchanged here.

Testing

Not compiled with MSVC and not run in a detonation. There is no MSVC here, and the MSBuild workflow is disabled_manually (see #212).

Instead, lookup.c/lookup.h were compiled verbatim against a small Win32 shim and checked as follows:

• gcc -Wall -Wextra -Werror builds cleanly for both -m64 and -m32. The C_ASSERT holds on both.
• With the old header, the same C_ASSERT fails the build as intended (size of array '__C_ASSERT__' is negative).
• With allocation failure injected, lookup_add, lookup_get_or_create and LOOKUP_THREAD return NULL and the table is left untouched. LOOKUP_MARK_SEEN does not fault.
• On the normal path, the payload comes back zeroed, the size round-trips through lookup_get, and a second get_or_create returns the same entry.
• Layout: offsetof(entry_t, data) is 24 on LP64 (was 20) and 12 on ILP32 (unchanged). sizeof(entry_t) is unchanged on both.
• The harness for the LOOKUP_THREAD version of Stop allocating 64 KB per path on the hooked call paths #216's pool was run against both headers: 300 threads over 32 recycled thread ids, ASan+UBSan, default sanitizer options.
  • Old header: UBSan reports member access within misaligned address ... for type 'struct path_scratch_t', which requires 8 byte alignment at 10 distinct source locations. LeakSanitizer reports all 136 retained 64 KB slots as leaked, because it only scans pointer-aligned words and every payload pointer was at 4 mod 8.
  • New header: no UBSan reports, no leak reports.

Are these related?

No. This change is self-contained and can be merged, reverted or deferred on its own.

#216 benefits from it if it moves to LOOKUP_THREAD: allocation failure on a thread's first path buffer then falls back to the heap instead of faulting in lookup_add. #216 does not require it, and neither PR changes the other's files.

Contrast with the NDEBUG case in #211, which genuinely is coupled: the assert() removals there are only correct because the same PR defines NDEBUG.

Textual conflicts: none expected. No other PR in the series touches lookup.c or lookup.h.

Series

#206, #207, #208, #209, #210, #211, #212, #213, #214, #215, #164, #216, this one.