Skip to content

Do not hold DeviceVarMtx across module compilation - #1485

Merged
pvelesko merged 2 commits into
mainfrom
2026-08-27-github-1393-compile-under-devicevarmtx
Aug 29, 2026
Merged

Do not hold DeviceVarMtx across module compilation#1485
pvelesko merged 2 commits into
mainfrom
2026-08-27-github-1393-compile-under-devicevarmtx

Conversation

@pvelesko

Copy link
Copy Markdown
Collaborator

getOrCreateModule() held Device::DeviceVarMtx for its whole body, including compile(), which enters the backend compiler. A compiler that terminates the process from there sends the atexit chain back through the same lock on the same thread:

#0  deallocateDeviceVariables        src/CHIPBackend.cc:1314   inner LOCK, blocks forever
#1  CHIPUninitializeCallOnce ()      src/CHIPDriver.cc:244
#5  CHIPUninitialize ()              src/CHIPDriver.cc:254
#6  __hipUnregisterFatBinary (...)   src/CHIPBindings.cc:6631
#7  __hip_module_dtor ()
#8  __run_exit_handlers (status=15)  ./stdlib/exit.c:108
#9  __GI_exit (status=...)           ./stdlib/exit.c:138
#10 clBuildProgram (...)                                       exit() from inside the build
#13 CHIPModuleOpenCL::compile (...)  src/backend/OpenCL/CHIPBackendOpenCL.cc:1219
#15 getOrCreateModule (...)          src/CHIPBackend.cc:1477   outer LOCK, still held

SPIRV-LLVM-Translator reaches this: SPIRVErrorLog::checkError() defaults to SPIRVDbgErrorHandlingKinds::Exit, so a rejected module calls exit() from inside clBuildProgram. The visible symptom is a ctest timeout on a test whose log ends at an InvalidModule line and never prints its own failure message, which reads like a hung kernel or a slow JIT rather than a build rejection.

std::mutex::lock() also expects the calling thread not to already own the mutex, so the old code was undefined behaviour, not only a hang.

The map lookup and the insertion each take the lock and compile() runs between them with the lock released. The insertion tolerates a module another thread compiled meanwhile, which no current path can produce because every HIP entry point serializes on ApiMtx first and the only thread chipStar spawns itself is EventMonitor, which never compiles.

Fixes #1393

Test: tests/runtime/TestFix1393ExitDuringModuleBuild. An LD_PRELOAD interposer injects the exit from inside the backend build call, but only while getOrCreateModule is on the stack. It hooks clBuildProgram, clCompileProgram and zeModuleCreate, so both backends and the rtdevlib compile+link path are covered, and it still fires on a warm module cache because the cache-hit paths call the same entry points. A driver script maps the outcomes: no injected exit skips, a process that has to be killed fails, and the injected exit status passes. On main the test hits the 60 s driver timeout, with this change it exits in 0.10 s.

Alternative considered: #1389 makes DeviceVarMtx recursive. Both resolve the hang and both exit cleanly. This one keeps the mutex non-recursive, so re-entering any of the 15 DeviceVarMtx critical sections stays a detectable error rather than a silent one, and it takes the lock off a call into third party code that can block, throw, or end the process.

Local verification ran on the Intel CPU OpenCL runtime because this machine's GPU is wedged. TestModuleCacheInvalidation fails on that runtime, verified identical on main, so it is unrelated to this change.

Exercises #1393: the device compiler terminating the process from inside a
module build must not leave chipStar deadlocked in its own exit handler.

The exit is injected by an LD_PRELOAD interposer that fires only while
Device::getOrCreateModule is on the stack, so the test does not depend on any
backend compiler rejecting a module. A driver script maps the outcomes: no
injected exit skips, a process that has to be killed fails, and the injected
exit status passes.

Fails on main by hitting the 60 s driver timeout.
@pvelesko
pvelesko force-pushed the 2026-08-27-github-1393-compile-under-devicevarmtx branch from 9b8c8ca to 83b4ef6 Compare August 28, 2026 12:01
getOrCreateModule() held Device::DeviceVarMtx for its whole body, including
compile(), which enters the backend compiler. A compiler that terminates the
process from there sends the atexit chain through deallocateDeviceVariables(),
which takes DeviceVarMtx again on the same thread and deadlocks the exit.
SPIRV-LLVM-Translator reaches this: SPIRVErrorLog::checkError() defaults to
SPIRVDbgErrorHandlingKinds::Exit, so a rejected module calls exit() from inside
clBuildProgram, and the run burns the full ctest timeout instead of failing.
mutex::lock() also expects the calling thread not to already own the mutex, so
the old code was undefined behaviour rather than only a hang.

The map lookup and the insertion each take the lock, and compile() runs between
them with the lock released. That is the shape getOrCreateModule(HostPtr)
already uses one frame up, so the two overloads now match.

The insertion cannot collide: every HIP entry point holds ApiMtx across the
whole call, so module compilation is single-entry per device. An assert records
that dependency rather than a branch handling a case no path can reach, which
would have implied a thread safety the surrounding code does not have anyway,
DeviceVarLookup_ being inserted outside the lock a few lines up.

Fixes #1393
@pvelesko
pvelesko force-pushed the 2026-08-27-github-1393-compile-under-devicevarmtx branch from 83b4ef6 to a8e8402 Compare August 29, 2026 11:23
@pvelesko
pvelesko merged commit 1ae3797 into main Aug 29, 2026
16 checks passed
@pvelesko
pvelesko deleted the 2026-08-27-github-1393-compile-under-devicevarmtx branch August 29, 2026 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runtime deadlocks in its own exit handler when the device compiler calls exit() during a module build

1 participant