Do not hold DeviceVarMtx across module compilation - #1485
Merged
pvelesko merged 2 commits intoAug 29, 2026
Conversation
Exercises #1393: the device compiler terminating the process from inside a module build must not leave chipStar deadlocked in its own exit handler. The exit is injected by an LD_PRELOAD interposer that fires only while Device::getOrCreateModule is on the stack, so the test does not depend on any backend compiler rejecting a module. A driver script maps the outcomes: no injected exit skips, a process that has to be killed fails, and the injected exit status passes. Fails on main by hitting the 60 s driver timeout.
pvelesko
force-pushed
the
2026-08-27-github-1393-compile-under-devicevarmtx
branch
from
August 28, 2026 12:01
9b8c8ca to
83b4ef6
Compare
getOrCreateModule() held Device::DeviceVarMtx for its whole body, including compile(), which enters the backend compiler. A compiler that terminates the process from there sends the atexit chain through deallocateDeviceVariables(), which takes DeviceVarMtx again on the same thread and deadlocks the exit. SPIRV-LLVM-Translator reaches this: SPIRVErrorLog::checkError() defaults to SPIRVDbgErrorHandlingKinds::Exit, so a rejected module calls exit() from inside clBuildProgram, and the run burns the full ctest timeout instead of failing. mutex::lock() also expects the calling thread not to already own the mutex, so the old code was undefined behaviour rather than only a hang. The map lookup and the insertion each take the lock, and compile() runs between them with the lock released. That is the shape getOrCreateModule(HostPtr) already uses one frame up, so the two overloads now match. The insertion cannot collide: every HIP entry point holds ApiMtx across the whole call, so module compilation is single-entry per device. An assert records that dependency rather than a branch handling a case no path can reach, which would have implied a thread safety the surrounding code does not have anyway, DeviceVarLookup_ being inserted outside the lock a few lines up. Fixes #1393
pvelesko
force-pushed
the
2026-08-27-github-1393-compile-under-devicevarmtx
branch
from
August 29, 2026 11:23
83b4ef6 to
a8e8402
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
getOrCreateModule() held Device::DeviceVarMtx for its whole body, including compile(), which enters the backend compiler. A compiler that terminates the process from there sends the atexit chain back through the same lock on the same thread:
SPIRV-LLVM-Translator reaches this:
SPIRVErrorLog::checkError()defaults toSPIRVDbgErrorHandlingKinds::Exit, so a rejected module callsexit()from insideclBuildProgram. The visible symptom is a ctest timeout on a test whose log ends at anInvalidModuleline and never prints its own failure message, which reads like a hung kernel or a slow JIT rather than a build rejection.std::mutex::lock()also expects the calling thread not to already own the mutex, so the old code was undefined behaviour, not only a hang.The map lookup and the insertion each take the lock and compile() runs between them with the lock released. The insertion tolerates a module another thread compiled meanwhile, which no current path can produce because every HIP entry point serializes on
ApiMtxfirst and the only thread chipStar spawns itself isEventMonitor, which never compiles.Fixes #1393
Test:
tests/runtime/TestFix1393ExitDuringModuleBuild. AnLD_PRELOADinterposer injects the exit from inside the backend build call, but only whilegetOrCreateModuleis on the stack. It hooksclBuildProgram,clCompileProgramandzeModuleCreate, so both backends and the rtdevlib compile+link path are covered, and it still fires on a warm module cache because the cache-hit paths call the same entry points. A driver script maps the outcomes: no injected exit skips, a process that has to be killed fails, and the injected exit status passes. On main the test hits the 60 s driver timeout, with this change it exits in 0.10 s.Alternative considered: #1389 makes DeviceVarMtx recursive. Both resolve the hang and both exit cleanly. This one keeps the mutex non-recursive, so re-entering any of the 15 DeviceVarMtx critical sections stays a detectable error rather than a silent one, and it takes the lock off a call into third party code that can block, throw, or end the process.
Local verification ran on the Intel CPU OpenCL runtime because this machine's GPU is wedged.
TestModuleCacheInvalidationfails on that runtime, verified identical on main, so it is unrelated to this change.