fix(math-cuda): driver-independent cuda builds (cudarc pin + cubin AOT)#800
Draft
MauroToscano wants to merge 2 commits into
Draft
fix(math-cuda): driver-independent cuda builds (cudarc pin + cubin AOT)#800MauroToscano wants to merge 2 commits into
MauroToscano wants to merge 2 commits into
Conversation
Two driver-independence fixes so `--features cuda` builds AND runs on the team's GPU hardware without CUDARC_CUDA_VERSION env or a hand-picked toolkit. 1. cudarc symbol floor. Pin cudarc's CUDA version to `cuda-12080` in crypto/math-cuda/Cargo.toml (was `cuda-version-from-build-system` + `fallback-latest`). Auto-detect bound the newest symbol set the build toolkit knew (e.g. CUDA 13.1 -> cuDevSmResourceSplit, gated behind cuda-13010/13020), which cudarc eagerly resolves at init and panics on a driver that predates it (580.x = CUDA 13.0 max). Our cudarc surface is all CUDA-11-era, so the 12.8 symbol set (a strict subset every >=12.8 driver exports) resolves cleanly everywhere. Replaces the per-script sed pin; drops it from scripts/gpu_test.sh and adjusts the compat shim in scripts/bench_abba.sh (still rewrites pre-pin baseline shas). 2. PTX ISA version. AOT-compile kernels to native cubin (SASS) via `nvcc --cubin -arch=sm_XX` instead of `--ptx`, loaded through `Ptx::from_binary` (cuModuleLoadData). A cubin carries pre-compiled SASS for a real arch, so the driver loads it regardless of the toolkit's PTX ISA version -- no CUDA_ERROR_UNSUPPORTED_PTX_VERSION and no silent CPU fallback when the toolkit is newer than the driver. Build+run are co-located and the arch is detected from nvidia-smi, so the arch always matches; a too-old toolkit now fails loudly at nvcc build time. CPU-only and nvcc-less stub builds unaffected (empty cubin stub). README "GPU Tests" updated.
… no-op - backend(): warn once when GPU init fails, so a cubin arch mismatch (AOT cubins are arch-specific) doesn't silently disable the GPU with no signal. Stale ptx->cubin comment fixed. - bench_abba.sh: warn when CUDARC_PIN can't apply (post-pin sha, anchor gone) instead of silently no-opping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 3 of 3, stacked on #799. Review this diff against
feat/gpu-trace-residency.What
Make
--features cudabuild and run across GPU driver versions with no environment juggling (previously neededCUDARC_CUDA_VERSION+ a hand-picked toolkit). Two independent fixes:cuda-12080inCargo.toml. Auto-detect bound the newest symbol set the build toolkit knew (a 13.1 toolkit pulls incuDevSmResourceSplit), which cudarc eagerly resolves at init and panics on a driver that predates it (e.g. 580.x = CUDA 13.0 max). Our cudarc surface is all CUDA-11-era, so the 12.8 subset resolves on every ≥12.8 driver.nvcc --cubin -arch=sm_XX), loaded viaPtx::from_binary. A cubin carries pre-compiled SASS for a real arch, so the driver loads it regardless of the toolkit's PTX ISA version — noUNSUPPORTED_PTX_VERSION, no silent CPU fallback when the toolkit is newer than the driver.This also replaces the per-script
CUDARC_PINsed hack.Note for review
nvidia-smi; a mismatch now fails loudly at nvcc time, andbackend()warns once if a cubin won't load rather than silently disabling the GPU).