From 855467fa830376a411ed65f717c5a72fea6706e6 Mon Sep 17 00:00:00 2001 From: MauroFab Date: Wed, 8 Jul 2026 18:37:43 -0300 Subject: [PATCH 1/4] fix(math-cuda): robust CUDA version handling for cuda builds Two driver-independence fixes so `--features cuda` builds AND runs on the team's GPU hardware without CUDARC_CUDA_VERSION env or a hand-picked toolkit. 1. cudarc symbol floor. Pin cudarc's CUDA version to `cuda-12080` in crypto/math-cuda/Cargo.toml (was `cuda-version-from-build-system` + `fallback-latest`). Auto-detect bound the newest symbol set the build toolkit knew (e.g. CUDA 13.1 -> cuDevSmResourceSplit, gated behind cuda-13010/13020), which cudarc eagerly resolves at init and panics on a driver that predates it (580.x = CUDA 13.0 max). Our cudarc surface is all CUDA-11-era, so the 12.8 symbol set (a strict subset every >=12.8 driver exports) resolves cleanly everywhere. Replaces the per-script sed pin; drops it from scripts/gpu_test.sh and adjusts the compat shim in scripts/bench_abba.sh (still rewrites pre-pin baseline shas). 2. PTX ISA version. AOT-compile kernels to native cubin (SASS) via `nvcc --cubin -arch=sm_XX` instead of `--ptx`, loaded through `Ptx::from_binary` (cuModuleLoadData). A cubin carries pre-compiled SASS for a real arch, so the driver loads it regardless of the toolkit's PTX ISA version -- no CUDA_ERROR_UNSUPPORTED_PTX_VERSION and no silent CPU fallback when the toolkit is newer than the driver. Build+run are co-located and the arch is detected from nvidia-smi, so the arch always matches; a too-old toolkit now fails loudly at nvcc build time. CPU-only and nvcc-less stub builds unaffected (empty cubin stub). README "GPU Tests" updated. --- .github/workflows/benchmark-gpu.yml | 15 +++-- .github/workflows/gpu-tests.yml | 3 +- README.md | 13 +++-- crypto/math-cuda/Cargo.toml | 18 +++++- crypto/math-cuda/build.rs | 89 ++++++++++++++++++----------- crypto/math-cuda/src/device.rs | 58 ++++++++++--------- scripts/bench_abba.sh | 9 ++- scripts/gpu_test.sh | 25 +------- 8 files changed, 131 insertions(+), 99 deletions(-) diff --git a/.github/workflows/benchmark-gpu.yml b/.github/workflows/benchmark-gpu.yml index be6a67e90..6928255d9 100644 --- a/.github/workflows/benchmark-gpu.yml +++ b/.github/workflows/benchmark-gpu.yml @@ -358,14 +358,13 @@ jobs: # the build through the CUDA prover path. NOTE: requires this PR's bench_abba.sh change # (the BENCH_FEATURES env) to be on main — i.e. it only takes effect after merge. # REBUILD=1: each Vast box is fresh, GPU-specific hardware — always rebuild both - # binaries (PTX is compiled for the detected arch); never trust a cached binary. - # CUDARC_PIN: pin cudarc to a fixed CUDA version (cuda-12080 = CUDA 12.8, matching the - # cuda_max_good>=12.8 offer floor) and drop fallback-latest, so cudarc binds a known - # symbol set instead of its newest. With fallback-latest cudarc requested a symbol the - # box's driver doesn't export (e.g. cuDevSmResourceSplit) -> runtime panic. This is the - # too-new end of the same compatibility window that MIN_DRIVER>=580 guards at the - # too-old end (older drivers lack cuCtxGetDevice_v2 and the GPU path falls back to CPU). - # nvidia-smi is logged for diagnosing driver issues. + # binaries (cubin is compiled for the detected arch); never trust a cached binary. + # CUDARC_PIN: compat shim for pre-pin baseline shas. cudarc's CUDA version is now pinned + # permanently in crypto/math-cuda/Cargo.toml (cuda-12080), so this no-ops on shas that + # carry the pin and only rewrites older baselines (where fallback-latest could request a + # symbol the box's driver doesn't export, e.g. cuDevSmResourceSplit -> runtime panic). + # MIN_DRIVER>=580 still guards the too-old end (older drivers lack cuCtxGetDevice_v2 and + # the GPU path falls back to CPU). nvidia-smi is logged for diagnosing driver issues. REMOTE="set -e; cd /workspace/lambda_vm; \ command -v python3 >/dev/null || { apt-get update -qq && apt-get install -y -qq python3; }; \ nvidia-smi || true; \ diff --git a/.github/workflows/gpu-tests.yml b/.github/workflows/gpu-tests.yml index 1a1f9a2b1..ddcce0ee3 100644 --- a/.github/workflows/gpu-tests.yml +++ b/.github/workflows/gpu-tests.yml @@ -287,7 +287,8 @@ jobs: ''|*[!A-Za-z0-9._/-]*) echo "::error::invalid ref: '$REF'"; exit 1 ;; esac # Check out the ref under test on the box, then run the CUDA test groups. - # gpu_test.sh owns the CUDARC_PIN / SYSROOT_DIR defaults — don't duplicate them here. + # gpu_test.sh owns the SYSROOT_DIR default — don't duplicate it here. (cudarc's CUDA + # version is pinned in crypto/math-cuda/Cargo.toml, so no CUDARC_PIN is needed.) REMOTE="set -e; cd /workspace/lambda_vm; \ git fetch --force origin '$REF'; \ git checkout -f FETCH_HEAD; \ diff --git a/README.md b/README.md index 2a9e3ed6e..b027da040 100644 --- a/README.md +++ b/README.md @@ -233,10 +233,15 @@ The CUDA test groups run only on a machine with an NVIDIA GPU and `nvcc`: - `make test-prover-cuda` — the prover/stark/crypto/ecsm suite with the GPU path enabled - `make test-prover-comprehensive-cuda` — the comprehensive all-instructions prove on the GPU path -The kernels are compiled by `nvcc` into PTX that the driver JIT-compiles at load, so the GPU's -driver must be new enough for the toolkit — an older driver rejects the PTX with -`CUDA_ERROR_UNSUPPORTED_PTX_VERSION`. These groups run automatically on a rented GPU in the merge -queue via `.github/workflows/gpu-tests.yml` (which filters offers on `cuda_max_good`). +The kernels are AOT-compiled by `nvcc` into native cubin (SASS) for the host GPU's real arch +(detected via `nvidia-smi`, or overridden with `CUDARC_NVCC_ARCH`), not PTX. This sidesteps the +PTX-ISA JIT version check, so a CUDA toolkit *newer* than the driver still loads and runs — no +`CUDA_ERROR_UNSUPPORTED_PTX_VERSION` and no need to hand-match the toolkit to the driver. The only +requirement is that the toolkit knows the GPU's compute capability (a too-old toolkit fails loudly +at `nvcc` build time). cudarc's host-side driver-API symbol set is likewise pinned to a safe floor +(`cuda-12080`) in `crypto/math-cuda/Cargo.toml`, so no `CUDARC_CUDA_VERSION` env is needed either. +These groups run automatically on a rented GPU in the merge queue via +`.github/workflows/gpu-tests.yml` (which filters offers on `cuda_max_good`). ## Benchmarking & Profiling diff --git a/crypto/math-cuda/Cargo.toml b/crypto/math-cuda/Cargo.toml index df4ae6770..7a28498da 100644 --- a/crypto/math-cuda/Cargo.toml +++ b/crypto/math-cuda/Cargo.toml @@ -6,12 +6,26 @@ edition = "2024" license.workspace = true [dependencies] +# cudarc CUDA version is PINNED to `cuda-12080` (CUDA 12.8) — do NOT restore +# `cuda-version-from-build-system` + `fallback-latest`. Rationale: +# * That auto-detect binds the newest symbol set the *build toolkit* knows +# (e.g. a CUDA 13.1 toolkit pulls in `cuDevSmResourceSplit`, gated behind +# `cuda-13010`/`cuda-13020`). cudarc eagerly resolves those symbols at CUDA +# init; a driver that predates them (e.g. 580.x = CUDA 13.0 max) has no such +# export, so the `dynamic-loading` resolver `.expect()`s and PANICS. +# * This crate's cudarc surface is entirely CUDA-11-era +# (CudaContext/CudaFunction/CudaSlice/CudaStream/LaunchConfig/PushKernelArg/ +# DriverError/Ptx — no green contexts). The 12.8 symbol set is a strict +# subset every >=12.8 driver exports, so pinning it resolves cleanly on any +# supported driver and a *newer* driver loses nothing we use. +# * This replaces the fragile per-script `sed` pin in scripts/gpu_test.sh. +# To move the floor (e.g. to use a newer driver-API symbol), bump this one +# feature deliberately — see crypto/math-cuda/build.rs and README "GPU Tests". cudarc = { version = "0.19", default-features = false, features = [ "driver", "nvrtc", "std", - "cuda-version-from-build-system", - "fallback-latest", + "cuda-12080", "dynamic-loading", ] } math = { path = "../math" } diff --git a/crypto/math-cuda/build.rs b/crypto/math-cuda/build.rs index 316a9c7ed..e419622d0 100644 --- a/crypto/math-cuda/build.rs +++ b/crypto/math-cuda/build.rs @@ -15,10 +15,11 @@ fn nvcc_path() -> PathBuf { } /// Query `nvidia-smi` for the local GPU's compute capability (e.g. "12.0" -/// for Blackwell). Returns a `compute_XX` target on success, falling back -/// to `compute_89` (Ada) when no GPU is visible or the query fails. +/// for Blackwell) and return a *real* arch (`sm_XX`) suitable for cubin +/// (SASS) generation. Falls back to `sm_89` (Ada) when no GPU is visible or +/// the query fails. fn detect_arch() -> String { - const FALLBACK: &str = "compute_89"; + const FALLBACK: &str = "sm_89"; let output = match Command::new("nvidia-smi") .args(["--query-gpu=compute_cap", "--format=csv,noheader"]) .output() @@ -40,13 +41,26 @@ fn detect_arch() -> String { None => return FALLBACK.to_string(), }; if major.chars().all(|c| c.is_ascii_digit()) && minor.chars().all(|c| c.is_ascii_digit()) { - format!("compute_{major}{minor}") + format!("sm_{major}{minor}") } else { FALLBACK.to_string() } } -fn compile_ptx(src: &str, out_name: &str, have_nvcc: bool) { +/// Normalize a user-supplied `CUDARC_NVCC_ARCH` override to a *real* arch +/// (`sm_XX`). cubin (SASS) generation rejects the *virtual* `compute_XX` +/// form, but we accept it (and a bare `XX`) for backwards compatibility. +fn to_real_arch(arch: &str) -> String { + if let Some(n) = arch.strip_prefix("compute_") { + format!("sm_{n}") + } else if arch.starts_with("sm_") { + arch.to_string() + } else { + format!("sm_{arch}") + } +} + +fn compile_kernel(src: &str, out_name: &str, have_nvcc: bool) { let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap()); let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap()); let src_path = manifest_dir.join("kernels").join(src); @@ -57,30 +71,39 @@ fn compile_ptx(src: &str, out_name: &str, have_nvcc: bool) { println!("cargo:rerun-if-env-changed=CUDA_PATH"); println!("cargo:rerun-if-env-changed=CUDARC_NVCC_ARCH"); - // When nvcc is missing from PATH, emit an empty PTX stub so the crate - // still compiles. include_str! in src/device.rs needs the file to exist - // at build time. Any runtime kernel call panics in cudarc when loading - // the empty module. We can't run GPU code without nvcc on the build - // host anyway. + // When nvcc is missing from PATH, emit an empty cubin stub so the crate + // still compiles. include_bytes! in src/device.rs needs the file to exist + // at build time. Any runtime kernel call fails to load the empty module and + // the caller falls back to CPU. We can't run GPU code without nvcc on the + // build host anyway. if !have_nvcc { - fs::write(&out_path, "").expect("failed to write empty PTX stub"); + fs::write(&out_path, "").expect("failed to write empty cubin stub"); return; } - // Emit PTX for a virtual architecture; the CUDA driver JIT-compiles it for the - // actual GPU at load time. Override with CUDARC_NVCC_ARCH to pin a specific - // compute capability. If unset, try `nvidia-smi` to match the host GPU - // (avoids JIT failures like nvcc-13.0 PTX rejected on Blackwell drivers); - // fall back to compute_89 (Ada) when detection fails. + // AOT-compile each kernel to a native cubin (SASS) for the host GPU's real + // arch, NOT to PTX. This sidesteps the driver's PTX-ISA JIT version check: + // a toolkit's PTX ISA is fixed by its CUDA version (e.g. CUDA 13.1 emits PTX + // .version 9.1), and a driver older than that toolkit rejects the module at + // load with CUDA_ERROR_UNSUPPORTED_PTX_VERSION -> every kernel silently + // falls back to CPU. A cubin carries pre-compiled SASS for a real arch, so + // the driver loads it directly as long as it supports that GPU (which the + // driver installed for that GPU always does) — regardless of the toolkit's + // CUDA version. See README "GPU Tests". // - // NOTE: this `-arch` only sets the *virtual arch*, not the PTX ISA version, which is - // fixed by this nvcc's CUDA toolkit. The runtime driver must support that toolkit's CUDA - // version or it rejects the PTX with CUDA_ERROR_UNSUPPORTED_PTX_VERSION — i.e. the box's - // driver CUDA must be >= the build toolkit's CUDA. See README "GPU Tests". - let arch = env::var("CUDARC_NVCC_ARCH").unwrap_or_else(|_| detect_arch()); + // Trade-off: a cubin is arch-specific (an `sm_120` cubin runs only on + // `sm_120`). We build+run on the same GPU box in every flow and detect the + // arch from that box's `nvidia-smi`, so this is exactly right. Override with + // CUDARC_NVCC_ARCH (compute_XX / sm_XX / bare XX all accepted) to + // cross-compile for a different arch; fall back to sm_89 (Ada) when + // detection fails. If the toolkit is too old to know the GPU's arch, nvcc + // fails loudly here at build time (better than a silent runtime fallback). + let arch = env::var("CUDARC_NVCC_ARCH") + .map(|a| to_real_arch(&a)) + .unwrap_or_else(|_| detect_arch()); let status = Command::new(nvcc_path()) - .args(["--ptx", "-O3", "-std=c++17", "-arch", &arch, "-o"]) + .args(["--cubin", "-O3", "-std=c++17", "-arch", &arch, "-o"]) .arg(&out_path) .arg(&src_path) .status() @@ -107,19 +130,19 @@ fn main() { .unwrap_or(false); if !have_nvcc { println!( - "cargo:warning=math-cuda: nvcc not found at {} — emitting empty PTX stubs. \ - Runtime GPU calls will panic. Install CUDA and rebuild for a working backend.", + "cargo:warning=math-cuda: nvcc not found at {} — emitting empty cubin stubs. \ + Runtime GPU calls fall back to CPU. Install CUDA and rebuild for a working backend.", nvcc_path().display() ); } - compile_ptx("arith.cu", "arith.ptx", have_nvcc); - compile_ptx("ntt.cu", "ntt.ptx", have_nvcc); - compile_ptx("keccak.cu", "keccak.ptx", have_nvcc); - compile_ptx("barycentric.cu", "barycentric.ptx", have_nvcc); - compile_ptx("deep.cu", "deep.ptx", have_nvcc); - compile_ptx("fri.cu", "fri.ptx", have_nvcc); - compile_ptx("inverse.cu", "inverse.ptx", have_nvcc); - compile_ptx("logup.cu", "logup.ptx", have_nvcc); - compile_ptx("constraint_interp.cu", "constraint_interp.ptx", have_nvcc); + compile_kernel("arith.cu", "arith.cubin", have_nvcc); + compile_kernel("ntt.cu", "ntt.cubin", have_nvcc); + compile_kernel("keccak.cu", "keccak.cubin", have_nvcc); + compile_kernel("barycentric.cu", "barycentric.cubin", have_nvcc); + compile_kernel("deep.cu", "deep.cubin", have_nvcc); + compile_kernel("fri.cu", "fri.cubin", have_nvcc); + compile_kernel("inverse.cu", "inverse.cubin", have_nvcc); + compile_kernel("logup.cu", "logup.cubin", have_nvcc); + compile_kernel("constraint_interp.cu", "constraint_interp.cubin", have_nvcc); } diff --git a/crypto/math-cuda/src/device.rs b/crypto/math-cuda/src/device.rs index bf75b696e..db6077c7a 100644 --- a/crypto/math-cuda/src/device.rs +++ b/crypto/math-cuda/src/device.rs @@ -90,16 +90,21 @@ impl Drop for PinnedStaging { } } -const ARITH_PTX: &str = include_str!(concat!(env!("OUT_DIR"), "/arith.ptx")); -const NTT_PTX: &str = include_str!(concat!(env!("OUT_DIR"), "/ntt.ptx")); -const KECCAK_PTX: &str = include_str!(concat!(env!("OUT_DIR"), "/keccak.ptx")); -const BARY_PTX: &str = include_str!(concat!(env!("OUT_DIR"), "/barycentric.ptx")); -const DEEP_PTX: &str = include_str!(concat!(env!("OUT_DIR"), "/deep.ptx")); -const FRI_PTX: &str = include_str!(concat!(env!("OUT_DIR"), "/fri.ptx")); -const INVERSE_PTX: &str = include_str!(concat!(env!("OUT_DIR"), "/inverse.ptx")); -const LOGUP_PTX: &str = include_str!(concat!(env!("OUT_DIR"), "/logup.ptx")); -const CONSTRAINT_INTERP_PTX: &str = - include_str!(concat!(env!("OUT_DIR"), "/constraint_interp.ptx")); +// Kernels are AOT-compiled to native cubin (SASS) by build.rs, embedded here, +// and loaded via `Ptx::from_binary` (cubin bytes -> cuModuleLoadData). This +// avoids the PTX-ISA/driver-version JIT check — see build.rs `compile_kernel`. +// An empty slice (nvcc-less stub build) fails to load at runtime and the caller +// falls back to CPU. +const ARITH_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/arith.cubin")); +const NTT_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/ntt.cubin")); +const KECCAK_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/keccak.cubin")); +const BARY_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/barycentric.cubin")); +const DEEP_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/deep.cubin")); +const FRI_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/fri.cubin")); +const INVERSE_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/inverse.cubin")); +const LOGUP_CUBIN: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/logup.cubin")); +const CONSTRAINT_INTERP_CUBIN: &[u8] = + include_bytes!(concat!(env!("OUT_DIR"), "/constraint_interp.cubin")); /// Number of CUDA streams in the pool. Larger pools let many rayon-parallel /// callers overlap on the GPU without serializing on stream ownership. The @@ -125,7 +130,7 @@ pub struct Backend { /// [`detect_vram_budget_bytes`]. vram_budget_bytes: u64, - // arith.ptx + // arith.cubin pub vector_add_u64: CudaFunction, pub gl_add: CudaFunction, pub gl_sub: CudaFunction, @@ -135,7 +140,7 @@ pub struct Backend { pub ext3_add: CudaFunction, pub ext3_sub: CudaFunction, - // ntt.ptx + // ntt.cubin pub bit_reverse_permute: CudaFunction, pub ntt_dit_level: CudaFunction, pub ntt_dit_8_levels: CudaFunction, @@ -152,7 +157,7 @@ pub struct Backend { pub pointwise_mul_row_major: CudaFunction, pub matrix_transpose_strided: CudaFunction, - // keccak.ptx + // keccak.cubin pub keccak256_leaves_base_row_major_row_pair: CudaFunction, pub keccak256_leaves_base_batched: CudaFunction, pub keccak256_leaves_base_row_pair_batched: CudaFunction, @@ -162,7 +167,7 @@ pub struct Backend { pub keccak_merkle_level: CudaFunction, pub merkle_gather_paths: CudaFunction, - // barycentric.ptx + // barycentric.cubin pub barycentric_base_batched: CudaFunction, pub barycentric_ext3_batched: CudaFunction, pub barycentric_base_batched_strided: CudaFunction, @@ -170,14 +175,14 @@ pub struct Backend { pub gather_rows_base: CudaFunction, pub gather_rows_ext3: CudaFunction, - // deep.ptx + // deep.cubin pub deep_composition_ext3_row: CudaFunction, - // fri.ptx + // fri.cubin pub fri_fold_ext3: CudaFunction, pub fri_update_twiddles: CudaFunction, - // inverse.ptx + // inverse.cubin pub compute_denoms_ext3: CudaFunction, pub block_inclusive_scan_fwd_ext3: CudaFunction, pub apply_block_offsets_fwd_ext3: CudaFunction, @@ -291,15 +296,16 @@ impl Backend { // we keep the current behaviour. retain_default_mempool(&ctx); - let arith = ctx.load_module(Ptx::from_src(ARITH_PTX))?; - let ntt = ctx.load_module(Ptx::from_src(NTT_PTX))?; - let keccak = ctx.load_module(Ptx::from_src(KECCAK_PTX))?; - let bary = ctx.load_module(Ptx::from_src(BARY_PTX))?; - let deep = ctx.load_module(Ptx::from_src(DEEP_PTX))?; - let fri = ctx.load_module(Ptx::from_src(FRI_PTX))?; - let inverse = ctx.load_module(Ptx::from_src(INVERSE_PTX))?; - let logup = ctx.load_module(Ptx::from_src(LOGUP_PTX))?; - let constraint_interp = ctx.load_module(Ptx::from_src(CONSTRAINT_INTERP_PTX))?; + let arith = ctx.load_module(Ptx::from_binary(ARITH_CUBIN.to_vec()))?; + let ntt = ctx.load_module(Ptx::from_binary(NTT_CUBIN.to_vec()))?; + let keccak = ctx.load_module(Ptx::from_binary(KECCAK_CUBIN.to_vec()))?; + let bary = ctx.load_module(Ptx::from_binary(BARY_CUBIN.to_vec()))?; + let deep = ctx.load_module(Ptx::from_binary(DEEP_CUBIN.to_vec()))?; + let fri = ctx.load_module(Ptx::from_binary(FRI_CUBIN.to_vec()))?; + let inverse = ctx.load_module(Ptx::from_binary(INVERSE_CUBIN.to_vec()))?; + let logup = ctx.load_module(Ptx::from_binary(LOGUP_CUBIN.to_vec()))?; + let constraint_interp = + ctx.load_module(Ptx::from_binary(CONSTRAINT_INTERP_CUBIN.to_vec()))?; let mut streams = Vec::with_capacity(STREAM_POOL_SIZE); for _ in 0..STREAM_POOL_SIZE { diff --git a/scripts/bench_abba.sh b/scripts/bench_abba.sh index fd0e7ad3e..a19b6ee20 100755 --- a/scripts/bench_abba.sh +++ b/scripts/bench_abba.sh @@ -133,9 +133,12 @@ if [ "$need_build" = "1" ]; then # -f: discard any prior worktree edit (e.g. the CUDARC_PIN sed below) before switching # refs, so the checkout can't conflict. git -C "$WT" checkout --quiet -f "$1" - # CUDARC_PIN: pin math-cuda's cudarc to a fixed CUDA version and drop fallback-latest, so - # cudarc binds a known driver-symbol set instead of its newest (which can request symbols - # the rented box's driver doesn't export, e.g. cuDevSmResourceSplit -> runtime panic). + # CUDARC_PIN: compat shim for benching *pre-pin* baseline shas. Newer shas pin cudarc's + # CUDA version permanently in crypto/math-cuda/Cargo.toml (feature `cuda-12080`), so this + # sed no-ops on them (the `cuda-version-from-build-system` anchor is gone). On an older + # baseline sha it still swaps in the pin + drops fallback-latest, so cudarc binds a known + # driver-symbol set instead of its newest (which can request symbols the rented box's + # driver doesn't export, e.g. cuDevSmResourceSplit -> runtime panic). if [ -n "${CUDARC_PIN:-}" ]; then sed -i "s/\"cuda-version-from-build-system\"/\"${CUDARC_PIN}\"/; /\"fallback-latest\"/d" \ "$WT/crypto/math-cuda/Cargo.toml" diff --git a/scripts/gpu_test.sh b/scripts/gpu_test.sh index 661339f11..1c5458a67 100755 --- a/scripts/gpu_test.sh +++ b/scripts/gpu_test.sh @@ -14,12 +14,10 @@ # group failed, which fails the workflow job and blocks the merge. # # Env: -# CUDARC_PIN cudarc CUDA-version feature to pin (default cuda-12080). See the sed below. # SYSROOT_DIR rv64 sysroot (default /opt/lambda-vm-sysroot, provisioned by the template). set -euo pipefail -CUDARC_PIN="${CUDARC_PIN:-cuda-12080}" export SYSROOT_DIR="${SYSROOT_DIR:-/opt/lambda-vm-sysroot}" log() { printf '\n=== %s ===\n' "$*"; } @@ -37,26 +35,9 @@ nvcc --version | tail -n 2 nvidia-smi nvidia-smi --query-gpu=name,driver_version,compute_cap --format=csv,noheader -# --- Pin cudarc so it binds a fixed driver-symbol set -------------------------- -# crypto/math-cuda/Cargo.toml uses `cuda-version-from-build-system` + `fallback-latest`; -# when detection falls back to "latest", cudarc requests symbols some boxes' driver doesn't -# export (e.g. cuDevSmResourceSplit / cuCtxGetDevice_v2) -> runtime panic. Pinning to a fixed, -# conservative CUDA version binds a known driver-symbol set instead. (This is cudarc's -# host-side driver-API floor — independent of the PTX/driver version the offer filter targets.) -log "pinning cudarc to $CUDARC_PIN" -# Guard the sed anchors: if math-cuda's cudarc features are ever renamed/reformatted, a silent -# no-op here would bring the fallback-latest driver-symbol panic back with a confusing signature. -for anchor in '"cuda-version-from-build-system"' '"fallback-latest"'; do - grep -qF "$anchor" crypto/math-cuda/Cargo.toml \ - || { echo "ERROR: sed anchor $anchor not found in crypto/math-cuda/Cargo.toml — update this script's cudarc pin" >&2; exit 1; } -done -# Restore the tracked file on exit so a manual run on a dev box doesn't leave the tree dirty -# (CI doesn't need this — the workflow re-checks-out before every run — but it's harmless there). -CUDARC_TOML_BACKUP="$(mktemp)" -cp crypto/math-cuda/Cargo.toml "$CUDARC_TOML_BACKUP" -trap 'cp "$CUDARC_TOML_BACKUP" crypto/math-cuda/Cargo.toml; rm -f "$CUDARC_TOML_BACKUP"' EXIT -sed -i "s/\"cuda-version-from-build-system\"/\"${CUDARC_PIN}\"/; /\"fallback-latest\"/d" \ - crypto/math-cuda/Cargo.toml +# cudarc's CUDA-version pin now lives permanently in crypto/math-cuda/Cargo.toml +# (feature `cuda-12080`), so this script no longer patches the manifest. Kernels +# are AOT-compiled to cubin by build.rs, so no PTX/driver-version juggling either. # --- Build the guest ELFs the tests prove --------------------------------------- # math-cuda parity needs none; cuda_path_integration / cuda_fallback prove an asm ELF; the From 260d873229d93f040f3fa2819b22aad05f4a1fb0 Mon Sep 17 00:00:00 2001 From: MauroFab Date: Thu, 9 Jul 2026 00:04:02 -0300 Subject: [PATCH 2/4] =?UTF-8?q?fix(math-cuda):=20review=20fixes=20?= =?UTF-8?q?=E2=80=94=20warn=20on=20cubin=20load=20failure=20+=20bench=20pi?= =?UTF-8?q?n=20no-op?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - backend(): warn once when GPU init fails, so a cubin arch mismatch (AOT cubins are arch-specific) doesn't silently disable the GPU with no signal. Stale ptx->cubin comment fixed. - bench_abba.sh: warn when CUDARC_PIN can't apply (post-pin sha, anchor gone) instead of silently no-opping. --- crypto/math-cuda/src/device.rs | 23 +++++++++++++++++++++-- scripts/bench_abba.sh | 13 ++++++++++--- 2 files changed, 31 insertions(+), 5 deletions(-) diff --git a/crypto/math-cuda/src/device.rs b/crypto/math-cuda/src/device.rs index db6077c7a..2bebe2cc0 100644 --- a/crypto/math-cuda/src/device.rs +++ b/crypto/math-cuda/src/device.rs @@ -197,7 +197,7 @@ pub struct Backend { pub logup_finalize_accum_ext3: CudaFunction, pub logup_assemble_aux_ext3: CudaFunction, - // constraint_interp.ptx + // constraint_interp.cubin pub constraint_interp_kernel: CudaFunction, pub constraint_composition_kernel: CudaFunction, @@ -510,7 +510,26 @@ pub fn backend() -> Result<&'static Backend> { if let Some(b) = BACKEND.get() { return Ok(b); } - let b = Backend::init()?; + let b = match Backend::init() { + Ok(b) => b, + Err(e) => { + // Backend init failing means every GPU entry point silently falls + // back to CPU. That is expected on a GPU-less host, but it also + // fires when the AOT cubins won't load — most often a build-host vs + // run-host GPU-arch mismatch (cubins are compiled for the detected + // `sm_XX`) or an empty nvcc-less stub. Warn once so the fallback is + // never silent: rebuild on the run host, or set `CUDARC_NVCC_ARCH`. + static WARNED: std::sync::Once = std::sync::Once::new(); + WARNED.call_once(|| { + eprintln!( + "math-cuda: GPU backend unavailable ({e}) — running on CPU. \ + If a GPU is present this is likely a kernel-cubin arch mismatch; \ + rebuild on the run host or set CUDARC_NVCC_ARCH to its sm_XX." + ); + }); + return Err(e); + } + }; let _ = BACKEND.set(b); Ok(BACKEND.get().expect("backend just initialised")) } diff --git a/scripts/bench_abba.sh b/scripts/bench_abba.sh index a19b6ee20..19d00a682 100755 --- a/scripts/bench_abba.sh +++ b/scripts/bench_abba.sh @@ -140,9 +140,16 @@ if [ "$need_build" = "1" ]; then # driver-symbol set instead of its newest (which can request symbols the rented box's # driver doesn't export, e.g. cuDevSmResourceSplit -> runtime panic). if [ -n "${CUDARC_PIN:-}" ]; then - sed -i "s/\"cuda-version-from-build-system\"/\"${CUDARC_PIN}\"/; /\"fallback-latest\"/d" \ - "$WT/crypto/math-cuda/Cargo.toml" - echo " cudarc pinned to ${CUDARC_PIN}" + if grep -q '"cuda-version-from-build-system"' "$WT/crypto/math-cuda/Cargo.toml"; then + sed -i "s/\"cuda-version-from-build-system\"/\"${CUDARC_PIN}\"/; /\"fallback-latest\"/d" \ + "$WT/crypto/math-cuda/Cargo.toml" + echo " cudarc pinned to ${CUDARC_PIN}" + else + # Post-pin shas already hard-pin cudarc in Cargo.toml, so the anchor is + # gone and the sed would silently no-op. Warn rather than mislead. + echo " WARNING: CUDARC_PIN=${CUDARC_PIN} ignored @ ${1:0:10} — cudarc is already" >&2 + echo " pinned in crypto/math-cuda/Cargo.toml (no build-system anchor to rewrite)." >&2 + fi fi if ! ( cd "$WT" && cargo build --release -p cli --features "$BENCH_FEATURES" >"$WORK/build_$2.log" 2>&1 ); then echo "ERROR: cargo build failed for $2 (@ ${1:0:10}). Tail of $WORK/build_$2.log:" >&2 From f32fb9a2622f950d9b1a68da8966b25f7acbf1a3 Mon Sep 17 00:00:00 2001 From: MauroFab Date: Mon, 13 Jul 2026 12:36:29 -0300 Subject: [PATCH 3/4] =?UTF-8?q?fix(math-cuda,scripts):=20review=20fixes=20?= =?UTF-8?q?=E2=80=94=20warn=20on=20sm=5F89=20arch=20fallback=20+=20bench?= =?UTF-8?q?=5Fabba=5Fgpu=20pin=20guard/restore=20+=20cubin=20comment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- crypto/math-cuda/build.rs | 40 +++++++++++++++++++++++---------------- prover/Cargo.toml | 4 ++-- scripts/bench_abba_gpu.sh | 27 +++++++++++++++++++++----- 3 files changed, 48 insertions(+), 23 deletions(-) diff --git a/crypto/math-cuda/build.rs b/crypto/math-cuda/build.rs index e419622d0..0a1e96c93 100644 --- a/crypto/math-cuda/build.rs +++ b/crypto/math-cuda/build.rs @@ -17,33 +17,41 @@ fn nvcc_path() -> PathBuf { /// Query `nvidia-smi` for the local GPU's compute capability (e.g. "12.0" /// for Blackwell) and return a *real* arch (`sm_XX`) suitable for cubin /// (SASS) generation. Falls back to `sm_89` (Ada) when no GPU is visible or -/// the query fails. +/// the query fails, warning loudly because an arch-specific cubin built for +/// the wrong GPU cannot load on the run host. fn detect_arch() -> String { const FALLBACK: &str = "sm_89"; + detect_arch_from_smi().unwrap_or_else(|| { + println!( + "cargo:warning=math-cuda: could not detect a GPU via nvidia-smi — cubins target \ + fallback {FALLBACK}; if the run host GPU differs, set CUDARC_NVCC_ARCH=sm_XX or \ + rebuild on the run host." + ); + FALLBACK.to_string() + }) +} + +/// Parse the compute capability out of `nvidia-smi` and format it as a real +/// `sm_XX` arch. Returns `None` on every path where no capability can be read +/// (nvidia-smi missing, command failed, or unparsable output) so the caller +/// warns before falling back. +fn detect_arch_from_smi() -> Option { let output = match Command::new("nvidia-smi") .args(["--query-gpu=compute_cap", "--format=csv,noheader"]) .output() { Ok(o) if o.status.success() => o, - _ => return FALLBACK.to_string(), - }; - let line = match std::str::from_utf8(&output.stdout) { - Ok(s) => s, - Err(_) => return FALLBACK.to_string(), + _ => return None, }; + let line = std::str::from_utf8(&output.stdout).ok()?; // First line, first comma-separated value (covers multi-GPU hosts). - let cap = match line.lines().next() { - Some(l) => l.split(',').next().unwrap_or("").trim(), - None => return FALLBACK.to_string(), - }; - let (major, minor) = match cap.split_once('.') { - Some((m, n)) => (m.trim(), n.trim()), - None => return FALLBACK.to_string(), - }; + let cap = line.lines().next()?.split(',').next().unwrap_or("").trim(); + let (major, minor) = cap.split_once('.')?; + let (major, minor) = (major.trim(), minor.trim()); if major.chars().all(|c| c.is_ascii_digit()) && minor.chars().all(|c| c.is_ascii_digit()) { - format!("sm_{major}{minor}") + Some(format!("sm_{major}{minor}")) } else { - FALLBACK.to_string() + None } } diff --git a/prover/Cargo.toml b/prover/Cargo.toml index 57e8114d0..195fb39d3 100644 --- a/prover/Cargo.toml +++ b/prover/Cargo.toml @@ -38,8 +38,8 @@ tiny-keccak = { version = "2.0", features = ["keccak"] } # `compute_precomputed_commitment_for_testing`. Only active under cargo test/bench. stark = { path = "../crypto/stark", features = ["test-utils"] } # Device-resident LDE handles for the cuda-gated GPU constraint-interp parity -# test (`tests/gpu_constraint_interp_real.rs`). Its build.rs stubs empty PTX when -# nvcc is absent, so this compiles on CPU-only hosts too; the test itself is +# test (`tests/gpu_constraint_interp_real.rs`). Its build.rs stubs an empty cubin +# when nvcc is absent, so this compiles on CPU-only hosts too; the test itself is # `#[cfg(feature = "cuda")]` and only runs with a GPU. math-cuda = { path = "../crypto/math-cuda" } diff --git a/scripts/bench_abba_gpu.sh b/scripts/bench_abba_gpu.sh index b54753d9a..dc011962e 100755 --- a/scripts/bench_abba_gpu.sh +++ b/scripts/bench_abba_gpu.sh @@ -45,15 +45,32 @@ ELF="$ROOT/$ELF_REL" INPUT="$ROOT/$INPUT_REL" if [ "${REBUILD:-0}" = "1" ] || [ ! -x "$WORK/cli" ]; then + # CUDARC_PIN: compat shim for benching *pre-pin* checkouts. Newer trees pin + # cudarc's CUDA version permanently in crypto/math-cuda/Cargo.toml, so the + # `cuda-version-from-build-system` anchor is gone and this sed no-ops on them. if [ -n "${CUDARC_PIN:-}" ]; then - sed -i "s/\"cuda-version-from-build-system\"/\"${CUDARC_PIN}\"/; /\"fallback-latest\"/d" \ - crypto/math-cuda/Cargo.toml - echo " cudarc pinned to ${CUDARC_PIN}" + if grep -q '"cuda-version-from-build-system"' crypto/math-cuda/Cargo.toml; then + sed -i "s/\"cuda-version-from-build-system\"/\"${CUDARC_PIN}\"/; /\"fallback-latest\"/d" \ + crypto/math-cuda/Cargo.toml + echo " cudarc pinned to ${CUDARC_PIN}" + else + # Post-pin checkouts already hard-pin cudarc in Cargo.toml, so the anchor + # is gone and the sed would silently no-op. Warn rather than mislead. + echo " WARNING: CUDARC_PIN=${CUDARC_PIN} ignored — cudarc is already" >&2 + echo " pinned in crypto/math-cuda/Cargo.toml (no build-system anchor to rewrite)." >&2 + fi fi echo "==> Building cli (features: $BENCH_FEATURES)" - cargo build --release -p cli --features "$BENCH_FEATURES" - cp target/release/cli "$WORK/cli" + # Restore the tracked Cargo.toml whether the build succeeds or fails, so a + # build error (set -e) can't leave the sed edit above in the working tree. + build_rc=0 + cargo build --release -p cli --features "$BENCH_FEATURES" || build_rc=$? [ -n "${CUDARC_PIN:-}" ] && git checkout -- crypto/math-cuda/Cargo.toml + if [ "$build_rc" -ne 0 ]; then + echo "ERROR: cargo build failed (rc=$build_rc)." >&2 + exit "$build_rc" + fi + cp target/release/cli "$WORK/cli" else echo "==> Reusing cached cli (REBUILD=1 to force)" fi From 20edc0e4b26fab6332ac4e6af62c78d723cdc64f Mon Sep 17 00:00:00 2001 From: MauroFab Date: Mon, 13 Jul 2026 13:13:39 -0300 Subject: [PATCH 4/4] refactor(scripts): shared ABBA bench lib + README GPU driver floor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bench_abba.sh and bench_abba_gpu.sh carried near-verbatim copies of the prove-time parsing, the CUDARC_PIN compat sed, and the paired-stats python — a stats fix applied to one would silently leave the other reporting different numbers for the same pairs. scripts/lib/bench_abba_common.sh now holds all three; the GPU bench gains the full analysis (exact Wilcoxon, stability diagnostics, verdicts) it previously lacked. README: state the cuda-12080 pin's driver floor (CUDA >= 12.8 / 570+) and that older drivers abort at CUDA init rather than CPU-fallback. --- README.md | 3 + scripts/bench_abba.sh | 147 ++------------------------- scripts/bench_abba_gpu.sh | 62 ++---------- scripts/lib/bench_abba_common.sh | 165 +++++++++++++++++++++++++++++++ 4 files changed, 187 insertions(+), 190 deletions(-) create mode 100644 scripts/lib/bench_abba_common.sh diff --git a/README.md b/README.md index b027da040..0967f34d6 100644 --- a/README.md +++ b/README.md @@ -240,6 +240,9 @@ PTX-ISA JIT version check, so a CUDA toolkit *newer* than the driver still loads requirement is that the toolkit knows the GPU's compute capability (a too-old toolkit fails loudly at `nvcc` build time). cudarc's host-side driver-API symbol set is likewise pinned to a safe floor (`cuda-12080`) in `crypto/math-cuda/Cargo.toml`, so no `CUDARC_CUDA_VERSION` env is needed either. +That pin makes the GPU path require a driver of CUDA >= 12.8 (driver branch 570+ — any +Blackwell-capable driver qualifies); on an older driver cudarc's eager symbol resolution aborts at +CUDA init rather than falling back to CPU. These groups run automatically on a rented GPU in the merge queue via `.github/workflows/gpu-tests.yml` (which filters offers on `cuda_max_good`). diff --git a/scripts/bench_abba.sh b/scripts/bench_abba.sh index 19d00a682..466fc298d 100755 --- a/scripts/bench_abba.sh +++ b/scripts/bench_abba.sh @@ -79,6 +79,8 @@ PROOF="/tmp/abba_proof.bin" ROOT="$(git rev-parse --show-toplevel)" cd "$ROOT" +# shellcheck source=scripts/lib/bench_abba_common.sh +. "$ROOT/scripts/lib/bench_abba_common.sh" # Fail fast on the toolchain the final stats step needs, before the ~30-min build. command -v python3 >/dev/null 2>&1 || { echo "ERROR: python3 is required (final stats step)." >&2; exit 1; } @@ -133,24 +135,9 @@ if [ "$need_build" = "1" ]; then # -f: discard any prior worktree edit (e.g. the CUDARC_PIN sed below) before switching # refs, so the checkout can't conflict. git -C "$WT" checkout --quiet -f "$1" - # CUDARC_PIN: compat shim for benching *pre-pin* baseline shas. Newer shas pin cudarc's - # CUDA version permanently in crypto/math-cuda/Cargo.toml (feature `cuda-12080`), so this - # sed no-ops on them (the `cuda-version-from-build-system` anchor is gone). On an older - # baseline sha it still swaps in the pin + drops fallback-latest, so cudarc binds a known - # driver-symbol set instead of its newest (which can request symbols the rented box's - # driver doesn't export, e.g. cuDevSmResourceSplit -> runtime panic). - if [ -n "${CUDARC_PIN:-}" ]; then - if grep -q '"cuda-version-from-build-system"' "$WT/crypto/math-cuda/Cargo.toml"; then - sed -i "s/\"cuda-version-from-build-system\"/\"${CUDARC_PIN}\"/; /\"fallback-latest\"/d" \ - "$WT/crypto/math-cuda/Cargo.toml" - echo " cudarc pinned to ${CUDARC_PIN}" - else - # Post-pin shas already hard-pin cudarc in Cargo.toml, so the anchor is - # gone and the sed would silently no-op. Warn rather than mislead. - echo " WARNING: CUDARC_PIN=${CUDARC_PIN} ignored @ ${1:0:10} — cudarc is already" >&2 - echo " pinned in crypto/math-cuda/Cargo.toml (no build-system anchor to rewrite)." >&2 - fi - fi + # CUDARC_PIN compat shim (no-op with a warning on post-pin shas) — see + # cudarc_pin_apply in scripts/lib/bench_abba_common.sh. + cudarc_pin_apply "$WT/crypto/math-cuda/Cargo.toml" if ! ( cd "$WT" && cargo build --release -p cli --features "$BENCH_FEATURES" >"$WORK/build_$2.log" 2>&1 ); then echo "ERROR: cargo build failed for $2 (@ ${1:0:10}). Tail of $WORK/build_$2.log:" >&2 tail -40 "$WORK/build_$2.log" >&2 @@ -171,17 +158,11 @@ fi # --- 3. Interleaved A/B/B/A measurement (fresh CSV -- pre-committed batch) --- run_prove() { # $1=binary -> echoes proving time (s) - local out t + local out # shellcheck disable=SC2086 # CONT_ARGS is intentionally word-split (0 or 2 args) out="$("$1" prove "$ELF" --private-input "$INPUT" -o "$PROOF" --time $CONT_ARGS 2>&1)" rm -f "$PROOF" - t="$(printf '%s\n' "$out" | grep -o 'Proving time: [0-9.]*' | awk '{print $3}')" - if [ -z "$t" ]; then - echo "ERROR: could not parse 'Proving time' from cli output:" >&2 - printf '%s\n' "$out" >&2 - exit 1 - fi - echo "$t" + extract_prove_time "$out" } echo "==> Running $N_PAIRS interleaved pairs (improvement: - = PR faster)" @@ -197,115 +178,5 @@ for i in $(seq 1 "$N_PAIRS"); do "$i" "$N_PAIRS" "$a" "$b" "$(awk "BEGIN{print ($a-$b)/$b*100}")" done -# --- 4. Paired t-test + robust median/Wilcoxon --- -python3 - "$WORK/pairs.csv" <<'PY' -import sys, csv, math - -rows = list(csv.DictReader(open(sys.argv[1]))) -A = [float(r['a_time']) for r in rows] # PR -B = [float(r['b_time']) for r in rows] # baseline -n = len(A) -# per-pair delta = (PR - baseline)/baseline: negative => PR (A) faster than baseline (B) -d = [(a - b) / b * 100.0 for a, b in zip(A, B)] - -# ---- parametric: paired t ---- -mean = sum(d) / n -var = sum((x - mean) ** 2 for x in d) / (n - 1) if n > 1 else 0.0 -sd = math.sqrt(var) -se = sd / math.sqrt(n) if n else float('inf') -TT = {1:12.706,2:4.303,3:3.182,4:2.776,5:2.571,6:2.447,7:2.365,8:2.306,9:2.262, - 10:2.228,11:2.201,12:2.179,13:2.160,14:2.145,15:2.131,16:2.120,17:2.110, - 18:2.101,19:2.093,20:2.086,21:2.080,22:2.074,23:2.069,24:2.064,25:2.060, - 26:2.056,27:2.052,28:2.048,29:2.045,30:2.042,35:2.030,40:2.021,50:2.009, - 60:2.000,80:1.990,120:1.980} -df = n - 1 -tc = TT.get(df) or (1.96 if df > 120 else TT[min(TT, key=lambda k: abs(k - df))]) -lo, hi = mean - tc * se, mean + tc * se - -# ---- robust: median + Wilcoxon signed-rank (tie-averaged ranks, EXACT p, pure stdlib) ---- -def median(xs): - s = sorted(xs); m = len(s) - return s[m // 2] if m % 2 else (s[m // 2 - 1] + s[m // 2]) / 2 - -nz = [x for x in d if x != 0.0] -m = len(nz) -order = sorted(range(m), key=lambda i: abs(nz[i])) -ranks = [0.0] * m -i = 0 -while i < m: # average ranks within ties on |d| - j = i - while j + 1 < m and abs(nz[order[j + 1]]) == abs(nz[order[i]]): - j += 1 - avg = (i + 1 + j + 1) / 2.0 - for k in range(i, j + 1): - ranks[order[k]] = avg - i = j + 1 -Wp = sum(r for r, x in zip(ranks, nz) if x > 0) -Wn = sum(r for r, x in zip(ranks, nz) if x < 0) -mu = m * (m + 1) / 4.0 -sig = math.sqrt(m * (m + 1) * (2 * m + 1) / 24.0) if m else 0.0 -z = (Wp - mu - (0.5 if Wp > mu else -0.5)) / sig if sig else 0.0 # normal approx (display only) -# EXACT two-sided p: enumerate the signed-rank null distribution. Each rank is +/- with -# prob 1/2, so the count of assignments giving W+=v is the coeff of x^v in prod(1 + x^rank) -# -- build it with a generating-function DP. Double the ranks so tie-averaged (half-integer) -# ranks become integers. No scipy; exact even at small n where the normal approx is loose. -if m: - ir = [int(round(2 * r)) for r in ranks] - poly = [1] - for r in ir: - nxt = [0] * (len(poly) + r) - for v, c in enumerate(poly): - if c: - nxt[v] += c # this rank negative -> adds 0 to W+ - nxt[v + r] += c # this rank positive -> adds r to W+ - poly = nxt - Wp2 = int(round(2 * Wp)) - p = min(1.0, 2.0 * min(sum(poly[:Wp2 + 1]), sum(poly[Wp2:])) / (1 << m)) -else: - p = 1.0 -med = median(d) - -# ---- server stability (byproduct): run-to-run jitter + within-session drift ---- -def cv(xs): - mm = sum(xs) / len(xs) - s = math.sqrt(sum((x - mm) ** 2 for x in xs) / (len(xs) - 1)) if len(xs) > 1 else 0.0 - return (s / mm * 100.0) if mm else 0.0 -mA, mB = sum(A) / n, sum(B) / n -cvA, cvB = cv(A), cv(B) -# reconstruct execution order (odd pair: A,B ; even pair: B,A) and normalize each -# run by its binary's mean so the A/B offset drops out, leaving pure machine drift. -seq = [] -for i in range(n): - seq += ([('A', A[i]), ('B', B[i])] if (i + 1) % 2 else [('B', B[i]), ('A', A[i])]) -nrm = [(t / (mA if lbl == 'A' else mB) - 1) * 100 for lbl, t in seq] -N = len(nrm); mi = (N - 1) / 2.0; mn = sum(nrm) / N -denom = sum((i - mi) ** 2 for i in range(N)) -slope = (sum((i - mi) * (nrm[i] - mn) for i in range(N)) / denom) if denom else 0.0 -half = N // 2 -drift_shift = sum(nrm[half:]) / (N - half) - sum(nrm[:half]) / half - -print("\n=== ABBA paired result (improvement: - = PR faster) ===") -print(f" pairs: {n} mean A (PR): {sum(A)/n:.3f}s mean B (base): {sum(B)/n:.3f}s") -print() -print(f" [parametric] paired-t mean {mean:+.2f}% sd {sd:.2f}% se {se:.2f}%") -print(f" 95% CI: [{lo:+.2f}%, {hi:+.2f}%] (t df={df} = {tc})") -pstr = f"{p:.4f}" if p >= 1e-4 else f"{p:.1e}" -print(f" [robust] median {med:+.2f}% Wilcoxon W+={Wp:.0f} W-={Wn:.0f} p(exact)={pstr} (z={z:+.2f})") -print() -print(" --- server stability (this run; compare across servers) ---") -print(f" run-to-run jitter: A CV {cvA:.2f}% B CV {cvB:.2f}% (lower = steadier)") -print(f" within-session drift: {slope * N:+.2f}% over the run, 1st->2nd half {drift_shift:+.2f}%") -print(f" (jitter -> Tier-1 cached gate floor; drift -> whether the cached baseline can be trusted)") -print() -if hi < 0 and p < 0.05: - print(f" VERDICT: REAL IMPROVEMENT - PR faster by ~{-mean:.2f}% (t-CI and Wilcoxon agree)") -elif lo > 0 and p < 0.05: - print(f" VERDICT: REAL REGRESSION - PR slower by ~{mean:.2f}% (t-CI and Wilcoxon agree)") -elif (hi < 0) != (p < 0.05): - print(f" VERDICT: BORDERLINE - parametric and robust disagree; suspect outlier pair(s).") - print(f" Trust the median ({med:+.2f}%); add pairs or inspect the per-pair list.") -else: - print(f" VERDICT: INCONCLUSIVE - effect not separable from 0 at n={n}.") - print(f" Point estimate ~{med:+.2f}% (median). Need more pairs to resolve.") -print(f"\n raw pairs: {sys.argv[1]}") -PY +# --- 4. Paired t-test + robust median/Wilcoxon (shared analysis) --- +abba_stats "$WORK/pairs.csv" "PR" "base" "ABBA paired result" diff --git a/scripts/bench_abba_gpu.sh b/scripts/bench_abba_gpu.sh index dc011962e..3c8f0643e 100755 --- a/scripts/bench_abba_gpu.sh +++ b/scripts/bench_abba_gpu.sh @@ -32,6 +32,8 @@ PROOF="/tmp/abba_gpu_proof.bin" ROOT="$(git rev-parse --show-toplevel)" cd "$ROOT" +# shellcheck source=scripts/lib/bench_abba_common.sh +. "$ROOT/scripts/lib/bench_abba_common.sh" command -v python3 >/dev/null 2>&1 || { echo "ERROR: python3 required." >&2; exit 1; } mkdir -p "$WORK" @@ -45,21 +47,9 @@ ELF="$ROOT/$ELF_REL" INPUT="$ROOT/$INPUT_REL" if [ "${REBUILD:-0}" = "1" ] || [ ! -x "$WORK/cli" ]; then - # CUDARC_PIN: compat shim for benching *pre-pin* checkouts. Newer trees pin - # cudarc's CUDA version permanently in crypto/math-cuda/Cargo.toml, so the - # `cuda-version-from-build-system` anchor is gone and this sed no-ops on them. - if [ -n "${CUDARC_PIN:-}" ]; then - if grep -q '"cuda-version-from-build-system"' crypto/math-cuda/Cargo.toml; then - sed -i "s/\"cuda-version-from-build-system\"/\"${CUDARC_PIN}\"/; /\"fallback-latest\"/d" \ - crypto/math-cuda/Cargo.toml - echo " cudarc pinned to ${CUDARC_PIN}" - else - # Post-pin checkouts already hard-pin cudarc in Cargo.toml, so the anchor - # is gone and the sed would silently no-op. Warn rather than mislead. - echo " WARNING: CUDARC_PIN=${CUDARC_PIN} ignored — cudarc is already" >&2 - echo " pinned in crypto/math-cuda/Cargo.toml (no build-system anchor to rewrite)." >&2 - fi - fi + # CUDARC_PIN compat shim (no-op with a warning on post-pin checkouts) — see + # cudarc_pin_apply in scripts/lib/bench_abba_common.sh. + cudarc_pin_apply crypto/math-cuda/Cargo.toml echo "==> Building cli (features: $BENCH_FEATURES)" # Restore the tracked Cargo.toml whether the build succeeds or fails, so a # build error (set -e) can't leave the sed edit above in the working tree. @@ -76,15 +66,11 @@ else fi run_prove() { # $1 = 0|1 (disable-flag) -> proving time (s) - local out t + local out out="$(LAMBDA_VM_DISABLE_GPU_COMPOSITION="$1" "$WORK/cli" prove "$ELF" \ --private-input "$INPUT" -o "$PROOF" --time 2>&1)" rm -f "$PROOF" - t="$(printf '%s\n' "$out" | grep -o 'Proving time: [0-9.]*' | awk '{print $3}')" - if [ -z "$t" ]; then - echo "ERROR: could not parse 'Proving time':" >&2; printf '%s\n' "$out" >&2; exit 1 - fi - echo "$t" + extract_prove_time "$out" } # Warm-up (PTX load, pools, pinned alloc) so pair 1 isn't an outlier. @@ -104,34 +90,6 @@ for i in $(seq 1 "$N_PAIRS"); do "$i" "$N_PAIRS" "$a" "$b" "$(awk "BEGIN{print ($a-$b)/$b*100}")" done -python3 - "$WORK/pairs.csv" <<'PY' -import sys, csv, math -rows = list(csv.DictReader(open(sys.argv[1]))) -A = [float(r['a_time']) for r in rows] # GPU composition ON -B = [float(r['b_time']) for r in rows] # GPU composition OFF (CPU) -n = len(A) -d = [(a - b) / b * 100.0 for a, b in zip(A, B)] -mean = sum(d) / n -var = sum((x - mean) ** 2 for x in d) / (n - 1) if n > 1 else 0.0 -sd = math.sqrt(var); se = sd / math.sqrt(n) if n else float('inf') -TT = {1:12.706,2:4.303,3:3.182,4:2.776,5:2.571,6:2.447,7:2.365,8:2.306,9:2.262, - 10:2.228,11:2.201,12:2.179,13:2.160,14:2.145,15:2.131,16:2.120,17:2.110, - 18:2.101,19:2.093,20:2.086,25:2.060,30:2.042,40:2.021,50:2.009,60:2.000} -df = n - 1 -tc = TT.get(df) or (1.96 if df > 120 else TT[min(TT, key=lambda k: abs(k - df))]) -lo, hi = mean - tc * se, mean + tc * se -def median(xs): - s = sorted(xs); m = len(s) - return s[m // 2] if m % 2 else (s[m // 2 - 1] + s[m // 2]) / 2 -med = median(d) -print("\n=== GPU-composition ABBA (A=GPU ON, B=CPU; - = GPU faster) ===") -print(f" pairs: {n} mean A (GPU): {sum(A)/n:.3f}s mean B (CPU): {sum(B)/n:.3f}s") -print(f" paired-t mean {mean:+.2f}% sd {sd:.2f}% se {se:.2f}% 95% CI [{lo:+.2f}%, {hi:+.2f}%]") -print(f" median {med:+.2f}%") -if hi < 0: - print(f" => GPU path faster by ~{-mean:.2f}% (CI below 0)") -elif lo > 0: - print(f" => GPU path slower by ~{mean:.2f}% (CI above 0)") -else: - print(f" => inconclusive at n={n} (CI straddles 0); point ~{med:+.2f}%") -PY +# Shared paired analysis (paired-t + exact Wilcoxon + stability + verdict) — +# same statistics as bench_abba.sh, by construction. +abba_stats "$WORK/pairs.csv" "GPU" "CPU" "GPU-composition ABBA" diff --git a/scripts/lib/bench_abba_common.sh b/scripts/lib/bench_abba_common.sh new file mode 100644 index 000000000..a16bd06b1 --- /dev/null +++ b/scripts/lib/bench_abba_common.sh @@ -0,0 +1,165 @@ +# bench_abba_common.sh — shared pieces of the ABBA prover benchmarks. +# +# Sourced by scripts/bench_abba.sh (two-ref: PR vs baseline binaries) and +# scripts/bench_abba_gpu.sh (one binary, runtime GPU toggle). Both benches +# parse the same cli output and MUST report identical statistics for the same +# pairs.csv, so the parsing and the analysis live here exactly once. +# +# Provides: +# extract_prove_time "" -> echoes the proving time (s) or dies +# cudarc_pin_apply -> CUDARC_PIN compat sed with no-op warn +# abba_stats
+# -> paired-t + exact Wilcoxon + stability + +# Parse `Proving time: ` out of a cli prove run's output; die loudly (with +# the full output on stderr) when it is missing, so a crashed prove can't +# silently feed an empty sample into the stats. +extract_prove_time() { # $1 = full cli output -> echoes time (s) + local t + t="$(printf '%s\n' "$1" | grep -o 'Proving time: [0-9.]*' | awk '{print $3}')" + if [ -z "$t" ]; then + echo "ERROR: could not parse 'Proving time' from cli output:" >&2 + printf '%s\n' "$1" >&2 + exit 1 + fi + echo "$t" +} + +# CUDARC_PIN: compat shim for benching *pre-pin* checkouts. Newer trees pin +# cudarc's CUDA version permanently in crypto/math-cuda/Cargo.toml (feature +# `cuda-12080`), so the `cuda-version-from-build-system` anchor is gone and the +# sed no-ops on them — warn rather than mislead. On an older checkout it still +# swaps in the pin + drops fallback-latest, so cudarc binds a known +# driver-symbol set instead of its newest (which can request symbols the rented +# box's driver doesn't export, e.g. cuDevSmResourceSplit -> runtime panic). +# No-op when CUDARC_PIN is unset. +cudarc_pin_apply() { # $1 = path to the checkout's crypto/math-cuda/Cargo.toml + [ -n "${CUDARC_PIN:-}" ] || return 0 + if grep -q '"cuda-version-from-build-system"' "$1"; then + sed -i "s/\"cuda-version-from-build-system\"/\"${CUDARC_PIN}\"/; /\"fallback-latest\"/d" "$1" + echo " cudarc pinned to ${CUDARC_PIN}" + else + echo " WARNING: CUDARC_PIN=${CUDARC_PIN} ignored — cudarc is already" >&2 + echo " pinned in $1 (no build-system anchor to rewrite)." >&2 + fi +} + +# Paired analysis over an ABBA pairs.csv (columns pair,a_time,b_time): +# paired-t 95% CI, robust median + EXACT Wilcoxon signed-rank, server-stability +# diagnostics, and a verdict. Labels name the two sides in the report +# (e.g. "PR"/"baseline" or "GPU ON"/"CPU"). Convention everywhere: +# delta = (A - B)/B, NEGATIVE = A faster. +abba_stats() { # $1=pairs.csv $2=label_A $3=label_B $4=report header + python3 - "$1" "$2" "$3" "$4" <<'PY' +import sys, csv, math + +path, LA, LB, HDR = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4] +rows = list(csv.DictReader(open(path))) +A = [float(r['a_time']) for r in rows] +B = [float(r['b_time']) for r in rows] +n = len(A) +# per-pair delta = (A - B)/B: negative => A faster than B +d = [(a - b) / b * 100.0 for a, b in zip(A, B)] + +# ---- parametric: paired t ---- +mean = sum(d) / n +var = sum((x - mean) ** 2 for x in d) / (n - 1) if n > 1 else 0.0 +sd = math.sqrt(var) +se = sd / math.sqrt(n) if n else float('inf') +TT = {1:12.706,2:4.303,3:3.182,4:2.776,5:2.571,6:2.447,7:2.365,8:2.306,9:2.262, + 10:2.228,11:2.201,12:2.179,13:2.160,14:2.145,15:2.131,16:2.120,17:2.110, + 18:2.101,19:2.093,20:2.086,21:2.080,22:2.074,23:2.069,24:2.064,25:2.060, + 26:2.056,27:2.052,28:2.048,29:2.045,30:2.042,35:2.030,40:2.021,50:2.009, + 60:2.000,80:1.990,120:1.980} +df = n - 1 +tc = TT.get(df) or (1.96 if df > 120 else TT[min(TT, key=lambda k: abs(k - df))]) +lo, hi = mean - tc * se, mean + tc * se + +# ---- robust: median + Wilcoxon signed-rank (tie-averaged ranks, EXACT p, pure stdlib) ---- +def median(xs): + s = sorted(xs); m = len(s) + return s[m // 2] if m % 2 else (s[m // 2 - 1] + s[m // 2]) / 2 + +nz = [x for x in d if x != 0.0] +m = len(nz) +order = sorted(range(m), key=lambda i: abs(nz[i])) +ranks = [0.0] * m +i = 0 +while i < m: # average ranks within ties on |d| + j = i + while j + 1 < m and abs(nz[order[j + 1]]) == abs(nz[order[i]]): + j += 1 + avg = (i + 1 + j + 1) / 2.0 + for k in range(i, j + 1): + ranks[order[k]] = avg + i = j + 1 +Wp = sum(r for r, x in zip(ranks, nz) if x > 0) +Wn = sum(r for r, x in zip(ranks, nz) if x < 0) +mu = m * (m + 1) / 4.0 +sig = math.sqrt(m * (m + 1) * (2 * m + 1) / 24.0) if m else 0.0 +z = (Wp - mu - (0.5 if Wp > mu else -0.5)) / sig if sig else 0.0 # normal approx (display only) +# EXACT two-sided p: enumerate the signed-rank null distribution. Each rank is +/- with +# prob 1/2, so the count of assignments giving W+=v is the coeff of x^v in prod(1 + x^rank) +# -- build it with a generating-function DP. Double the ranks so tie-averaged (half-integer) +# ranks become integers. No scipy; exact even at small n where the normal approx is loose. +if m: + ir = [int(round(2 * r)) for r in ranks] + poly = [1] + for r in ir: + nxt = [0] * (len(poly) + r) + for v, c in enumerate(poly): + if c: + nxt[v] += c # this rank negative -> adds 0 to W+ + nxt[v + r] += c # this rank positive -> adds r to W+ + poly = nxt + Wp2 = int(round(2 * Wp)) + p = min(1.0, 2.0 * min(sum(poly[:Wp2 + 1]), sum(poly[Wp2:])) / (1 << m)) +else: + p = 1.0 +med = median(d) + +# ---- server stability (byproduct): run-to-run jitter + within-session drift ---- +def cv(xs): + mm = sum(xs) / len(xs) + s = math.sqrt(sum((x - mm) ** 2 for x in xs) / (len(xs) - 1)) if len(xs) > 1 else 0.0 + return (s / mm * 100.0) if mm else 0.0 +mA, mB = sum(A) / n, sum(B) / n +cvA, cvB = cv(A), cv(B) +# reconstruct execution order (odd pair: A,B ; even pair: B,A) and normalize each +# run by its binary's mean so the A/B offset drops out, leaving pure machine drift. +seq = [] +for i in range(n): + seq += ([('A', A[i]), ('B', B[i])] if (i + 1) % 2 else [('B', B[i]), ('A', A[i])]) +nrm = [(t / (mA if lbl == 'A' else mB) - 1) * 100 for lbl, t in seq] +N = len(nrm); mi = (N - 1) / 2.0; mn = sum(nrm) / N +denom = sum((i - mi) ** 2 for i in range(N)) +slope = (sum((i - mi) * (nrm[i] - mn) for i in range(N)) / denom) if denom else 0.0 +half = N // 2 +drift_shift = sum(nrm[half:]) / (N - half) - sum(nrm[:half]) / half + +print(f"\n=== {HDR} (improvement: - = {LA} faster) ===") +print(f" pairs: {n} mean A ({LA}): {sum(A)/n:.3f}s mean B ({LB}): {sum(B)/n:.3f}s") +print() +print(f" [parametric] paired-t mean {mean:+.2f}% sd {sd:.2f}% se {se:.2f}%") +print(f" 95% CI: [{lo:+.2f}%, {hi:+.2f}%] (t df={df} = {tc})") +pstr = f"{p:.4f}" if p >= 1e-4 else f"{p:.1e}" +print(f" [robust] median {med:+.2f}% Wilcoxon W+={Wp:.0f} W-={Wn:.0f} p(exact)={pstr} (z={z:+.2f})") +print() +print(" --- server stability (this run; compare across servers) ---") +print(f" run-to-run jitter: A CV {cvA:.2f}% B CV {cvB:.2f}% (lower = steadier)") +print(f" within-session drift: {slope * N:+.2f}% over the run, 1st->2nd half {drift_shift:+.2f}%") +print(f" (jitter -> Tier-1 cached gate floor; drift -> whether the cached baseline can be trusted)") +print() +if hi < 0 and p < 0.05: + print(f" VERDICT: REAL IMPROVEMENT - {LA} faster by ~{-mean:.2f}% (t-CI and Wilcoxon agree)") +elif lo > 0 and p < 0.05: + print(f" VERDICT: REAL REGRESSION - {LA} slower by ~{mean:.2f}% (t-CI and Wilcoxon agree)") +elif (hi < 0) != (p < 0.05): + print(f" VERDICT: BORDERLINE - parametric and robust disagree; suspect outlier pair(s).") + print(f" Trust the median ({med:+.2f}%); add pairs or inspect the per-pair list.") +else: + print(f" VERDICT: INCONCLUSIVE - effect not separable from 0 at n={n}.") + print(f" Point estimate ~{med:+.2f}% (median). Need more pairs to resolve.") +print(f"\n raw pairs: {path}") +PY +}