Skip to content

Fix CUDA-Linux build: retry without sccache on nvcc device-compile crash #385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bernardladenthin merged 1 commit into from
Aug 9, 2026
Merged

Fix CUDA-Linux build: retry without sccache on nvcc device-compile crash #385

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,12 +137,16 @@ cmake -Bbuild $LAUNCH $@ || exit 1
# still fails fast (and is reported) instead of triggering a wasteful uncached rebuild. The
# "Compiler not supported" signature additionally covers the CUDA case: if wrapping nvcc breaks
# (sccache declining/erroring on the nvcc driver), the retry rebuilds the full-arch CUDA job
# without any launcher rather than redding it.
# without any launcher rather than redding it. "Compiler killed by signal" covers a second CUDA
# failure mode seen with CUDA 13.3's nvcc: sccache's nvcc wrapping loses an intermediate .ptx file
# for a "-virtual" architecture target (e.g. 75-virtual), so fatbinary aborts with "Could not open
# input file '*.ptx'" and sccache reports the underlying nvcc invocation as killed — an sccache/nvcc
# incompatibility, not a real compile error, so it gets the same uncached-retry treatment.
build_log="$(mktemp 2>/dev/null || echo "/tmp/jllama-build.$$.log")"
cmake --build build --config Release -j"${JOBS}" 2>&1 | tee "$build_log"
build_rc=${PIPESTATUS[0]}
if [ "$build_rc" -ne 0 ]; then
if [ -n "$LAUNCH" ] && grep -qiE 'sccache: error|Server startup failed|cache storage failed|Compiler not supported' "$build_log"; then
if [ -n "$LAUNCH" ] && grep -qiE 'sccache: error|Server startup failed|cache storage failed|Compiler not supported|Compiler killed by signal' "$build_log"; then
echo "build.sh: build failed via an sccache cache error — retrying WITHOUT cache (clean reconfigure)."
rm -f "$build_log"
rm -rf build && mkdir -p build
Expand Down
10 changes: 9 additions & 1 deletion CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -526,7 +526,15 @@ v0.16.0 + the probe this is no longer a risk.) Job-by-job status:
**100%** on CUDA / CUBIN / device-code (139 CUDA hits, 99.86% overall, 3 misses), cutting the job
from **~51 min cold to ~15 min warm**. The first-run debug diagnostics (`SCCACHE_LOG` /
`SCCACHE_ERROR_LOG` / `RUST_BACKTRACE`) were dropped once confirmed; `sccache --show-stats` still
prints the hit table every run.
prints the hit table every run. **CUDA 13.3 regression (caught on the b10333/CUDA-13.3 publish
dispatch, run 31339594933):** nvcc 13.3's device-compile pipeline broke sccache's nvcc wrapping —
`fatbinary fatal: Could not open input file 'acc.compute_75.ptx'` immediately followed by
`sccache: Compiler killed by signal 1`, on the very first `.cu` TU (a cold-cache miss, not a hit
issue) — an sccache/nvcc incompatibility for `-virtual` architecture targets (e.g. `75-virtual`),
not a real compile error. The existing mid-build retry-without-cache mechanism (see below) didn't
catch it because its trigger regex didn't include this failure's wording; `build.sh`'s regex now
also matches `Compiler killed by signal`, so this failure mode falls back to an uncached, green
`-O3` build like every other sccache/nvcc incompatibility instead of redding the job.
3. `crosscompile-linux-aarch64` — ✅ **enabled**, now a **native `ubuntu-24.04-arm` build** (not
dockcross): `build.sh` self-fetches the aarch64 static-musl sccache (the fetch block in
`build.sh` maps `uname -m` → `x86_64`/`aarch64`) and the probe guards it. See "Linux aarch64:
Expand Down
Loading