diff --git a/.github/build.sh b/.github/build.sh index 6257904b..95245f9e 100755 --- a/.github/build.sh +++ b/.github/build.sh @@ -137,12 +137,16 @@ cmake -Bbuild $LAUNCH $@ || exit 1 # still fails fast (and is reported) instead of triggering a wasteful uncached rebuild. The # "Compiler not supported" signature additionally covers the CUDA case: if wrapping nvcc breaks # (sccache declining/erroring on the nvcc driver), the retry rebuilds the full-arch CUDA job -# without any launcher rather than redding it. +# without any launcher rather than redding it. "Compiler killed by signal" covers a second CUDA +# failure mode seen with CUDA 13.3's nvcc: sccache's nvcc wrapping loses an intermediate .ptx file +# for a "-virtual" architecture target (e.g. 75-virtual), so fatbinary aborts with "Could not open +# input file '*.ptx'" and sccache reports the underlying nvcc invocation as killed — an sccache/nvcc +# incompatibility, not a real compile error, so it gets the same uncached-retry treatment. build_log="$(mktemp 2>/dev/null || echo "/tmp/jllama-build.$$.log")" cmake --build build --config Release -j"${JOBS}" 2>&1 | tee "$build_log" build_rc=${PIPESTATUS[0]} if [ "$build_rc" -ne 0 ]; then - if [ -n "$LAUNCH" ] && grep -qiE 'sccache: error|Server startup failed|cache storage failed|Compiler not supported' "$build_log"; then + if [ -n "$LAUNCH" ] && grep -qiE 'sccache: error|Server startup failed|cache storage failed|Compiler not supported|Compiler killed by signal' "$build_log"; then echo "build.sh: build failed via an sccache cache error — retrying WITHOUT cache (clean reconfigure)." rm -f "$build_log" rm -rf build && mkdir -p build diff --git a/CLAUDE.md b/CLAUDE.md index 5fef2328..5c903605 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -526,7 +526,15 @@ v0.16.0 + the probe this is no longer a risk.) Job-by-job status: **100%** on CUDA / CUBIN / device-code (139 CUDA hits, 99.86% overall, 3 misses), cutting the job from **~51 min cold to ~15 min warm**. The first-run debug diagnostics (`SCCACHE_LOG` / `SCCACHE_ERROR_LOG` / `RUST_BACKTRACE`) were dropped once confirmed; `sccache --show-stats` still - prints the hit table every run. + prints the hit table every run. **CUDA 13.3 regression (caught on the b10333/CUDA-13.3 publish + dispatch, run 31339594933):** nvcc 13.3's device-compile pipeline broke sccache's nvcc wrapping — + `fatbinary fatal: Could not open input file 'acc.compute_75.ptx'` immediately followed by + `sccache: Compiler killed by signal 1`, on the very first `.cu` TU (a cold-cache miss, not a hit + issue) — an sccache/nvcc incompatibility for `-virtual` architecture targets (e.g. `75-virtual`), + not a real compile error. The existing mid-build retry-without-cache mechanism (see below) didn't + catch it because its trigger regex didn't include this failure's wording; `build.sh`'s regex now + also matches `Compiler killed by signal`, so this failure mode falls back to an uncached, green + `-O3` build like every other sccache/nvcc incompatibility instead of redding the job. 3. `crosscompile-linux-aarch64` — ✅ **enabled**, now a **native `ubuntu-24.04-arm` build** (not dockcross): `build.sh` self-fetches the aarch64 static-musl sccache (the fetch block in `build.sh` maps `uname -m` → `x86_64`/`aarch64`) and the probe guards it. See "Linux aarch64: