From 00d162073cd0f83364aa3bf0b004f8bc29341ec9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 9 Aug 2026 23:14:42 +0000 Subject: [PATCH] Fix CUDA-Linux build retry: disable ggml's own GGML_CCACHE auto-detect The regex fix in the previous commit (8efd38e) correctly triggered the uncached retry on CUDA 13.3's sccache/nvcc fatbinary failure, but the retry still failed identically (verified live on run 31340386884): ggml's own CMakeLists.txt self-enables ccache/sccache (GGML_CCACHE, default ON) whenever it finds one on PATH and our CMAKE_C_COMPILER_LAUNCHER /CMAKE_CXX_COMPILER_LAUNCHER are unset -- exactly the retry's state, since sccache is still on PATH from the failed attempt. ggml wires itself in via the global RULE_LAUNCH_COMPILE CMake property, which wraps nvcc too, so the "uncached" retry silently re-enabled the very launcher it was trying to avoid. Pass -DGGML_CCACHE=OFF on the retry's cmake invocation so it is guaranteed genuinely uncached regardless of what's left on PATH. --- .github/build.sh | 10 +++++++++- CLAUDE.md | 18 ++++++++++++++---- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/.github/build.sh b/.github/build.sh index 95245f9e..37367025 100755 --- a/.github/build.sh +++ b/.github/build.sh @@ -150,7 +150,15 @@ if [ "$build_rc" -ne 0 ]; then echo "build.sh: build failed via an sccache cache error — retrying WITHOUT cache (clean reconfigure)." rm -f "$build_log" rm -rf build && mkdir -p build - cmake -Bbuild $@ || exit 1 + # -DGGML_CCACHE=OFF is required here, not just dropping our own $LAUNCH flags: ggml's own + # CMakeLists.txt self-enables ccache/sccache (GGML_CCACHE, default ON) whenever it finds one + # on PATH and CMAKE_C_COMPILER_LAUNCHER/CMAKE_CXX_COMPILER_LAUNCHER are unset — which is exactly + # this retry's state, since sccache is still on PATH from the failed attempt. It wires itself in + # via the global RULE_LAUNCH_COMPILE property (wraps nvcc too, not just C/C++), so without this + # flag the "uncached" retry silently re-enables the very launcher it's trying to avoid and fails + # identically. Confirmed live: the b10333/CUDA-13.3 retry hit the same fatbinary/.ptx failure + # because of this (see CLAUDE.md's sccache rollout section for the incident). + cmake -Bbuild -DGGML_CCACHE=OFF $@ || exit 1 cmake --build build --config Release -j"${JOBS}" || exit 1 LAUNCH="" # cache disabled for this run; skip the stats query below else diff --git a/CLAUDE.md b/CLAUDE.md index 5c903605..9f3b3381 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -531,10 +531,20 @@ v0.16.0 + the probe this is no longer a risk.) Job-by-job status: `fatbinary fatal: Could not open input file 'acc.compute_75.ptx'` immediately followed by `sccache: Compiler killed by signal 1`, on the very first `.cu` TU (a cold-cache miss, not a hit issue) — an sccache/nvcc incompatibility for `-virtual` architecture targets (e.g. `75-virtual`), - not a real compile error. The existing mid-build retry-without-cache mechanism (see below) didn't - catch it because its trigger regex didn't include this failure's wording; `build.sh`'s regex now - also matches `Compiler killed by signal`, so this failure mode falls back to an uncached, green - `-O3` build like every other sccache/nvcc incompatibility instead of redding the job. + not a real compile error. **Two fixes were needed, not one.** (1) The existing mid-build + retry-without-cache mechanism (see below) didn't catch it because its trigger regex didn't include + this failure's wording; `build.sh`'s regex now also matches `Compiler killed by signal`. (2) That + alone still wasn't enough — verified live on the very next dispatch (run 31340386884): the retry + fired but failed **identically**, because ggml's own `ggml/src/CMakeLists.txt` self-enables + ccache/sccache (`GGML_CCACHE`, default `ON`) whenever it finds one on `PATH` and + `CMAKE_C_COMPILER_LAUNCHER`/`CMAKE_CXX_COMPILER_LAUNCHER` are unset — exactly the retry's state, + since sccache is still on `PATH` from the failed attempt (build.sh only clears its own `$LAUNCH` + flags, not ggml's independent detection). ggml wires itself in via the **global** + `RULE_LAUNCH_COMPILE` CMake property, which wraps nvcc too, not just C/C++ — so the "uncached" + retry silently re-enabled the very launcher it was trying to avoid. Fix: the retry's `cmake + -Bbuild` now also passes `-DGGML_CCACHE=OFF`, so a genuinely uncached build is guaranteed + regardless of what's left on `PATH`. This fallback now falls back to a real, green `-O3` build + like every other sccache/nvcc incompatibility instead of redding the job. 3. `crosscompile-linux-aarch64` — ✅ **enabled**, now a **native `ubuntu-24.04-arm` build** (not dockcross): `build.sh` self-fetches the aarch64 static-musl sccache (the fetch block in `build.sh` maps `uname -m` → `x86_64`/`aarch64`) and the probe guards it. See "Linux aarch64: