Skip to content

Fix CUDA-Linux build retry: disable ggml's own GGML_CCACHE auto-detect - #386

Merged
bernardladenthin merged 1 commit into
mainfrom
claude/cuda-toolkit-action-update-0675dk
Aug 9, 2026
Merged

Fix CUDA-Linux build retry: disable ggml's own GGML_CCACHE auto-detect#386
bernardladenthin merged 1 commit into
mainfrom
claude/cuda-toolkit-action-update-0675dk

Conversation

@bernardladenthin

Copy link
Copy Markdown
Owner

Summary

  • Fix CUDA-Linux build: retry without sccache on nvcc device-compile crash #385's regex fix correctly triggered the uncached retry on CUDA 13.3's sccache/nvcc fatbinary failure, but verified live on the very next dispatch (run 31340386884) the retry failed identically
  • Root cause: ggml's own ggml/src/CMakeLists.txt self-enables ccache/sccache (GGML_CCACHE, default ON) whenever it finds one on PATH and CMAKE_C_COMPILER_LAUNCHER/CMAKE_CXX_COMPILER_LAUNCHER are unset — exactly the retry's state, since sccache is still on PATH from the failed attempt (build.sh only clears its own $LAUNCH flags, not ggml's independent detection)
  • ggml wires itself in via the global RULE_LAUNCH_COMPILE CMake property, which wraps nvcc too — so the "uncached" retry was silently re-enabling the very launcher it was trying to avoid
  • Fix: pass -DGGML_CCACHE=OFF on the retry's cmake -Bbuild invocation, guaranteeing a genuinely uncached build regardless of what's left on PATH
  • Updated the CLAUDE.md incident note (added on Fix CUDA-Linux build: retry without sccache on nvcc device-compile crash #385) to describe both fixes together

Test plan

  • Root cause confirmed live against the actual failing job logs (two consecutive dispatches, same failure signature both times)
  • CI is green on this branch (CUDA-Linux job on the retry path)
  • Docs (CLAUDE.md) updated

Related issues / PRs

Follow-up to #385, which fixed the retry trigger but not the retry's actual effectiveness

Checklist

  • I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md
  • My commits follow Conventional Commits
  • No security-sensitive changes

https://claude.ai/code/session_01CtT7KLNmC8gEoDS1EiGGwD


Generated by Claude Code

The regex fix in the previous commit (8efd38e) correctly triggered the
uncached retry on CUDA 13.3's sccache/nvcc fatbinary failure, but the
retry still failed identically (verified live on run 31340386884):
ggml's own CMakeLists.txt self-enables ccache/sccache (GGML_CCACHE,
default ON) whenever it finds one on PATH and our CMAKE_C_COMPILER_LAUNCHER
/CMAKE_CXX_COMPILER_LAUNCHER are unset -- exactly the retry's state,
since sccache is still on PATH from the failed attempt. ggml wires
itself in via the global RULE_LAUNCH_COMPILE CMake property, which
wraps nvcc too, so the "uncached" retry silently re-enabled the very
launcher it was trying to avoid.

Pass -DGGML_CCACHE=OFF on the retry's cmake invocation so it is
guaranteed genuinely uncached regardless of what's left on PATH.
@claude

claude Bot commented Aug 9, 2026

Copy link
Copy Markdown

Review Summary: This PR correctly fixes a subtle build system issue where ggml's independent GGML_CCACHE auto-detection was re-enabling the problematic sccache launcher on the retry path. The fix of passing -DGGML_CCACHE=OFF is well-targeted and properly documented. The solution is defensive, minimal, and verified live on the actual failing job. Ready to merge.

@claude

claude Bot commented Aug 9, 2026

Copy link
Copy Markdown

The fix in build.sh is well-implemented. Key observations:

  1. Line 161 correctly passes -DGGML_CCACHE=OFF to disable ggml's independent cache detection during the retry
  2. The comment (lines 153-160) is excellent - it clearly explains the root cause and why this flag is necessary
  3. The flag is placed at the right point: after the build directory is cleaned but before cmake configures the project
  4. This prevents ggml from auto-discovering sccache on PATH and wiring it back in via RULE_LAUNCH_COMPILE

The CLAUDE.md documentation update is comprehensive and explains both the immediate fix (regex for 'Compiler killed by signal') and this deeper fix (disabling GGML_CCACHE). The narrative clearly shows why two separate fixes were needed.

@sonarqubecloud

sonarqubecloud Bot commented Aug 9, 2026

Copy link
Copy Markdown

@bernardladenthin
bernardladenthin merged commit 8079cc6 into main Aug 9, 2026
11 of 17 checks passed
@bernardladenthin
bernardladenthin deleted the claude/cuda-toolkit-action-update-0675dk branch August 9, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants