[Fix] Green CI: skip signal-health benchmark when the dogfood DB is absent (regression from ae077b7) - #12
Merged
Merged
Conversation
… just an LFS pointer CI (`.github/workflows/main.yml` -> `./bin/run-evals`) has been red since ae077b7 [Chore] Untrack machine-local .claude state. Root cause: - The signal-health benchmark (spec/benchmarks/health/database_signal_spec.rb) runs against the live committed .claude/memory.sqlite3 and its before(:all) guard skipped the suite only when that file EXISTED but wasn't a SQLite file (an unresolved git-lfs pointer on CI's lfs:false checkout). - ae077b7 untracked the DB, so on CI it is now ABSENT entirely. File.exist? is false, the guard is bypassed, and the suite runs against a fresh empty DB created by ensure_project! — where "has more than 5 active project facts (sanity floor)" fails with "only 0 active project facts; suspect ingest broken or DB nuked". `bundle exec rspec spec/benchmarks/ --tag benchmark` then exits 1, so bin/run-evals reports SOME CHECKS FAILED. Fix: skip the suite unless the DB is present AND a real SQLite file, covering both the absent (untracked/gitignored/fresh-clone) and lfs-pointer cases. These contracts only mean something against the live DB; they still run locally. Reproduced with CLAUDE_PROJECT_DIR/CLAUDE_CONFIG_DIR pointed at empty dirs (1 failure -> 10 pending after the fix); LFS-pointer case also skips; a present live DB still runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
mainCI (.github/workflows/main.yml→./bin/run-evals) has been red sinceae077b7[Chore] Untrack machine-local .claude state — unrelated to any feature work. The failing step is the DevMemBench benchmark run (bundle exec rspec spec/benchmarks/ --tag benchmark), not the unit suite (bundle exec rake, which is green).Root cause
The signal-health benchmark (
spec/benchmarks/health/database_signal_spec.rb) asserts contracts against the live committed.claude/memory.sqlite3(the gem's own dogfood DB). Itsbefore(:all)guard skipped the suite only when the file existed but wasn't a SQLite file — i.e. an unresolved git-lfs pointer on CI'slfs: falsecheckout.ae077b7untracked the DB, so on CI it is now absent entirely:File.exist?(project_db)→false→ the guard is bypassed (never skips).ensure_project!."has more than 5 active project facts (sanity floor)"fails: "only 0 active project facts; suspect ingest broken or DB nuked".rspec … --tag benchmarkexits 1 →bin/run-evalsprintsSOME CHECKS FAILED (exit code: 1).Fix
Skip the suite unless the DB is present AND a real SQLite file, covering both the absent (untracked/gitignored/fresh-clone) and lfs-pointer cases. These contracts only mean something against the live DB — they still run locally where it exists.
Verification
Reproduced and fixed by pointing
CLAUDE_PROJECT_DIR/CLAUDE_CONFIG_DIRat empty dirs (CI's absent-DB condition):10 examples, 1 failure(the sanity floor).10 examples, 0 failures, 10 pending(all skip).28 examples, 0 failures.CI on this PR is the authoritative end-to-end check (it has working fastembed + the real absent-DB checkout). One-file change; no production code touched.