Skip to content

[Fix] Green CI: skip signal-health benchmark when the dogfood DB is absent (regression from ae077b7) - #12

Merged
codenamev merged 1 commit into
mainfrom
fix/ci-benchmark-green
Jul 16, 2026
Merged

[Fix] Green CI: skip signal-health benchmark when the dogfood DB is absent (regression from ae077b7)#12
codenamev merged 1 commit into
mainfrom
fix/ci-benchmark-green

Conversation

@codenamev

Copy link
Copy Markdown
Owner

Problem

main CI (.github/workflows/main.yml./bin/run-evals) has been red since ae077b7 [Chore] Untrack machine-local .claude state — unrelated to any feature work. The failing step is the DevMemBench benchmark run (bundle exec rspec spec/benchmarks/ --tag benchmark), not the unit suite (bundle exec rake, which is green).

Root cause

The signal-health benchmark (spec/benchmarks/health/database_signal_spec.rb) asserts contracts against the live committed .claude/memory.sqlite3 (the gem's own dogfood DB). Its before(:all) guard skipped the suite only when the file existed but wasn't a SQLite file — i.e. an unresolved git-lfs pointer on CI's lfs: false checkout.

ae077b7 untracked the DB, so on CI it is now absent entirely:

  • File.exist?(project_db)false → the guard is bypassed (never skips).
  • The suite runs against a fresh empty DB created by ensure_project!.
  • "has more than 5 active project facts (sanity floor)" fails: "only 0 active project facts; suspect ingest broken or DB nuked".
  • rspec … --tag benchmark exits 1 → bin/run-evals prints SOME CHECKS FAILED (exit code: 1).

Fix

Skip the suite unless the DB is present AND a real SQLite file, covering both the absent (untracked/gitignored/fresh-clone) and lfs-pointer cases. These contracts only mean something against the live DB — they still run locally where it exists.

Verification

Reproduced and fixed by pointing CLAUDE_PROJECT_DIR/CLAUDE_CONFIG_DIR at empty dirs (CI's absent-DB condition):

  • Before: 10 examples, 1 failure (the sanity floor).
  • After: 10 examples, 0 failures, 10 pending (all skip).
  • LFS-pointer case (present-but-not-SQLite) also skips.
  • A present live DB still runs the suite (local validation unchanged).
  • Full DB/embedding-independent benchmark subset (health + resolution + fts5 + relevance) under absent-DB: 28 examples, 0 failures.

CI on this PR is the authoritative end-to-end check (it has working fastembed + the real absent-DB checkout). One-file change; no production code touched.

Note: the eval-scenario X/7 [FAIL] lines in bin/run-evals output are a cosmetic display artifact (it counts skipped real-mode examples as non-passed); they do not drive the exit code — only the benchmark step did. Left as-is to keep this fix minimal.

… just an LFS pointer

CI (`.github/workflows/main.yml` -> `./bin/run-evals`) has been red since
ae077b7 [Chore] Untrack machine-local .claude state. Root cause:

- The signal-health benchmark (spec/benchmarks/health/database_signal_spec.rb)
  runs against the live committed .claude/memory.sqlite3 and its before(:all)
  guard skipped the suite only when that file EXISTED but wasn't a SQLite file
  (an unresolved git-lfs pointer on CI's lfs:false checkout).
- ae077b7 untracked the DB, so on CI it is now ABSENT entirely. File.exist?
  is false, the guard is bypassed, and the suite runs against a fresh empty DB
  created by ensure_project! — where "has more than 5 active project facts
  (sanity floor)" fails with "only 0 active project facts; suspect ingest
  broken or DB nuked". `bundle exec rspec spec/benchmarks/ --tag benchmark`
  then exits 1, so bin/run-evals reports SOME CHECKS FAILED.

Fix: skip the suite unless the DB is present AND a real SQLite file, covering
both the absent (untracked/gitignored/fresh-clone) and lfs-pointer cases. These
contracts only mean something against the live DB; they still run locally.

Reproduced with CLAUDE_PROJECT_DIR/CLAUDE_CONFIG_DIR pointed at empty dirs
(1 failure -> 10 pending after the fix); LFS-pointer case also skips; a present
live DB still runs.
@codenamev
codenamev merged commit b03206c into main Jul 16, 2026
1 check passed
@codenamev
codenamev deleted the fix/ci-benchmark-green branch July 16, 2026 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant