fix(eval): build the vector index so semantic benchmarks can report a number; stop tests writing to $HOME

[nadiadatepe-eng]

Pull Request

Linked issue

Closes #711 — agent_baseline / search_quality / multi_hop_retrieval always report no_graph_results, because the eval framework never builds the vector index.

Closes #712 — running the test suite writes pytest temp paths into the developer's real ~/.code-review-graph/registry.json.

Both were found while reproducing the README benchmarks, not from a user report.

What & why

Two independent fixes, one commit each.

1. The eval framework never built the vector index

run_eval builds the graph and calls run_post_processing(store), but that step's embedding refresh is a refresh, not a bootstrap: refresh_embeddings() returns early on a graph with no existing vectors — deliberately, so no build path can silently load a model or incur API cost. Nothing else in the eval framework populated the index.

So every benchmark that puts a natural-language question through hybrid_search — agent_baseline, search_quality, multi_hop_retrieval — fell through to FTS5, which scores a full sentence against no matching document and returns nothing. Every row came back status="no_graph_results", and aggregate() excludes those rows, so the run reported ok_rows: 0 / median: None rather than an error.

Reproduced on the shipped fastapi config. The graph itself was healthy — 6287 nodes, FTS5 populated with 6287 rows — but all three agent_questions returned 0 hits:

'How does include_router register routes on the application' -> 0 hits
'Where does APIRoute build its route handler'                -> 0 hits
'How does solve_dependencies resolve Depends parameters'     -> 0 hits

Bare identifiers worked fine ('APIRoute' → 5 hits), which is what makes the failure quiet: FTS5 is doing its job, it just can't match a sentence.

After building the index, the same three questions produce usable rows:

question	baseline_tokens	graph_tokens	ratio
include_router register routes	132479	4711	28.1x
APIRoute build route handler	192817	2339	82.4x
solve_dependencies / Depends	150384	2213	68.0x

This is likely why no agent_baseline CSVs exist under evaluate/results/ even though README.md cites the path evaluate/results/<repo>_agent_baseline_*.csv as the realistic-baseline measurement.

Changes:

• run_eval(embed=..., embedding_provider=..., embedding_model=...), plus _build_embedding_index(), which reads the graph through the runner's already-open GraphStore. Default off, so the cost invariant refresh_embeddings protects is unchanged.
• _warn_if_semantic_index_missing() runs before the benchmarks and names the affected ones, so the failure announces itself instead of arriving as an empty aggregate.
• agent_baseline.aggregate() reports no_graph_results_rows / no_baseline_match_rows. Excluded rows are counted, not just dropped — "no result" and "every query failed" had the same signature before.
• CLI: eval --embed / --embed-provider / --embed-model.

run_eval's new parameters are optional and trailing, so the signature stays source-compatible.

2. The test suite wrote into the developer's home directory

Running pytest left entries like this in the real ~/.code-review-graph/registry.json:

{"path": "/tmp/pytest-of-<user>/pytest-0/test_registry_data_dir_overrid0/project",
 "data_dir": "/tmp/pytest-of-<user>/pytest-0/.../external"}

Two routes reached it:

• Registry() defaults to ~/.code-review-graph/registry.json, and incremental.get_data_dir() constructs one internally (incremental.py:316). Tests covering data-dir resolution therefore both read and wrote the registry of whoever ran the suite. Besides the pollution, that made those tests machine-dependent: a developer with a registered repo could get a different result from one without.
• daemon.py built CONFIG_PATH, PID_PATH, STATE_PATH and DaemonConfig.log_dir from Path.home() as import-time constants.

The common root is that an import-time constant cannot be redirected afterwards — by the time a fixture could set anything, the value is frozen. So the paths resolve per call:

• constants.crg_home() reads $CRG_HOME, falling back to ~/.code-review-graph. Same convention as the existing CRG_DATA_DIR.
• registry.default_registry_path() and four daemon.default_*() helpers route through it.
• daemon.CONFIG_PATH / PID_PATH / STATE_PATH keep working via a PEP 562 module __getattr__, with __dir__ so they stay visible to introspection. Not covered: import * — the module defines no __all__, and adding one would change what the star exports for every other name.
• tests/conftest.py points $CRG_HOME at a tmp directory for every test. Autouse and unconditional: an opt-in fixture stops protecting a test the day someone forgets to request it.

Scoped deliberately — the editor-integration installers in skills.py write to ~/.codex, ~/.cursor and ~/.config/opencode, which are outside CRG state and whose tests already patch Path.home() themselves.

How it was tested

uv run pytest tests/ --tb=short -q
# 1980 passed, 5 skipped, 2 xpassed in 30.70s

uv run ruff check code_review_graph/
# All checks passed!

uv run mypy code_review_graph/ --ignore-missing-imports --no-strict-optional
# Success: no issues found in 66 source files

End-to-end, from a cold start with the graph database deleted:

# without --embed: the failure now announces itself
$ code-review-graph eval --benchmark agent_baseline --repo fastapi
  no vector index — agent_baseline will score natural-language questions
  against FTS5 alone and return zero hits. Re-run with --embed.

# with --embed: 3 of 3 rows usable
$ code-review-graph eval --benchmark agent_baseline --repo fastapi --embed
  28.1x / 82.4x / 68.0x

The home-directory fix was verified by deleting ~/.code-review-graph and running the full suite: it is no longer recreated. test_daemon.py was the file recreating it (bisected).

Each new test was checked against a reverted source tree to confirm it actually fails without the fix, rather than passing vacuously.

Note on reproducibility

docs/REPRODUCING.md states two runs on different machines produce identical numbers. That holds for the parsing/FTS/graph benchmarks, but the embedding-backed ratios drift slightly between runs on the same machine (82.3 → 82.4, 67.9 → 68.0). Not addressed here — flagging it in case you want the claim narrowed to the deterministic benchmarks.

Checklist

• Tests added for new functionality
• All tests pass: uv run pytest tests/ --tb=short -q
• Linting passes: uv run ruff check code_review_graph/
• Type checking passes: uv run mypy code_review_graph/ --ignore-missing-imports --no-strict-optional

[jordandunmire97-ai]

Idealization of what is prospering