fix(llm): distinguish failed chunks from model omissions - #3617
GREYGROUPJP wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Formal verification. No changes could be formally verified in this run.
Graphify review — findings
Splits the corpus-parallel uncovered-files warning into two distinct messages: files whose semantic chunk raised before returning a result now get a "failed before returning a usable result" warning that points at the chunk errors above, while files the model genuinely returned but omitted keep the "returned a response but omitted them" message. Tracks which paths belonged to failed chunks via a new failed_files set so the two cases are classified correctly at reconciliation time.
No blocking issues surfaced. 2 lower-confidence candidates did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 913 functions depend on the 270 functions this change touches.
Health — this change adds coupling hotspots:
- new:
deduplicate_entities()— 77 callers, 24 callees - new:
build_merge()— 76 callers, 14 callees - new:
extract_files_direct()— 17 callers, 20 callees - new:
build()— 52 callers, 6 callees - new:
extract_corpus_parallel()— 27 callers, 11 callees - new:
_call_claude_cli()— 33 callers, 9 callees - new:
dispatch_command()— 2 callers, 125 callees - new:
_extract_with_adaptive_retry()— 22 callers, 10 callees - …and 16 more — each is listed as a finding
Verification — 913 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 578 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
41 of 286 test file(s) selected (14%) via static blast radius.
tests/test_backend_env_isolation.py— impacttests/test_backend_extras.py— impacttests/test_build.py— impacttests/test_build_merge_dedup_scope.py— impacttests/test_build_merge_hyperedges_and_prune.py— impacttests/test_build_merge_shrink_guard.py— impacttests/test_carried_hyperedge_remap.py— impacttests/test_charmap_encoding.py— impacttests/test_chunking.py— impact, changed-testtests/test_claude_cli_backend.py— impacttests/test_corrupt_graph_json.py— impacttests/test_cross_extension_reexport_self_cycle.py— impacttests/test_dedup.py— impacttests/test_dedup_remaps_hyperedges.py— impacttests/test_dedup_survivor_richness.py— impacttests/test_evidence_binding.py— impacttests/test_file_slice.py— impacttests/test_global_graph.py— impacttests/test_go_qualified_resolution.py— impacttests/test_hyperedge_member_shapes.py— impacttests/test_image_vision.py— impacttests/test_injection_sentinel_coverage.py— impacttests/test_issue_3472_source_file_collision.py— impacttests/test_label_retry.py— impacttests/test_labeling.py— impacttests/test_llm_backends.py— impacttests/test_llm_parser.py— impacttests/test_llm_parser_reasoning.py— impacttests/test_no_dedup_flag.py— impacttests/test_non_string_node_ids.py— impacttests/test_ollama.py— impacttests/test_ollama_retry_cap.py— impacttests/test_oversized_document_slicing.py— impacttests/test_partial_cache.py— impacttests/test_pdf_slicing.py— impacttests/test_pdf_token_estimate.py— impacttests/test_provider_registry.py— impacttests/test_prs.py— impacttests/test_prune_sweeps_orphans.py— impacttests/test_semantic_fragment_sanitize.py— impacttests/test_unverified_semantic_shrink.py— impact
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
Formal verification
Could not verify: Could not verify extract\_corpus\_parallel.
The verifier did not have enough to check extract\_corpus\_parallel, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: no capturable inputs from the test suite; property tier: parameter `root` is annotated `Path` — outside the synthesizable primitive/collection set
· 24 more finding(s) on lines outside this diff (see the check run).
Problem
When a semantic chunk raises before returning a usable result, the final reconciliation warning says the model returned a response but omitted the affected files. This contradicts the preceding chunk error and points operators toward model-output debugging instead of the dependency or provider failure that actually occurred.
Change
Track the files assigned to failed chunks and split uncovered files into two groups:
Each group now receives an accurate warning. The existing omission wording remains unchanged for successful responses that omit files.
Fixes #3616.
Validation
pytest tests/ -q --tb=short: 5,643 passed, 96 skipped.ruff check graphify/llm.py tests/test_chunking.py: passed.git diff --check: passed.