Route markdown documents through the structural extraction pass in the skill pipeline - #3614
ayushcodes10 wants to merge 3 commits into
Conversation
Fixes issue 2383. Part A only ever fed the detect step's code category into extract(), so a markdown file went to Part B's semantic pass alone. extract() already knows how to parse a markdown file's heading tree and links deterministically at zero token cost, the exact same treatment code already gets, and the CLI's own AST only update path already runs it this way. Nothing routed a document there in the skill pipeline, so its structural layer was silently absent everywhere but that one path. Every markdown shaped document extension extract() already dispatches on its own, md, mdx, qmd, and skill, now also joins the same extract call code files go through, in addition to its existing semantic pass, not instead of it. Every other document extension, plain text, rst, html, yaml, and converted office files, is unaffected. Regenerated every platform skill artifact from the updated shared fragment; every skillgen guard still passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Checks that Part A's file selection reads the document category, not just code, filters it to the extensions extract() already dispatches as markdown, and still funnels everything through the same single extract call code files use rather than a second, separate path that could drift out of sync with it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
Routes markdown-shaped documents (.md/.mdx/.qmd/.skill) through the deterministic AST pass in every skill host's Part A, so extract() builds their heading tree and contains/references edges at zero token cost alongside the existing semantic pass, instead of leaving those documents to the LLM alone. Renames the collector from code_files to ast_files and pulls document entries from the detect manifest, matching what the AST-only graphify update CLI path already did. Regenerates the expected skill fixtures and adds tests asserting the new wording and step-body parity across hosts.
No blocking issues surfaced. 14 lower-confidence candidates did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 1076 functions depend on the 1076 functions this change touches.
Health — this change adds coupling hotspots:
- new:
test_audit_catches_a_dropped_non_allowlisted_heading()— 0 callers, 6 callees
Verification — 1076 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 1076 function(s) in the blast radius were not formally verified this run
Test selection
Test selection
286 of 286 test file(s) selected (100%) via static blast radius.
Escalated to a full run for safety — the selection is not trustworthy on its own (see below). CI should run the whole suite.
tests/test_affected_cli.py— full-run-safetytests/test_affected_member_seed.py— full-run-safetytests/test_agents_platform.py— full-run-safetytests/test_analyze.py— full-run-safetytests/test_anthropic_custom_endpoint.py— full-run-safetytests/test_antigravity_install.py— full-run-safetytests/test_apm_fallback_version.py— full-run-safetytests/test_architecture_doc.py— full-run-safetytests/test_astro_extraction.py— full-run-safetytests/test_astro_import_ids.py— full-run-safetytests/test_atomic_canvas_export.py— full-run-safetytests/test_atomic_version_stamp.py— full-run-safetytests/test_atomic_writes.py— full-run-safetytests/test_backend_env_isolation.py— full-run-safetytests/test_backend_extras.py— full-run-safetytests/test_benchmark.py— full-run-safetytests/test_benchmark_raw_graph.py— full-run-safetytests/test_build.py— full-run-safetytests/test_build_merge_dedup_scope.py— full-run-safetytests/test_build_merge_hyperedges_and_prune.py— full-run-safetytests/test_build_merge_shrink_guard.py— full-run-safetytests/test_builtin_global_type_refs.py— full-run-safetytests/test_cache.py— full-run-safetytests/test_callflow_html.py— full-run-safetytests/test_cargo_introspect.py— full-run-safetytests/test_carried_hyperedge_remap.py— full-run-safetytests/test_case_sensitive_resolution.py— full-run-safetytests/test_charmap_encoding.py— full-run-safetytests/test_chunking.py— full-run-safetytests/test_cjs_module_extension.py— full-run-safetytests/test_claude_cli_backend.py— full-run-safetytests/test_claude_md.py— full-run-safetytests/test_cli_broken_pipe.py— full-run-safetytests/test_cli_export.py— full-run-safetytests/test_cli_help.py— full-run-safetytests/test_cluster.py— full-run-safetytests/test_codebuddy.py— full-run-safetytests/test_community_hub_labels.py— full-run-safetytests/test_community_labels_skill.py— full-run-safetytests/test_confidence.py— full-run-safetytests/test_corrupt_graph_json.py— full-run-safetytests/test_cpp_nested_and_cli.py— full-run-safetytests/test_cpp_objc_cross_file_calls.py— full-run-safetytests/test_cpp_preprocess.py— full-run-safetytests/test_cross_extension_reexport_self_cycle.py— full-run-safetytests/test_cross_language_call_resolution.py— full-run-safetytests/test_cross_repo_external_call_guards.py— full-run-safetytests/test_cross_repo_member_calls.py— full-run-safetytests/test_cross_repo_shared_types.py— full-run-safetytests/test_csharp_call_site_generic_args.py— full-run-safety- … and 236 more
non-code file(s) changed (
CHANGELOG.md,graphify/skill-agents.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md…) → running the full suite for safety (a code graph can't see config/fixture/data deps)
changed code file(s) with no mapped test (
CHANGELOG.md,graphify/skill-agents.md,graphify/skill-amp.md,graphify/skill-claw.md,graphify/skill-codex.md…) — a coverage gap or a missing link — running the full suite rather than only the selected tests
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
· 1 more finding(s) on lines outside this diff (see the check run).
1 participant
Summary
Fixes #2383.
graphify/extractors/markdown.pyis a deterministic, zero-token markdown extractor that emits heading nodes,containsedges for heading nesting, andreferencesedges for links — all atEXTRACTEDconfidence. In the/graphifyskill pipeline it never ran on.mdfiles: Part A's structural (AST) extraction only ever fed the detection step'scodecategory intoextract(), while.mdfiles classify asdocumentand route to Part B's semantic (LLM) pass alone. Code gets both layers (AST baseline + semantic); documents only ever got one.This is the same routing gap the CLI's own
graphify updatepath already closed for its.md-only case (#2014) — but that fix never touched the skill pipeline, which has its own separate file-selection logic in Part A.Fix
Part A's file selection now also includes markdown-shaped documents (
.md,.mdx,.qmd,.skill— exactly the extensionsextract()already dispatches toextract_markdownon its own) from thedocumentdetection category, funneling them through the sameextract()call code files already go through. This is purely additive: documents still go to Part B for semantic extraction exactly as before, and every other document type (.txt,.rst,.html,.yaml, converted Office files) is untouched, sinceextract()doesn't know how to parse those anyway.Test plan
tests/test_skillgen.py: Part A's rendered file-selection logic reads thedocumentcategory (not justcode), filters it to the markdown extensionsextract()recognizes, and routes everything through the single sharedextract()call rather than a second, separately-maintained path.python3 -m tools.skillgen --check/--audit-coverage/--monolith-roundtrip: all OK.🤖 Generated with Claude Code