Speed up indexing and large architecture graph loads

[Modsofthenation]

Indexing and workspace graph loads were dominated by walking node_modules (and other skip trees) with Path.rglob("*"), then re-running every architecture rule on each GET /api/architecture. That is what makes a large installed JS/Django tree feel stuck on both Index and Architecture.

https://github.com/Modsofthenation/grapejs-editor is not reachable from this environment (404, including with the repo token), so measurements used a synthetic 40-app Django + 40-feature React tree with ~8k node_modules files — the same failure mode as an installed GrapesJS editor — plus the public EdgeTX-Dashboards and GrapesJS/grapesjs checkouts.

Measured (synthetic large monorepo)

Step	Before	After
Source scan (iter_source_files)	242 ms	4 ms
Full index_repo	1084 ms	464 ms
summarize_index (rules + drift)	307 ms	25 ms
architecture_report without graph	326 ms	40 ms
architecture_report with graph	336 ms	52 ms

Same graph size after the change: 3801 nodes / 4640 edges (architecture view 1760 / 1960). Minified vendor.min.js is no longer indexed.

On EdgeTX-Dashboards and grapesjs (no node_modules in the shallow clone) index stays in the ~300 ms range; architecture load drops because findings are cached.

What changed

• Pruned walks (loadpath.scan): never descend into node_modules, .git, dist, .next, etc. Used by index, detect, OpenAPI discovery, and Django boot.
• Skip junk sources: .min.js / .bundle.js / .d.ts and files over 1 MB.
• Cache findings at index time (findings_json). Workspace load reuses them unless loadpath.yml changed.
• Batch SQLite upserts with in-graph merge so Celery/Dramatiq definitions still win over call-site placeholders.
• Faster extract (line index instead of source[:pos].count("\\n"), no ast.dump for queryset names) and O(1) route stitch by URL template.
• Large 2D graphs use onlyRenderVisibleElements and fewer layout passes.

Tests: tests/unit/test_index_scan.py locks the prune/skip/cache behavior. Full unit + integration + non-Playwright e2e passed locally; UI vitest 91/91.

  

[coderabbitai]

Important

Review available on request

• 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 122d61aa-a775-4db8-bbe8-a934a5882af2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}