Implement Slice Ten finding selection performance

[sidmohan0]

Objective

Fix quadratic finding selection on dense, disjoint input in the shared Rust transformation path. Addresses both repeated overlap scans and linear duplicate lookup from #10 while preserving selected findings, provenance, validation errors, and source order.

Closes #10.

Changes

• Index duplicate groups by validated entity type and byte range, preserving the original encounter-order preference within each group.
• Return disjoint findings after source sorting; otherwise sort by preference once and use a BTreeMap to check neighboring accepted intervals. These paths take O(m log m) selection comparisons with O(m) additional index storage.
• Preserve the original overlap algorithm when equal-length groups mix present and absent confidence. The existing preference can be cyclic, so this compatibility fallback remains quadratic. All current built-in detectors, including structured PERSON, use the faster path.
• Add differential regression tests and a reproducible release-mode benchmark, with results and limitations in docs/finding-selection-performance.md.

At 4,096 input findings, local selection-only medians were:

Workload	Previous	New	Speedup
Disjoint	60.29 ms	0.56 ms	107×
Overlap clusters	51.02 ms	0.88 ms	58×
Duplicate groups	8.11 ms	0.31 ms	26×

Both implementations ran in the same release build on macOS ARM64, using seven alternating samples. These measurements exclude scanning, validation, output construction, bindings, and provider I/O; they are not end-to-end speedup claims.

Package versions remain at 0.2.0. Prepare the combined PERSON + performance 0.3.0 release separately after review.

Verification

• cargo fmt --all --check
• cargo clippy --workspace --all-targets --all-features -- -D warnings
• cargo test --workspace --all-features — 81 passed; manual benchmark ignored in the normal suite
• cargo test -p datafog-core --release finding_selection_benchmark -- --ignored --nocapture
• 9,000 randomized reference comparisons plus confidence-cycle, duplicate-order, validation, and interval-boundary regressions
• Installed Python wheel, packed Node package, and browser WASM package conformance tests
• Engineering documentation links resolve
• No detection semantics changed; existing positive and negative fixtures pass
• Reviewed the diff for unrelated refactoring or duplicated binding logic; no new dependencies or public API changes

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
data-fog	🟢 Ready	View Preview	Sep 4, 2026, 11:49 PM

💡 Tip: Enable Automations to automatically generate PRs for you.