Implement Slice Eleven text offset and record optimizations

[sidmohan0]

Objective

Reduce repeated text walks and field copying that remain after #13. On a local Node benchmark with 1,024 email findings in one Unicode field, a complete scan-and-transform call improves from 242.3 ms to 3.2 ms (75.6×) with identical findings and transformation results.

This is a candidate for inclusion in 0.3.0. Package versions remain at 0.2.0; no release is published here.

Changes

• Add a lazy Core TextIndex with one checkpoint per 256 code points. Reuse it for finding validation, scan offsets, and Node/WASM UTF-16 conversion, including unordered ranges and separate structured fields.
• Track output code-point positions incrementally during transformation.
• Convert Node structured transformation/restoration records directly using borrowed source/output fields instead of copying the output field for every record.
• Add Unicode/error regressions, dense installed-package checks, and a reproducible baseline/candidate Node benchmark. Document the new additive Rust helper and measurements in docs/bookkeeping-performance.md.

Existing operations, configuration, output shapes, error precedence, selection policies, and provider sequencing are preserved. No new dependencies or model assets are added. Reusing validation results across preparation/completion stages is deferred; this PR makes offset work reusable within each stage.

Measured tradeoffs

Same-process, alternating release builds on macOS ARM64 with Node 24; seven-round medians, full-result comparisons, provider I/O excluded. Baseline: the merge of #13 (083eaaab6bcfcfba9a31bc8ce1cfc99f1c2a3cae).

Combined scan-and-transform	Baseline	Candidate
One customer / 3 findings	0.0179 ms	0.0180 ms
100 customers / 300 findings	1.430 ms	1.454 ms
Long Unicode field / 1 finding	2.555 ms	1.345 ms
Dense Unicode / 512 findings	61.189 ms	1.564 ms
Dense Unicode / 1,024 findings	242.311 ms	3.204 ms
Dense ASCII / 1,024 findings	137.840 ms	3.265 ms

Small/customer operations were approximately 0–4% slower depending on the operation. The long sparse Unicode scan-only case was about 10% slower, although its combined operation improved. These are local synthetic measurements, not universal latency guarantees. Checkpoint memory grows with visited text; no process-level memory benchmark is claimed.

Verification

• cargo fmt --all --check
• cargo clippy --workspace --all-targets --all-features -- -D warnings
• cargo test --workspace --all-features — 84 passed, one existing manual benchmark ignored
• Installed Python wheel conformance tests
• npm run test:package --prefix bindings/node — includes dense Unicode byte/code-point/UTF-16 ranges, reversed findings, redaction/removal/emoji masking, and structured tokenization/restoration
• npm run test:package --prefix bindings/wasm — includes dense range checks in Chromium and TypeScript checks
• Baseline/candidate benchmark compares complete outputs for every workload and operation
• Mintlify build validation, public-document link checks, and engineering-note relative links
• Existing positive/negative detector fixtures remain unchanged and pass; no detection semantics change
• Diff reviewed for unrelated refactoring, duplicated detection logic, and new dependencies

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
data-fog	🟢 Ready	View Preview	Sep 5, 2026, 12:48 AM

💡 Tip: Enable Automations to automatically generate PRs for you.