Compact completed items at their first lifecycle position

[ymichael]

Human comments

What was wrong

Settled item lifecycles retain separate starts and deltas after PR1 pruning. This implements the revised first-position PR2 design independently of #3802, preserving visible history while removing redundant physical rows. Rebased onto 0bb64f3789, including main's reader fixes #3874 and #3876.

The earlier implementation also counted compacted physical rows against timeline work limits, causing some pages to reconstruct much more history and read slower. The final reader counts reconstructed records, fetches metadata alongside selected rows, avoids duplicate parsing, and reuses reconstruction within explicit memory-accounting limits.

What changed

• Keep item/completed, its completion ID, payload and original timestamp; move its physical sequence to the first eligible lifecycle position. One versioned internal metadata column preserves original ordering, timing and necessary start/delta content. Eligible completed command deltas may become empty timing markers.
• Compact only unique, compatible, settled lifecycles. Skip ambiguous, oversized, late and user/context-boundary crossings. Also skip assistant lifetimes crossing another assistant completion or manager output, preserving latest-output selection.
• Use bounded discovery and atomic progress in the existing idle background sweep. Compaction does not run in synchronous ingestion/turn-completion cleanup. Reuse PR1 rewrite generation and notifications to refresh open timelines.
• Reconstruct only selected owners for timeline projection and fork inheritance. Charge embedded records to the timeline work budget. Reuse parsed payloads and a per-connection reconstruction cache capped at 8 million accounted text characters / 10,000 reconstructed records; hits require identical metadata and completion payloads. Current scope and snapshot filtering apply after reuse. These are accounting limits, not a byte-exact heap cap.
• Raw SDK/CLI/JSON exports return physical records; deleted start/delta IDs and old raw positions are not preserved. Human timelines preserve visible ordering, output, timing and edit cards. Existing completion ownership and edit/fork rules remain.
• No new tables or indexes, scheduler, allocator, virtual raw-event lookup layer, daemon protocol change or SQLite checkpoint-setting change. Generated migration 0127 adds only the internal metadata column.

Design and raw contract · Verification and limitations.

How you verified

• Final source: 611 DB tests, 198 affected server tests, DB/server typechecks pass through Turbo. Added checks cover logical work limits, inline metadata, parsed payloads, malformed JSON, cache invalidation, scope/snapshot filtering, eviction and oversized bypass. Existing tests cover eligibility, atomicity, output ownership, rewriting/caches and idle-only scheduling.
• Actual worker on the full sanitized copy: 1,001,984 events / 1,001,982 net rows removed; 1,223,270,400 allocated bytes saved. File remains 6,006,726,656 bytes; freed pages are reusable. No new tables or indexes.
• Independent full-copy verification: all 2,326 complete timelines and all 2,082,639 logical records match except 119,716 authorized empty command markers. Highwater, provider recovery, existing side tables and outputs match. Initial synthetic UI verification covered rewrite refresh, output/diff expansion, timing and reload; fork/rewind checks are documented. No provider-resume or iOS claim.
• Complete reader traversals, not a 30-page sample: all 3,027 product-setting pages and all 2,733 expanded stress pages across every thread match main exactly, including cursors and context. Product settings use a 1,500-event budget, 20 segments, lazy nested rows and 32,000 inline output characters; stress uses a 10,000-event budget and expanded nested rows.
• Three alternating warm samples per version/request: summed page medians improve 12.1% for product settings and 11.4% for stress; median paired improvement is about 1.1 ms in both. No warm elapsed case exceeds both 5 ms and 10% overhead. The two originally blocking large pages improve 214.11 → 200.16 ms and 191.56 → 151.21 ms in repeated stress checks.
• Repeated all 535 selected noisy/slow/control requests, six samples per version and cold/warm application-cache mode: all responses match and no elapsed case exceeds both thresholds. Three remaining CPU-only outliers improve in twelve-trial repeats. First-call results and all original samples remain documented; this is not a promise that every first read is faster.
• Before backfill, both versions read the same uncompacted copy for 45 large-thread requests, nine samples per version/mode. All responses match; median added elapsed time is 0.88 ms application-cold / 0.76 ms warm, with no elapsed case above both thresholds. Summed warm elapsed medians add 2.7%, CPU 3.4%; two CPU-only outliers remain. This small pre-backfill cost is explicitly retained in the report.

The final timing harness uses the same ES-module Drizzle and native SQLite drivers as the app and asserts constructor identity. Earlier harnesses mixed a CommonJS driver with an ES-module schema, so their timing figures are superseded. Measurements cover the complete server timeline builder, excluding HTTP/browser rendering. Application-cold means empty per-connection JS caches, not cold disk. Results use private full database copies and production SQLite cache/mmap settings on a shared host; they are not traffic-weighted latency guarantees.

Maintenance remains asynchronous, with existing SQLite limitations. The historical worker pass took 304.5 seconds of processing; a short scheduling sample estimated about 107 idle hours for catch-up, not an end-to-end guarantee. Compaction performs no synchronous ingestion writes. Historical samples still observed 322 ms live / 372 ms background stalls; a controlled replay isolated checkpoint cost. No checkpoint setting was changed and no sub-50 ms cap is claimed. This reader performance pass does not establish new maintenance bounds.

Evidence: parent thread thr_vmdgc3ke5y, pr2-perf-fix/report.md, final production-* scripts/results and test logs; original worker evidence in pr2-review/final. Both copies are mode 0600 and contain zero Connect records across seven plugin-ID tables. No live database or copied Connect credentials/configuration were used. No merge or deployment.

AGENT GENERATED