feat: zstd compression

[0xbrayo]

closes #493

[greptile-apps]

Confidence Score: 5/5

[High impact] The PR appears safe to merge; the latest change fixes the previously reported overwrite.

Summary

Adds optional zstd compression using a dictionary trained from stored events.

• Event data takes less space when zstd can shrink it.
• Event queries and exports return JSON from compressed rows.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Startup without dictionary] --> B[Begin immediate transaction]
    B --> C[Read events and train dictionary]
    C -->|Usable dictionary| D[Store dictionary and rewrite events]
    D --> E[Commit]
    E --> F[Use dictionary]
    C -->|No dictionary or error| G[Roll back]

Loading

_{Reviews (4) · Last reviewed commit: "fix(datastore): take the write lock befo..." · Reviewed by Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 87.91209% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.31%. Comparing base (656f3c9) to head (3abdde1).
⚠️ Report is 218 commits behind head on master.

Files with missing lines	Patch %	Lines
aw-datastore/src/compression.rs	76.47%	4 Missing ⚠️
aw-datastore/src/datastore.rs	90.00%	3 Missing ⚠️
aw-datastore/src/export.rs	75.00%	2 Missing ⚠️
aw-datastore/tests/datastore.rs	94.28%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #618       +/-   ##
===========================================
+ Coverage   70.81%   85.31%   +14.49%     
===========================================
  Files          51       88       +37     
  Lines        2916    12555     +9639     
===========================================
+ Hits         2065    10711     +8646     
- Misses        851     1844      +993     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[0xbrayo]

🤖 Claude, on behalf of @0xbrayo

@greptile review

[0xbrayo]

🤖 Claude, on behalf of @0xbrayo

Benchmark: dictionary compression on a real-world database

Test data: byte-for-byte snapshot of a live database, 544,673 events across 19 buckets (~12.5 months of aw-watcher-window/afk/etc data).

Methodology:

1. Started from the v4 snapshot and migrated it to v5 using the pre-compression code (commit 6f6c64a, i.e. current master schema), then ran VACUUM to get a clean baseline size.
2. Took that fresh v5 database and ran it through this branch's v5→v6 migration (--features compression-zstd), then VACUUMed again for a clean "after" size.

Results:

	size
v5 (master, uncompressed)	72,458,240 bytes (69.1 MiB)
v6 (this PR, dictionary-compressed)	46,047,232 bytes (43.9 MiB)
reduction	36.4%

Other notes:

• 431,310 / 544,673 events (79%) ended up dictionary-compressed; the rest were small enough that storing them raw was already smaller, so they're kept as-is.
• One-time migration + dictionary training (64 KiB dict, level 6) took ~3 seconds for all 544,673 events, wrapped in a single transaction.
• Verified correctness: read events back through the REST API after migration — data round-trips intact (app/title fields, durations, timestamps all correct).

[0xbrayo]

🤖 Claude, on behalf of @0xbrayo

Why a custom implementation instead of sqlite-zstd?

sqlite-zstd (phiresky) implements essentially the same idea — dictionary-trained zstd compression of a column — and was the inspiration for this approach in #493. We didn't pull it in directly for a few reasons specific to this codebase:

• License: sqlite-zstd is LGPL-2.0; this project's other dependencies are permissively licensed, so avoiding it sidesteps that friction.
• Schema fit: it works by moving the target column into a shadow _zstd table and replacing the original table with a view (plus triggers). That would conflict with our existing hand-rolled migration system (_migrate_vX_to_vY, the ON DELETE CASCADE FK, the (bucketrow, starttime) index) and complicate the query layer.
• Operational simplicity: sqlite-zstd expects periodic zstd_incremental_maintenance() calls to retrain dictionaries and vacuum shadow tables. Here, the dictionary is trained once on first upgrade and reused — no background maintenance task needed.
• Android: this implementation is just the zstd crate operating on raw bytes in a normal column, with no SQLite virtual tables/extension registration — the simplest thing to guarantee works with the bundled SQLite used on Android.

The underlying technique (train a shared dictionary once, compress every row against it) is the same; it's just fit directly into this project's existing schema/migration model rather than taken as a generic extension.

[0xbrayo]

🤖 Claude, on behalf of @0xbrayo

@greptile review

[0xbrayo]

🤖 Claude, on behalf of @0xbrayo

@greptile review