Skip to content

Add RunEndBool encoding for efficient boolean array compression#8467

Draft
joseph-isaacs wants to merge 1 commit into
developfrom
claude/runend-bool-compression-r148yw
Draft

Add RunEndBool encoding for efficient boolean array compression#8467
joseph-isaacs wants to merge 1 commit into
developfrom
claude/runend-bool-compression-r148yw

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR introduces a new RunEndBool encoding specialized for compressing boolean arrays with long runs. Since boolean values strictly alternate in run-end encoding, we can store only the run end positions plus a single start flag, rather than maintaining a separate values array like the generic RunEnd encoding.

Changes

  1. New vortex-runend-bool crate with:

    • RunEndBoolArray type and metadata serialization
    • Specialized compression/decompression for boolean runs
    • Compute kernels: filter, take, invert, is_constant, is_sorted, min_max
    • Scalar access and slicing operations
    • Efficient run-preserving filtering that merges adjacent runs with identical values
  2. Self-contained run-end indexingRunEndBool implements its own find_physical_index / find_slice_end_index (ctx-based, mirroring vortex-runend's SearchSortedPrimitiveArray approach). vortex-runend itself is left unchanged.

  3. Integration with compression pipeline:

    • New BoolRunEndScheme in vortex-btrblocks for automatic bool array compression
    • Estimates compression ratio based on run count (profitable when average run length > 8), cascading the run ends child for further compression
    • Registered in the default compression scheme list
  4. File format support:

    • RunEndBool registered in the default session; arrays serialize/deserialize correctly in the Vortex file format
    • Round-trip and compute-conformance covered by crate-level tests; test_run_heavy_bool_uses_runend in vortex-btrblocks asserts the compressor auto-selects the encoding

Design Notes

  • Boolean runs strictly alternate, so run i has value value_at_index(i, start) where even indices equal start and odd indices equal !start
  • The encoding preserves run structure through filtering and slicing operations
  • Sparse filters (< 10% of runs kept) decode via per-index binary search; dense filters scan runs once and stay run-end encoded
  • All compute operations respect optional validity arrays

@codspeed-hq

codspeed-hq Bot commented Jun 17, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 19.09%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 4 improved benchmarks
❌ 1 regressed benchmark
✅ 1592 untouched benchmarks
⏩ 4 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_into_canonical[(100, 100)] 272.2 µs 307.1 µs -11.39%
Simulation chunked_bool_canonical_into[(1000, 10)] 26.8 µs 16.4 µs +63.47%
Simulation chunked_varbinview_canonical_into[(1000, 10)] 191.4 µs 154.8 µs +23.66%
Simulation chunked_varbinview_into_canonical[(1000, 10)] 205.9 µs 169.8 µs +21.26%
Simulation eq_i64_constant 319.8 µs 290.1 µs +10.25%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/runend-bool-compression-r148yw (8133d43) with develop (ee2cd67)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Jun 17, 2026 — with Claude
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Jul 3, 2026
@joseph-isaacs joseph-isaacs force-pushed the claude/runend-bool-compression-r148yw branch from 1fca4d5 to 53d3cdb Compare July 3, 2026 04:27
@joseph-isaacs joseph-isaacs removed the stale This PR is stale and will be auto-closed soon label Jul 3, 2026 — with Claude
…election

Reintroduce the `vortex-runend-bool` encoding on top of the current
`develop`. Boolean runs strictly alternate, so a `RunEndBoolArray` stores
only the run `ends`, the value of the first run (`start`), and optional
validity, rather than a separate values array.

The encoding is self-contained against develop's reworked `runend` search
API: `find_physical_index` / `find_slice_end_index` are local free
functions built on `SearchSortedPrimitiveArray` + `match_each_unsigned_
integer_ptype!`, threading `&mut ExecutionCtx` through every caller
(scalar_at, take, filter, slice). The dense run-preserving filter path is
preserved. Parent kernels (slice, filter, take, not) are registered via
`session.kernels().register_execute_parent_kernel`, and `with_buffers` is
implemented per the current `VTable` contract.

Wire the encoding into `vortex-file`'s `register_default_encodings` and add
a `BoolRunEndScheme` to `vortex-btrblocks` so run-heavy bool arrays are
automatically run-end encoded.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs joseph-isaacs force-pushed the claude/runend-bool-compression-r148yw branch from 53d3cdb to 8133d43 Compare July 3, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant