Skip to content

Apply position deletes at equal sequence numbers - #81

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
position-delete-inclusive-sequence
Sep 17, 2026
Merged

osipovartem merged 1 commit into
embucket-sync-df55.0.0from
position-delete-inclusive-sequence

Conversation

@osipovartem

Copy link
Copy Markdown

Summary

  • apply Iceberg v2 position deletes when the delete sequence number is equal to the data file sequence number
  • extract the production join filter construction so its exact vectorized expression can be regression-tested
  • preserve the existing no-delete fast path and physical hash anti-join behavior

Iceberg v2 requires position deletes to apply when delete_sequence_number >= data_sequence_number. The existing filter used strict >, which could expose rows deleted in the same sequence. Equality deletes intentionally retain their separate strict sequence semantics.

Performance

This changes only the comparison operator inside the existing vectorized DataFusion join filter. It adds no per-row scalar conversion, materialization, or work to scans without position deletes. Replacing the current hash anti-join with Apache-style delete bitmaps and Parquet row selection will be evaluated separately with benchmarks.

Verification

  • cargo +1.95.0 test -p datafusion_iceberg position_deletes_include_equal_data_sequence_numbers
  • cargo +1.95.0 test -p datafusion_iceberg --test position_delete
  • cargo +1.95.0 clippy -p datafusion_iceberg --all-targets -- -D warnings
  • cargo +1.95.0 fmt --all -- --check

@osipovartem
osipovartem merged commit 36e86d5 into embucket-sync-df55.0.0 Sep 17, 2026
2 checks passed
@osipovartem
osipovartem deleted the position-delete-inclusive-sequence branch September 17, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant