Preserve evolved manifest bounds in DF55 fork - #90
Merged
osipovartem merged 1 commit intoSep 24, 2026
Merged
Conversation
osipovartem
force-pushed
the
fix/df55-width-tolerant-manifest-bounds
branch
from
September 24, 2026 14:54
b02d662 to
bef12e3
Compare
Author
|
Local pruning microbenchmark before merge: 10,000 current-schema data-file bounds, 30 min_values passes per trial, five trials, identical debug test profile. Median baseline (c32b9e0): 74.37 ms; median PR head (bef12e3): 73.62 ms. No measurable regression on the common current-schema path. Historical incompatible partition specs intentionally produce unknown pruning stats to avoid false negatives; such tables may read more manifest metadata until per-spec pruning is added. Temporary benchmark code was removed; the PR worktree is clean. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The replacement avoids the former selected-schema threading, public
ManifestFieldTypeIndex, and rewrite-only reader mode. A follow-up benchmark should quantify the conservative manifest-pruning cost on tables retaining many old partition specs; correctness takes precedence over guessing whether fields with different specs are compatible.Local validation
cargo +1.95.0 test -q -p iceberg-rust-spec(177 active passed)cargo +1.95.0 test -q -p iceberg-rust --lib(172 active passed)cargo +1.95.0 test -q -p datafusion_iceberg --lib(89 passed)cargo +1.95.0 test -q -p iceberg-rust --test overwrite_test(3 passed)cargo +1.95.0 clippy -p iceberg-rust-spec -p iceberg-rust -p datafusion_iceberg --all-targets -- -D warningscargo +1.95.0 fmt --all -- --checkThis PR does not update Rustice's dependency pin.