Skip to content

Preserve evolved manifest bounds in DF55 fork - #90

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
fix/df55-width-tolerant-manifest-bounds
Sep 24, 2026
Merged

osipovartem merged 1 commit into
embucket-sync-df55.0.0from
fix/df55-width-tolerant-manifest-bounds

Conversation

@osipovartem

@osipovartem osipovartem commented Sep 24, 2026 •

Copy link
Copy Markdown

Summary

  • Port the consumer-boundary design from Decode manifest bounds after schema promotion JanKaul/iceberg-rust#410 to the DF55 fork: decode manifest bounds using the embedded schema and promote them only for DataFusion statistics and pruning.
  • Accept old 4-byte numeric/date bounds under promoted types without automatically rewriting those bounds to current widths.
  • Preserve the fork's cached historical partition-type candidate decoder. When old source fields are no longer available, retain the manifest and discard only optional summaries. Mixed partition specs produce unknown pruning stats instead of false exclusions; same-spec old bounds remain prunable.
  • Propagate manifest read/serialization errors in the rewrite path instead of silently skipping entries.

The replacement avoids the former selected-schema threading, public ManifestFieldTypeIndex, and rewrite-only reader mode. A follow-up benchmark should quantify the conservative manifest-pruning cost on tables retaining many old partition specs; correctness takes precedence over guessing whether fields with different specs are compatible.

Local validation

  • cargo +1.95.0 test -q -p iceberg-rust-spec (177 active passed)
  • cargo +1.95.0 test -q -p iceberg-rust --lib (172 active passed)
  • cargo +1.95.0 test -q -p datafusion_iceberg --lib (89 passed)
  • cargo +1.95.0 test -q -p iceberg-rust --test overwrite_test (3 passed)
  • cargo +1.95.0 clippy -p iceberg-rust-spec -p iceberg-rust -p datafusion_iceberg --all-targets -- -D warnings
  • cargo +1.95.0 fmt --all -- --check
  • Independent read-only review approved after addressing the retained-manifest/list decode case.

This PR does not update Rustice's dependency pin.

@osipovartem
osipovartem force-pushed the fix/df55-width-tolerant-manifest-bounds branch from b02d662 to bef12e3 Compare September 24, 2026 14:54
@osipovartem osipovartem changed the title Decode evolved manifest bounds in DF55 fork Preserve evolved manifest bounds in DF55 fork Sep 24, 2026
@osipovartem

Copy link
Copy Markdown
Author

Local pruning microbenchmark before merge: 10,000 current-schema data-file bounds, 30 min_values passes per trial, five trials, identical debug test profile. Median baseline (c32b9e0): 74.37 ms; median PR head (bef12e3): 73.62 ms. No measurable regression on the common current-schema path. Historical incompatible partition specs intentionally produce unknown pruning stats to avoid false negatives; such tables may read more manifest metadata until per-spec pruning is added. Temporary benchmark code was removed; the PR worktree is clean.

@osipovartem
osipovartem merged commit 11ef6a0 into embucket-sync-df55.0.0 Sep 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant