Skip to content

Assign Iceberg field IDs recursively in list elements - #91

Merged
osipovartem merged 1 commit into
embucket-sync-df55.0.0from
fix/nested-variant-parquet-roundtrip
Sep 25, 2026
Merged

osipovartem merged 1 commit into
embucket-sync-df55.0.0from
fix/nested-variant-parquet-roundtrip

Conversation

@osipovartem

Copy link
Copy Markdown

Problem

new_fields_with_ids assigned an ID to a list element but not to fields inside a struct element. Iceberg table creation for ARRAY(OBJECT(v VARIANT)) then failed with a missing Parquet field ID.

Change

  • Assign IDs recursively through list elements while keeping Variant storage fields unnumbered.
  • Reserve existing IDs and skip them when allocating new IDs, avoiding duplicates.
  • Return an error on ID exhaustion instead of overflowing; propagate the new Result<Fields, Error> API through the two fork planner callsites.

Validation

  • Red/green regressions for list-of-struct, mixed existing IDs, i32::MAX, and exhaustion.
  • cargo +1.95.0 test -p iceberg-rust-spec --lib: 181 passed, 1015 ignored.
  • cargo +1.95.0 check -p datafusion_iceberg -p datafusion-iceberg-sql.
  • cargo +1.95.0 clippy -p iceberg-rust-spec -p datafusion_iceberg -p datafusion-iceberg-sql --all-targets --all-features -- -D warnings; formatting check.
  • Independent read-only review: approved.

Downstream callers of new_fields_with_ids must handle Result; Rustice callsites are being updated.

@osipovartem
osipovartem merged commit a51b048 into embucket-sync-df55.0.0 Sep 25, 2026
2 checks passed
@osipovartem
osipovartem deleted the fix/nested-variant-parquet-roundtrip branch September 25, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant