feat: support geometry and geography types by wirybeaver · Pull Request #2933 · apache/iceberg-rust

wirybeaver · 2026-07-31T02:10:14Z

Summary

This draft PR adds Iceberg Geometry/Geography primitive type support by reusing arrow-rs/parquet-geospatial support instead of introducing a local geospatial model.

Adds GeometryType and GeographyType with an Iceberg-owned EdgeInterpolationAlgorithm, translating to parquet_geospatial::WkbEdges at the Arrow boundary.
Converts Geometry/Geography to Arrow WKB extension metadata and enables Parquet geospatial logical type writing.
Maps Avro, Glue, and HMS representations to bytes/binary.
Rejects non-null JSON defaults, blocks unsupported partition transforms, and skips byte min/max statistics for spatial values.

Related issues

Related to #2411 and #1884.

Supersedes #2653, which was closed automatically by the stale bot. Existing review discussion remains available there.

brgr-s · 2026-07-31T13:31:23Z

In case you are interested, PR #2019 (CLOSED as stale) also has code related to geo types.

paleolimbot

There are a few details around the conversion of types, but this is looking good to me!

paleolimbot · 2026-08-02T01:45:27Z

+            return Ok(field_type.clone());
+        }
+
+        let wkb_type = field.try_extension_type::<WkbType>().map_err(|err| {


Unfortunately, until arrow-rs 59.1.0 there are some corner cases that slip through here (fixed in apache/arrow-rs#10065 ). This is still OK, it just won't work for geography and does a few funny things with corner case CRSes until the arrow-rs dependency updates.

Thanks for flagging this. I am leaving these corner cases as a limitation of the current arrow-rs dependency for now; they will be resolved when iceberg-rust upgrades to arrow-rs 59.1.0 or later.

paleolimbot · 2026-08-02T01:56:33Z

+        let crs = wkb_type.metadata().crs.as_ref().map(|crs| match crs {
+            serde_json::Value::String(value) => value.clone(),
+            other => other.to_string(),
+        });


Iceberg has a slightly different CRS restriction than Parquet, where it greatly prefers Authority:Code over PROJJSON (because schemas are written so frequently to iceberg files, and PROJJSON is sufficiently verbose that it causes performance/space issues.

Because of this (and the buggy WkbType until arrow-rs 59.1.0), you may want to implement WkbType yourself specifically for Iceberg (e.g., with a utility to extract the iceberg-preferred CRS, which can be derived from PROJJSON using the id member).

Addressed the CRS representation in 889ccfb: Arrow string CRS values are retained, while PROJJSON is converted from its id.authority and id.code to Iceberg-preferred AUTHORITY:CODE, with invalid metadata rejected. I kept arrow-rs WkbType rather than introducing an Iceberg-specific copy because parquet 58.4 consumes its own WkbType; replacing only the Iceberg Arrow type would not fix the pre-59.1 Geography behavior and could make Parquet logical-type writing incompatible.

paleolimbot

I am not qualified to approve this from the icberg-rust end of things, but the geography type conversion seems solid (I suggested some optional improvements to errors inline). Thank you!

Happy to review any follow-ups on the statistics/pruning end of things if you are interested (or to attempt them if you aren't!).

I kept arrow-rs WkbType rather than introducing an Iceberg-specific copy because parquet 58.4 consumes its own WkbType; replacing only the Iceberg Arrow type would not fix the pre-59.1 Geography behavior and could make Parquet logical-type writing incompatible.

I think this is fine, although anybody who wants to write Geography will have to write invalid metadata into Iceberg (and will receive invalid metadata that will be rejected when reading using the arrow reader).

The workaround for this is to rewrite the metadata just for the Parquet reader, but I get how it's not worth adding that here since it will automatically resolve when arrow-rs is bumped. Our workaround for the write side is here:

https://github.com/apache/sedona-db/blob/ad83ab697b43998a4546359257015df71e210f18/rust/sedona-geoparquet/src/writer.rs#L681-L718

Can this be documented somehow?

paleolimbot · 2026-08-03T13:02:51Z

+                    Error::new(
+                        ErrorKind::DataInvalid,
+                        "PROJJSON CRS id must contain a non-empty authority",
+                    )


I think these errors are OK (since they speak to invalid PROJJSON rather than a limitation of the iceberg format).

Agreed. I kept these validation errors unchanged because they describe malformed PROJJSON metadata rather than an Iceberg representation limitation.

paleolimbot · 2026-08-03T13:04:47Z

+    };
+
+    match crs {
+        serde_json::Value::String(crs) => Ok(Some(crs.clone())),


You may want to check here that the crs is less than some threshold of bytes (say, 128 bytes). Alternatively, you could validate that it looks like an authority:code string but that is harder. Mostly you want to catch WKT2 CRSes or PROJJSON CRSes that were accidentally escaped as strings to avoid manifest files that are written with many kilobytes of overhead per geometry field.

Added a 128-byte limit for CRS strings in 6c87c62, with boundary and Arrow import tests. This prevents accidentally escaped WKT2/PROJJSON values from adding large per-field schema overhead.

paleolimbot · 2026-08-03T13:25:57Z

+            PrimitiveType::Geometry(geometry) => match geometry.crs() {
+                Some(crs) => write!(f, "geometry({crs})"),
+                None => write!(f, "geometry"),
+            },
+            PrimitiveType::Geography(geography) => {
+                let algorithm = geography.algorithm();
+                match (geography.crs(), algorithm) {
+                    (None, EdgeInterpolationAlgorithm::Spherical) => write!(f, "geography"),
+                    (Some(crs), EdgeInterpolationAlgorithm::Spherical) => {
+                        write!(f, "geography({crs})")
+                    }
+                    (crs, algorithm) => write!(
+                        f,
+                        "geography({}, {})",
+                        crs.unwrap_or(DEFAULT_GEOSPATIAL_CRS),
+                        edge_interpolation_algorithm_as_str(algorithm)
+                    ),
+                }
+            }


Do you have to escape the CRS at all here? It is an arbitrary string and could contain ) or , which I don't think would roundtrip (and may be invalid). Perhaps neither of those are concerns here.

There is no escaping syntax in the Iceberg type-string grammar. To guarantee that values produced by Display can be parsed back, 6c87c62 rejects CRS strings containing , or ) at construction time and adds round-trip validation tests.

paleolimbot · 2026-08-03T13:30:31Z

+    #[test]
+    fn test_min_max_aggregator_skips_geospatial_byte_statistics() {


No problem if statistics are out of scope for this PR (but I'm also happy to review their addition here or in a follow up). Getting geography statistics from the Parquet writer are a bit of a pain (you have to register a static geography-aware bounder, an example of which we have in SedonaDB).

Thanks. I will keep geospatial bounds/statistics out of scope for this PR; the current writer explicitly avoids treating WKB byte ordering as geospatial min/max statistics. Geography-aware bounds can be added in a focused follow-up.

wirybeaver · 2026-08-05T16:22:26Z

@paleolimbot Thanks for calling this out. I documented the arrow-rs 58.x Geography metadata limitation in 9a5d028 on the public Arrow schema conversion APIs. The docs now explain both directions: standards-compliant GeoArrow edges metadata may be interpreted as Geometry when importing, while exported Geography metadata uses the legacy algorithm key for parquet-rs 58.x compatibility and may be rejected or misinterpreted by standards-compliant Arrow readers. The note also links to apache/arrow-rs#10065, where this is fixed for 59.1.0+.

paleolimbot · 2026-08-05T16:28:50Z

Great!

Can this be unmarked as draft? (i.e., is it ready for review from an iceberg committer?)

paleolimbot · 2026-08-06T15:42:48Z

It seems like there's a merge conflict here. When that's fixed and CI is clean I'm happy to go fishing for iceberg-rust committers for review since this is a huge feature for us that will unblock a lot of things!

wirybeaver mentioned this pull request Jul 31, 2026

Support geo types #1884

Open

This was referenced Jul 31, 2026

feat: support geometry and geography types #2653

Closed

EPIC: v3 Support Tracking #2411

Open

paleolimbot reviewed Aug 2, 2026

View reviewed changes

paleolimbot approved these changes Aug 3, 2026

View reviewed changes

wirybeaver marked this pull request as ready for review August 6, 2026 01:50

wirybeaver added 6 commits August 6, 2026 02:16

feat: support geometry and geography types

44899b9

fix: decouple geography edge algorithm API

e13b80e

fix: normalize geospatial CRS metadata

7af379e

fix: validate geospatial CRS syntax

4e85787

docs: note geography metadata compatibility

f097709

fix: avoid duplicate large binary match arm

3ae7041

wirybeaver force-pushed the support-geometry-geography-types branch from 9a5d028 to 3ae7041 Compare August 6, 2026 02:24

		#[test]
		fn test_min_max_aggregator_skips_geospatial_byte_statistics() {

Uh oh!

Conversation

wirybeaver commented Jul 31, 2026

Summary

Related issues

Uh oh!

brgr-s commented Jul 31, 2026

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wirybeaver commented Aug 5, 2026

Uh oh!

paleolimbot commented Aug 5, 2026

Uh oh!

paleolimbot commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants