feat: DH-23168: Coerce parquet uint64 column - #8390
Conversation
Cherry-pick of deephaven#8350 Co-authored-by: Jianfeng Mao <4297243+jmao-denver@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds configurable Parquet UINT_64 reading while preserving values accurately.
Changes:
- Defaults
UINT_64columns toBigInteger. - Adds checked and signed
longcoercion modes. - Adds Java/Python tests and documentation.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
py/server/tests/test_parquet.py |
Tests Python coercion modes. |
py/server/deephaven/parquet.py |
Exposes Python configuration API. |
extensions/parquet/table/src/test/java/io/deephaven/parquet/table/TestParquetSchemaReader.java |
Tests schema inference. |
extensions/parquet/table/src/test/java/io/deephaven/parquet/table/ParquetTableReadWriteTest.java |
Tests end-to-end reads. |
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetSchemaReader.java |
Infers target column types. |
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/ParquetInstructions.java |
Adds coercion instructions. |
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/pagestore/topage/ToLongPage.java |
Supports checked long conversion. |
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/pagestore/topage/ToBigIntegerPage.java |
Supports exact conversion. |
extensions/parquet/table/src/main/java/io/deephaven/parquet/table/location/ParquetColumnLocation.java |
Selects appropriate materializers. |
extensions/parquet/table/build.gradle |
Adds test dependency. |
extensions/parquet/base/src/main/java/io/deephaven/parquet/base/materializers/LongFromUnsignedLongMaterializer.java |
Implements checked conversion. |
extensions/parquet/base/src/main/java/io/deephaven/parquet/base/materializers/BigIntegerFromUnsignedLongMaterializer.java |
Implements exact conversion. |
docs/python/reference/data-import-export/Parquet/readTable.md |
Documents read configuration. |
docs/python/reference/data-import-export/Parquet/ColumnInstruction.md |
Documents Python option. |
docs/python/how-to-guides/data-import-export/parquet-formats.md |
Explains Python coercions. |
docs/groovy/reference/data-import-export/Parquet/readTable.md |
Adds related documentation. |
docs/groovy/how-to-guides/data-import-export/parquet-instructions.md |
Documents Java instructions. |
docs/groovy/how-to-guides/data-import-export/parquet-formats.md |
Explains Groovy coercions. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if ci.codec_name: | ||
| builder.addColumnCodec(ci.column_name, ci.codec_name, ci.codec_args) | ||
| builder.useDictionary(ci.column_name, ci.use_dictionary) | ||
| if ci.unsigned_long_target is not None: |
There was a problem hiding this comment.
pydoc for the field says "it is ignored when writing". that's not entirely true at the moment. it should say "it is rejected when writing". i would push back against silently discarding the passed-in option because that feels gross. if they passed in something invalid, rejection is the right answer
so if anything, that one-word change in the pydoc would be my vote. i wouldn't think it's highly critical or anything, though
rcaudy
left a comment
There was a problem hiding this comment.
Code owner approval for .py files.
Cherry-pick of #8350