[wgtmac]

Rationale for this change

Parquet LIST columns that are effectively fixed-length, such as vector embeddings, pay the full Dremel definition/repetition-level reconstruction cost on read. This patch adds a page-level fast path that detects fixed-length LIST pages and reads their child values directly.

What changes are included in this PR?

• Add a reader property, ArrowReaderProperties::set_fixed_list_fast_path(bool), disabled by default.
• Inspect encoded definition/repetition levels for non-nullable LIST pages and infer a fixed list width.
• Read fixed-length child values across data pages without materializing the normal Dremel level arrays.
• Preserve the existing path for nullable or unsupported nested columns.
• Carry the list phase across page boundaries, including pages that split a repeated record.
• Add PLAIN, BYTE_STREAM_SPLIT, and ALP benchmarks for ordinary LIST and FixedSizeList columns.
• Add a design and benchmark report in parquet_vector_reader_optimization.md.

Are these changes tested?

• make -C cpp/build-release -j8 parquet
• make -C cpp/build-release -j8 parquet_shared
• make -C cpp/build-release -j8 parquet-benchmarks using Conan's local benchmark/1.9.4 package
• 158 relevant LIST/FixedSizeList reader tests pass
• clang-format --dry-run --Werror passes for modified C++ files
• Release Google Benchmark runs cover PLAIN, BYTE_STREAM_SPLIT, and ALP at list widths 3, 16, and 768

The previous macOS Intel CI run also reported arrow-s3fs-test and arrow-s3fs-module-test timeouts; these are unrelated S3 test timeouts rather than failures in the modified Parquet targets.

Are there any user-facing changes?

Yes. Applications can opt into the fixed-length LIST fast path with ArrowReaderProperties::set_fixed_list_fast_path(true). It remains disabled by default. If enabled for a variable-width LIST and a later page changes width after values have been consumed, the reader returns an explicit error instructing callers to disable the property.

Was AI used for this PR?

Yes. AI was used to research the Hardwood fixed-length LIST design, implement the reader fast path and benchmarks, investigate CI failures, and analyze benchmark output. The code and PR description were reviewed in the local workspace before submission.

PR code and description written by:

• Human
• AI

Reviewed before submission by:

• Human
• AI
• Not reviewed

[github-actions]

Thanks for opening a pull request!

This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.

If this is not a minor PR, could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

After updating the title, you can mark the pull request as ready for review.

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[rok]

Hey @wgtmac, thanks for working on optimized read paths for fixed-length lists!

I've tasked @MashrafeeAryan with the same goal and he already has a write path PR up: #55251 (though I'm not sure he's ready for review yet, Mashrafee?).
Regarding the read path - since you already started, would you like to continue and we help you with review? If you don't have the time we can also have Mashrafee give this a try first, but completely up to you.

[wgtmac]

Thanks @rok for the heads-up! I happened to have access to OpenAI Astra these days so I just let it explore optimization opportunities alone. Although it has proved some positive findings, I don't have enough time to continue in the near future. Feel free to take it over @MashrafeeAryan.

[rok]

Thanks @wgtmac ! :)
Would you also be open to review once @MashrafeeAryan has a PR up?