Skip to content

parquet plain dict bench - #10591

Merged
Jefffrey merged 2 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/parquet-Plain-dict-bench
Aug 11, 2026
Merged

parquet plain dict bench#10591
Jefffrey merged 2 commits into
apache:mainfrom
Rich-T-kid:rich-T-kid/parquet-Plain-dict-bench

Conversation

@Rich-T-kid

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

see #10590

What changes are included in this PR?

introduces benchmarks for reading string values into dictionary arrays

Are these changes tested?

n/a

Are there any user-facing changes?

no

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Aug 7, 2026
@Rich-T-kid Rich-T-kid changed the title Rich t kid/parquet plain dict bench parquet plain dict bench Aug 7, 2026
@Rich-T-kid
Rich-T-kid force-pushed the rich-T-kid/parquet-Plain-dict-bench branch from 906a25b to f4ee01b Compare August 7, 2026 21:55

@Jefffrey Jefffrey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something i noticed was the other benches in this file seem to be at the page iterator or array reader level, whilst this new one is pretty much at the top level with the record batch reader api

is this something to be concerned about, or the behaviour we're trying to benchmark here can only be done via the top level api?

@Rich-T-kid

Copy link
Copy Markdown
Contributor Author

something i noticed was the other benches in this file seem to be at the page iterator or array reader level, whilst this new one is pretty much at the top level with the record batch reader api

is this something to be concerned about, or the behaviour we're trying to benchmark here can only be done via the top level api?

@Jefffrey The existing benchmarks measure the time to decode an already dictionary-encoded Parquet page into an Arrow dictionary array. This PR's benchmarks measure converting a plain-encoded (non-dictionary) string column into a dictionary array.

I used the top-level API here because it's the less verbose, and mirrors how I'd expect DataFusion to actually use it. That said, it's likely possible to do something similar with the finer-grained APIs. should I update the PR?

@Jefffrey Jefffrey added the development-process Related to development process of arrow-rs label Aug 11, 2026

@Jefffrey Jefffrey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be fine, thanks

@Jefffrey
Jefffrey merged commit fc16607 into apache:main Aug 11, 2026
18 checks passed
@Rich-T-kid
Rich-T-kid deleted the rich-T-kid/parquet-Plain-dict-bench branch August 11, 2026 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-process Related to development process of arrow-rs parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants