parquet plain dict bench - #10591
Conversation
906a25b to
f4ee01b
Compare
Jefffrey
left a comment
There was a problem hiding this comment.
something i noticed was the other benches in this file seem to be at the page iterator or array reader level, whilst this new one is pretty much at the top level with the record batch reader api
is this something to be concerned about, or the behaviour we're trying to benchmark here can only be done via the top level api?
@Jefffrey The existing benchmarks measure the time to decode an already dictionary-encoded Parquet page into an Arrow dictionary array. This PR's benchmarks measure converting a plain-encoded (non-dictionary) string column into a dictionary array. I used the top-level API here because it's the less verbose, and mirrors how I'd expect DataFusion to actually use it. That said, it's likely possible to do something similar with the finer-grained APIs. should I update the PR? |
Which issue does this PR close?
Rationale for this change
see #10590
What changes are included in this PR?
introduces benchmarks for reading string values into dictionary arrays
Are these changes tested?
n/a
Are there any user-facing changes?
no