Repository navigation

GH-50628: [C++][Parquet] Rework RecordReader to introduce flat optional optimization - #50629

Merged

pitrou merged 67 commits into

apache:mainfrom

AntoinePrv:levels-bitmap

Oct 28, 2026

Merged

GH-50628: [C++][Parquet] Rework RecordReader to introduce flat optional optimization#50629
pitrou merged 67 commits into
apache:mainfrom
AntoinePrv:levels-bitmap

Commits on Oct 28, 2026
- Move TypedColReaderImpl::Skip
  
  AntoinePrv
  authored and
  pitrou
  committed
  9f708e9
- Break TypedRecordReader inheritance on TypedColumnReader
  
  AntoinePrv
  authored and
  pitrou
  committed
  e7f73f9
- Add ReadValuesBuffer
  
  AntoinePrv
  authored and
  pitrou
  committed
  1a10b16
- Split ColumnReaderImplBase definition and declaration
  
  AntoinePrv
  authored and
  pitrou
  committed
  f6ed50f
- Add DefRepLevelsDecoder
  
  AntoinePrv
  authored and
  pitrou
  committed
  d5971a6
- Explicit int32_t for LevelDecoder
  
  AntoinePrv
  authored and
  pitrou
  committed
  a35c679
- Change ValuesBuffer API
  
  AntoinePrv
  authored and
  pitrou
  committed
  3515051
- Add StackedLevelDecoder and prepare for modularity
  
  AntoinePrv
  authored and
  pitrou
  committed
  6a87536
- Remove DefRepLevelsDecoder
  
  AntoinePrv
  authored and
  pitrou
  committed
  31d8014
- Split TypedRecordReader implementation
  
  AntoinePrv
  authored and
  pitrou
  committed
  7a0cd67
- First RequiredTypedRecordReader
  
  AntoinePrv
  authored and
  pitrou
  committed
  1ff7170
- Remove RecordReader protected members
  
  AntoinePrv
  authored and
  pitrou
  committed
  ee371fe
- Fully use RequiredRecordReader
  
  AntoinePrv
  authored and
  pitrou
  committed
  5ae16d4
- Rename ReadValues -> ValueSink
  
  AntoinePrv
  authored and
  pitrou
  committed
  56ada15
- Split ReserveValuesAndIsValid
  
  AntoinePrv
  authored and
  pitrou
  committed
  289c470
- Devirtualize hooks with CRTP
  
  AntoinePrv
  authored and
  pitrou
  committed
  e681a96
- Simplify template
  
  AntoinePrv
  authored and
  pitrou
  committed
  d759e50
- Remove StackedLevelDecoder
  
  AntoinePrv
  authored and
  pitrou
  committed
  437dccd
- Remove mark_as_written
  
  AntoinePrv
  authored and
  pitrou
  committed
  d768334
- Make some of ColumnReaderImplBase private
  
  AntoinePrv
  authored and
  pitrou
  committed
  b625b36
- Fix horrible dictionnary handling
  
  AntoinePrv
  authored and
  pitrou
  committed
  7f14e9d
- Simplify and rename ColumnChunkReader
  
  AntoinePrv
  authored and
  pitrou
  committed
  d7622de
- Add RleBitPackedToBitmapDecoder::CountUpTo
  
  AntoinePrv
  authored and
  pitrou
  committed
  4d661c6
- Add BitPackedToBitmapDecoder
  
  AntoinePrv
  authored and
  pitrou
  committed
  a849c55
- Add RleBitPackedToBitmapDecoder::Advance
  
  AntoinePrv
  authored and
  pitrou
  committed
  1a806f4
- Add LevelToBitmapDecoder
  
  AntoinePrv
  authored and
  pitrou
  committed
  5be09d4
- Add ValiditySinkBuffer
  
  AntoinePrv
  authored and
  pitrou
  committed
  1bf3bef
- Auto reserve space in sinks:
  
  AntoinePrv
  authored and
  pitrou
  committed
  f5886d4
- Fix integer types
  
  AntoinePrv
  authored and
  pitrou
  committed
  48440a1
- Nit in ColumnChunkReader
  
  AntoinePrv
  authored and
  pitrou
  committed
  84ae7ea
- Eagerly read dict
  
  AntoinePrv
  authored and
  pitrou
  committed
  583ca55
- Cleanups
  
  AntoinePrv
  authored and
  pitrou
  committed
  e2960ef
- Add FlatOptionalTypedRecordReader
  
  AntoinePrv
  authored and
  pitrou
  committed
  ee767e0
- Add some doc
  
  AntoinePrv
  authored and
  pitrou
  committed
  8b25393
- Factorize ReadDictionary
  
  AntoinePrv
  authored and
  pitrou
  committed
  0dc39c6
- Simplify read_dense_for_nullable
  
  AntoinePrv
  authored and
  pitrou
  committed
  455663f
- Factor Sinks with DataSinkBuffer
  
  AntoinePrv
  authored and
  pitrou
  committed
  24aef79
- Use LevelBufferSink in TypedRecordReader
  
  AntoinePrv
  authored and
  pitrou
  committed
  7e2b596
- Relax record reader level tests flat optional
  
  AntoinePrv
  authored and
  pitrou
  committed
  8acde88
- Fix compiler errrors
  
  AntoinePrv
  authored and
  pitrou
  committed
  e1f0032
- Safe decoded number check and func rename
  
  AntoinePrv
  authored and
  pitrou
  committed
  3e02ab9
- Explicit values_written contract
  
  AntoinePrv
  authored and
  pitrou
  committed
  7e82f89
- Fix LevelDecoder::max_level type
  
  AntoinePrv
  authored and
  pitrou
  committed
  fb45f21
- Fix integer convertion
  
  AntoinePrv
  authored and
  pitrou
  committed
  4590d8d
- Factorize level decoders
  
  AntoinePrv
  authored and
  pitrou
  committed
  a7292d5
- Fixt typos
  
  AntoinePrv
  authored and
  pitrou
  committed
  6f574d6
- Move level decoder utility and rename file
  
  AntoinePrv
  authored and
  pitrou
  committed
  fb2d2b2
- Move level decoder adapters to level_decoder.h
  
  AntoinePrv
  authored and
  pitrou
  committed
  d6631fb
- Fix small typos
  
  AntoinePrv
  authored and
  pitrou
  committed
  24fdf3c
- Fix issues in level counting
  
  AntoinePrv
  authored and
  pitrou
  committed
  64c4b6c
- Fix extra alloc
  
  AntoinePrv
  authored and
  pitrou
  committed
  4e223cd
- Add RecordReader::Make flag
  
  AntoinePrv
  authored and
  pitrou
  committed
  3d72494
- Throw on FlatOptional def_levels
  
  AntoinePrv
  authored and
  pitrou
  committed
  0fb214c
- Always call RecordReader with flat optiona optimization
  
  AntoinePrv
  authored and
  pitrou
  committed
  8b870ae
- Addition flat optional test
  
  AntoinePrv
  authored and
  pitrou
  committed
  b51fc9d
- Detemplate kdic
  
  AntoinePrv
  authored and
  pitrou
  committed
  490351e
- Add flat optiona bse in flba and byte array
  
  AntoinePrv
  authored and
  pitrou
  committed
  68bf579
- Ad more flba and array test
  
  AntoinePrv
  authored and
  pitrou
  committed
  9d2afe3
- Make RecordReader::Make params struct public
  
  AntoinePrv
  authored and
  pitrou
  committed
  cd6ffd8
- Run RecordReader tests with and without optimizations
  
  AntoinePrv
  authored and
  pitrou
  committed
  daae30a
- Rename flat_optiona_optimization as allow_levels_optimization
  
  AntoinePrv
  authored and
  pitrou
  committed
  8ddab16
- Add level optim option to RowGroupReader
  
  AntoinePrv
  authored and
  pitrou
  committed
  0ba2b58
- Reuse validity map
  
  AntoinePrv
  authored and
  pitrou
  committed
  5ea4fa3
- Lazily pop dictionnary
  
  AntoinePrv
  authored and
  pitrou
  committed
  0c52aa4
- Address some review comments
  
  AntoinePrv
  authored and
  pitrou
  committed
  cddae55
- Fix typo
  
  AntoinePrv
  authored and
  pitrou
  committed
  0d67901
- Add missing gTest PrintTo
  
  AntoinePrv
  authored and
  pitrou
  committed
  c3601e5

Commits

Commits on Oct 28, 2026