[C++][Parquet] Writer emits an empty first data page for a repeated column when data_page_size is small, and the reader then drops the row group's rows #54533

Describe the bug, including details regarding any error messages, version, and platform.

When the first write batch of a list/map column chunk contains a single record, the Parquet writer can begin the column chunk with a data page that holds no values (num_values = 0, num_rows = 0). This happens when data_page_size is small. The Arrow reader takes that empty page as the end of the column chunk, so every row of that row group is silently missing on read, while the file metadata still counts them. DuckDB and Polars skip the empty page and read the rows.

This happens with a one-row file, and with any file whose last row group (or any row group, e.g. row_group_size=1) holds exactly one row.

Reproducer

import pyarrow as pa
import pyarrow.parquet as pq

list_bool = pa.list_(pa.bool_())

# 1) one-row table
t = pa.table({"c0": pa.array([[True]], list_bool)})
pq.write_table(t, "one_row.parquet", data_page_version="2.0", data_page_size=64)
print(pq.ParquetFile("one_row.parquet").metadata.num_rows)   # 1
print(pq.read_table("one_row.parquet").num_rows)             # expected 1

# 2) the last row group holds a single row
t = pa.table({"c0": pa.array([[True], [False], [True]], list_bool)})
pq.write_table(t, "last_rg.parquet", data_page_version="2.0", data_page_size=64,
               row_group_size=2)
print(pq.read_table("last_rg.parquet")["c0"].to_pylist())    # expected [[True], [False], [True]]

# 3) default encoding (dictionary), any dictionary-encodable leaf type, V1 pages, very small page size
t = pa.table({"c0": pa.array([[1], [2], [3]], pa.list_(pa.int32()))})
pq.write_table(t, "dict.parquet", data_page_size=4, row_group_size=2)
print(pq.read_table("dict.parquet")["c0"].to_pylist())       # expected [[1], [2], [3]]

# 4) with a second column the read fails instead
t = pa.table({"id": [1], "c0": pa.array([[True]], list_bool)})
pq.write_table(t, "two_cols.parquet", data_page_version="2.0", data_page_size=64)
pq.read_table("two_cols.parquet")

Output on main:

1
0
[[True], [False]]
[[1], [2]]
Traceback (most recent call last):
  ...
pyarrow.lib.ArrowInvalid: Column 1 named c0 expected length 1 but got length 0

DuckDB 1.5.x and Polars 2.0 read one_row.parquet as [[True]] and last_rg.parquet as all three rows. With write_page_index=True, ClickHouse refuses the file because the offset index lists two pages with the same first_row_index 0 (the empty page at offset 4, size 26, then the real page at offset 30, size 32): Invalid offset index: PageLocation(offset=30, compressed_page_size=32, first_row_index=0), prev offset: 30, prev row: 0.

I decoded the page headers of one_row.parquet. The column chunk holds two DATA_PAGE_V2 pages: first num_values=0, num_nulls=0, num_rows=0, encoding=RLE with a 4-byte body (only the RLE length prefix), then the real page with num_values=1, num_rows=1.

Scope (main, a column chunk whose first write batch holds one record):

The writer can also emit an empty page in the middle of a chunk. For example, 1025 one-element list<bool> rows with data_page_size=64 produce pages of 1024, 0 and 1 values. Arrow happens to read that file correctly.

Versions: pyarrow 27.0.0.dev13 (nightly built from main at 9f03413), Python 3.12, Linux x86_64 (Ubuntu 22.04). With the released wheels, cases 1, 2 and 4 reproduce from 22.0.0 on, and case 3 from 23.0.0 on (on 22.0.0, dictionary encoding is affected only with V2 pages); 22.0.0 through 26.0.0 were tested. pyarrow 21.0.0 writes a single page and reads the row back correctly.

Where it comes from

• Writer: DoInBatchesRepeated calls action(offset, check_page_limit_end_offset - offset, /*check_page_limit=*/true) with zero levels when the last record of the batch starts at offset, which is always the case for a batch holding a single record (column_writer.cc#L1219-L1224). CommitWriteAndCheckPageLimit then calls AddDataPage() when EstimatedDataEncodedSize() >= data_pagesize(), without checking that anything is buffered (column_writer.cc#L1835-L1846). An empty encoder's estimate is not 0:

  • RleBooleanEncoder returns 4 + RlePreserveBufferSize(0, 1) = 4 + 65 = 69 bytes (encoder.cc#L1754-L1756, encoder.cc#L444-L451).
  • The dictionary encoder returns 1 + 5 = 6 bytes (encoder.cc#L499-L503).

  These numbers match the thresholds in the table above exactly.

• The zero-length call became possible with [C++][Parquet] Page sizes for repeated columns can overflow int32 with page index enabled #47027 (GH-47027: [C++][Parquet] Fix repeated column pages not being written when reaching page size limit #47032, first released in 22.0.0), which changed if (offset < last_record_begin_offset) to if (offset <= last_record_begin_offset). The later rewrite in GH-47030: [C++][Parquet] Add setting to limit the number of rows written per page #47090 kept this behaviour.

• Reader: HasNextInternal() returns false when the next page has num_buffered_values_ == 0 (column_reader.cc#L776-L785). So ReadRecords returns 0 on a leading empty page, and LeafReader::LoadBatch moves on to the next row group (arrow/reader.cc#L510-L514).

Possible fixes:

• Writer: do not call AddDataPage() when nothing is buffered (num_buffered_values_ == 0), or skip the zero-length action call.
• Reader: skip a data page with zero values instead of treating it as the end of the column chunk. This would make files already written by 22.0.0–26.0.0 readable by future readers only: pyarrow 21.0.0 also reads one_row.parquet written by main as 0 rows, so a reader-only fix does not help older readers.

Impact: data_page_size is a public writer option. With small values, rows of list/map columns written by pq.write_table silently disappear when the file is read back with pq.read_table or pandas read_parquet, while the metadata still reports them. Multi-column files fail to load. Found by differential testing of Parquet writers and readers.

Component(s)

C++, Parquet, Python

Component(s)

C++, Parquet