[C++][Python] CSV writer writes a null of a one-column table as an empty line, which read_csv skips by default, so null rows are lost on a round trip #54535

Describe the bug, including details regarding any error messages, version, and platform.

The CSV writer emits a null cell as the unquoted null_string, which is empty by default. In a table with a single column, a null row therefore becomes an empty line. Arrow's CSV readers skip empty lines by default (ParseOptions.ignore_empty_lines = True), so writing such a table with default options and reading it back silently drops every null row. ds.write_dataset(..., format="csv") and ds.dataset(..., format="csv") behave the same.

Reproducer

import io
import pyarrow as pa
import pyarrow.csv as csv

t = pa.table({"c": pa.array([1, None, 3, None, 5], pa.int64())})
buf = io.BytesIO()
csv.write_csv(t, buf)
print(buf.getvalue())

back = csv.read_csv(io.BytesIO(buf.getvalue()))
print(back.num_rows, back["c"].to_pylist())   # expected 5 [1, None, 3, None, 5]

# string column: an empty string is quoted, a null is not
s = pa.table({"s": pa.array(["a", None, "", "b"])})
buf = io.BytesIO()
csv.write_csv(s, buf)
print(buf.getvalue(), csv.read_csv(io.BytesIO(buf.getvalue()))["s"].to_pylist())

Output on main:

b'"c"\n1\n\n3\n\n5\n'
3 [1, 3, 5]
b'"s"\n"a"\n\n""\n"b"\n' ['a', '', 'b']

The file has 5 data rows but reads back as 3; the rows no longer line up with their positions, and no error or warning is raised. The other options behave as follows:

• quoting_style="all_valid" and "none" give the same result.
• With include_header=False, a leading null becomes an empty first line.
• Reading with ParseOptions(ignore_empty_lines=False) recovers all 5 rows.
• pandas read_csv (default skip_blank_lines=True) also reads 3 rows.
• Polars and DuckDB happen to read 5.

For comparison, Python's csv module writes a lone empty field as "" precisely so that the row is not an empty line (csv.writer(f).writerow([None]) writes "").

Expected behavior: a table written by write_csv with default options reads back with the same rows from read_csv with default options. For example, the writer could emit a null in a row with a single column as a quoted empty string ("") or as some other non-empty marker, like the Python csv module does.

For non-string columns, quoting is lossless: with default reader options a quoted "" already reads back as null (ConvertOptions.quoted_strings_can_be_null = True), e.g. csv.read_csv(io.BytesIO(b'"c"\n1\n""\n3\n')) gives [1, None, 3]. For string columns, quoting would make the null read back as an empty string, so the writer may prefer another marker there. Either way, it should not emit an empty line that the default reader ignores.

Versions: pyarrow 27.0.0.dev13 (nightly built from main at 9f03413), Python 3.12, Linux x86_64 (Ubuntu 22.04). The released pyarrow 21.0.0 through 26.0.0 wheels behave the same.

Where it comes from

• Both column populators copy null_string_ followed by the end characters for a null, without quoting and regardless of the number of columns (csv/writer.cc#L277-L286 for UnquotedColumnPopulator, csv/writer.cc#L413-L421 for QuotedColumnPopulator).
• The reader's default is ignore_empty_lines = true. The option's own comment says that only with false does "an empty line represent a single empty value (assuming a one-column CSV file)" (csv/options.h#L59-L61).

Impact: exporting and re-importing one-column data with pyarrow's default CSV options is common, for example a list of ids or a single selected column. Doing so silently loses every null row, which changes row counts and breaks positional alignment with other data. Found by differential testing of CSV round trips.

Component(s)

C++, Python

Component(s)

C++