GH-39914: [pyarrow] Fall back to schema field type for unparseable complex dtype metadata

[tritsystem]

Rationale for this change

pd.read_parquet() (and Table.to_pandas() generally, with no
types_mapper/dtype_backend="pyarrow") raises TypeError for any
column whose pandas metadata records a complex ArrowDtype -- list,
struct, or dictionary -- as its numpy_type, e.g.
"list<item: string>[pyarrow]". This has been broken since the metadata
round-tripping was introduced, and is the long-standing root cause of
pandas-dev/pandas#53011 (open since 2023, still reproducing today with
no workaround at the pandas level).

import pandas as pd
import pyarrow as pa

data = pd.DataFrame({
    "a": pd.Series([["x"], ["x", "y"]], dtype=pd.ArrowDtype(pa.list_(pa.string()))),
})
data.to_parquet("data.parquet")     # SUCCESS
pd.read_parquet("data.parquet")     # TypeError: data type 'list<item: string>[pyarrow]' not understood

#39914 (this PR's linked issue) previously fixed the case where an
explicit types_mapper / dtype_backend="pyarrow" is passed at read
time -- #44720 resolved those columns earlier in _get_extension_dtypes,
so the buggy string-parsing branch is skipped entirely for that path.
The default backend, and any metadata-only path with no
types_mapper, still hit the original crash -- which is why the pandas
issue has stayed open and users keep re-discovering it (see the thread's
comments through 2025-08).

What changes are included in this PR?

In _get_extension_dtypes(), when pandas_dtype(dtype) raises
TypeError on the metadata's numpy_type string, fall back to building
the ArrowDtype directly from the schema's actual field type
(table.schema.field(name).type) instead of propagating the error. The
schema always carries the precise type regardless of whether the
metadata string happens to be parseable, and this is exactly the pattern
several commenters on the pandas issue independently proposed (e.g.
this comment)
but that was never implemented in a merged fix.

Are these changes tested?

Added test_to_pandas_extension_dtypes_mapping_complex_type_no_types_mapper,
covering both a list and a struct ArrowDtype column round-tripped
through Table.to_pandas() with no types_mapper -- the exact case
that previously crashed. Confirmed red on unpatched code / green after
the fix.

Also manually verified (not just the new test) against pandas 3.0.4 +
pyarrow 24.0.0:

• list, struct, and dictionary ArrowDtype columns round-trip through
  both the default backend and dtype_backend="pyarrow".
• A double write/read/write/read round-trip of a struct column (the
  scenario from this comment)
  still works.
• Plain numpy dtypes and simple extension dtypes (Int64) are
  unaffected -- no regression to the already-working paths.

Are there any user-facing changes?

This PR contains a "Critical Fix".
This makes pd.read_parquet(...) (default backend, no
dtype_backend="pyarrow" needed) work for any DataFrame containing a
complex ArrowDtype column (list/struct/dictionary), which currently
raises an opaque TypeError and has no workaround short of stripping the
pandas metadata before writing.

• GitHub Issue: [Python][Parquet] Parquet files created from Pandas dataframes with Arrow-backed list columns cannot be read by pd.read_parquet #39914

AI Generation Disclosure

I used Claude Code (an AI coding assistant) for this PR. It found the bug by searching for confirmed, still-open, high-severity issues across large open-source projects; root-caused it by tracing pandas_compat.py against the linked pandas issue and the earlier partial fix (#44720); wrote the fallback in _get_extension_dtypes() and the new test; drafted this description; and ran the red/green check and the manual round-trip checks above against a fresh install of pandas 3.0.4 + pyarrow 24.0.0. I'm happy to make any changes and to answer questions about it myself.

[github-actions]

⚠️ GitHub issue #39914 has been automatically assigned in GitHub to PR creator.

[tritsystem]

@jorisvandenbossche, would you be able to take a look, or point me to the right reviewer? This fixes GH-39914: Table.to_pandas() / pd.read_parquet() raise TypeError when the pandas metadata records a list, struct or dictionary ArrowDtype (e.g. "list<item: string>[pyarrow]"), which is the root cause of pandas-dev/pandas#53011. The fix falls back to the schema field's type when that dtype string can't be parsed. Checks are green and the branch merges cleanly.

[github-actions]

⚠️ GitHub issue #39914 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #39914 has been automatically assigned in GitHub to PR creator.

[jorisvandenbossche]

I am fine with adding such a try/except here, but I think the action should then be a continue, i.e. don't treat this column any way specific on the pyarrow side. IMO we should not just assume the users wants an ArrowDtype whenever the dtype fails parsing (that can happen for other reasons as well)