[fix](arrow-flight) Keep the Doris type of a nested LARGEINT in the Arrow schema

[morningman]

What problem does this PR solve?

Related Jira: DORIS-28389

Problem Summary:

Arrow has no equivalent for LARGEINT, IPV4, IPV6, JSON or VARIANT. Each of them travels over Flight SQL as some other Arrow type and is then indistinguishable from a column that is natively of that type -- a LARGEINT and a STRING both arrive as utf8, an IPV4 and an INT both arrive as int32. The doris_type field metadata is the only thing that tells them apart.

That metadata was attached in get_arrow_schema_from_block and get_arrow_schema_from_expr_ctxs, which only see the top level columns. convert_to_arrow_type recurses into ARRAY / MAP / STRUCT building Arrow types, and Arrow keeps metadata on the Field rather than on the DataType, so every nested element lost it: ListType(item_type) synthesizes a bare "item" field, MapType(key_type, val_type) synthesizes bare "key"/"value" fields, and the STRUCT branch built its fields without going through the metadata helper.

Reading this through a Python ADBC client:

SELECT CAST(495 AS LARGEINT)                        AS scalar_value,
       named_struct('count', CAST(495 AS LARGEINT)) AS struct_value,
       array(CAST(495 AS LARGEINT))                 AS array_value,
       map('k', CAST(495 AS LARGEINT))              AS map_value;

Before -- only the top level column is identifiable:

scalar_value: string   metadata={'doris_type': 'LARGEINT'}
struct_value: struct<count: string>
  count: string        metadata={}
array_value: list<item: string>
  item: string         metadata={}
map_value: map<string, string>
  entries: struct<key: string not null, value: string>
    key: string        metadata={}
    value: string      metadata={}

After:

scalar_value: string   metadata={'doris_type': 'LARGEINT'}
struct_value: struct<count: string>
  count: string        metadata={'doris_type': 'LARGEINT'}
array_value: list<item: string>
  item: string         metadata={'doris_type': 'LARGEINT'}
map_value: map<string, string>
  entries: struct<key: string not null, value: string>
    key: string        metadata={}
    value: string      metadata={'doris_type': 'LARGEINT'}

A nested IPV4 was the worse case of the same defect: it arrives as its 32 bits read as a signed int32 (192.168.1.1 as -1062731519) with nothing left to say it was ever an address.

What is changed?

convert_to_arrow_type now builds the child Fields through create_arrow_field_with_metadata at every level, for ARRAY, MAP and STRUCT. The names and the nullability are exactly the ones Arrow's own constructors produced -- "item" nullable, "key" non-nullable, "value" nullable -- so only the metadata is new, and the record batch builders, which are made from this same schema (FromBlockToRecordBatchConverter reads _schema->field(idx)->type()), are unaffected. DataType::Equals ignores metadata by default, so the batch still type-matches the schema.

The metadata lookup is also completed: JSONB and VARIANT are serialized as utf8 too and were carrying no doris_type at all, not even at the top level. The values (JSON, VARIANT) match what the FE reports for the same column under ARROW:FLIGHT:SQL:TYPE_NAME in FlightSqlSchemaHelper.

No value or Arrow type changes -- this is metadata only.

Release note

Fix Arrow Flight SQL losing the Doris logical type (doris_type field metadata) of LARGEINT / IPV4 / IPV6 nested inside STRUCT, ARRAY and MAP, which made a nested LARGEINT indistinguishable from a STRING for ADBC clients. JSON and VARIANT columns now carry the same metadata as well.

Check List (For Author)

• Test

  • Unit Test

  New be/test/format/arrow/arrow_row_batch_test.cpp, 7 cases:
  nested LARGEINT in STRUCT / ARRAY / MAP value / MAP key; every level of
  array<struct<largeint>> and map<string, array<array<largeint>>>;
  nested IPV4 / IPV6 / JSON / VARIANT; a negative control asserting STRING and
  INT carry no metadata nested or not; an assertion that the nested types still
  compare equal to arrow::list(utf8()) / arrow::map(utf8(), utf8()) with
  check_metadata=false and differ only with check_metadata=true; and one
  that runs convert_to_arrow_batch over the new schema and checks
  ValidateFull(), schema equality including metadata, and the values.

  Verified the assertions are not vacuous: with the arrow_row_batch.cpp change
  reverted, 5 of the 7 fail; the negative control and the batch-building guard
  correctly stay green.

  • Manual test (add detailed scripts or steps below)

  Built BE locally and ran the Jira's repro through
  adbc_driver_flightsql (Python), plus a real table with
  largeint, array<largeint>, struct<count:largeint, name:string>,
  map<string,largeint>, ipv4 and json columns. Nested fields now carry
  doris_type at every depth, a STRING sibling inside the same STRUCT correctly
  carries none, and the values are unchanged (LARGEINT extremes round trip
  losslessly as text).

• Behavior changed:

  • Yes.

  The Arrow schema returned over Flight SQL now carries doris_type field
  metadata on nested fields, and on top level JSON / VARIANT columns. Arrow
  types, field names, nullability and values are unchanged, so a client that
  ignores metadata sees no difference.

• Does this need documentation?

  • No.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FP36MDAsUyKSSDQXQohQsK

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?