[rustyconover]

Draft until the Arrow format vote. This implements the proposed Arrow format 1.6 change from apache/arrow#49704. The format change is apache/arrow#54959, and the companion drafts are apache/arrow#53994 (C++), apache/arrow-go#1381 (Go) and apache/arrow-rs#11622 (Rust).

DictionaryEncoding gains an optional custom_metadata that holds the metadata of the dictionary value type. It is the IPC counterpart of ArrowSchema.dictionary->metadata, while Field.custom_metadata corresponds to the parent ArrowSchema.metadata. Because nanoarrow works on ArrowSchema directly, the mapping is lossless. The member has the same semantics as Field.custom_metadata, applied to the dictionary values, so this PR writes and reads all keys of ArrowSchema.dictionary->metadata.

• Regenerated header: both commits regenerate flatcc_generated.h with dev/update_fbs.sh (flatcc e3e44836).
  • The first commit regenerates from apache/arrow main as committed, so it only changes comments: upstream docstring fixes and the Tensor/SparseTensor deprecation notes.
  • The second commit regenerates from apache/arrow main plus the draft Schema.fbs change. Its header diff is only the new member.
• Encoder: writes schema->dictionary->metadata (all keys) to DictionaryEncoding.custom_metadata, omitted when NULL or empty, and keeps writing schema->metadata to Field.custom_metadata. Previously the value metadata was silently dropped.
• Decoder: keeps Field.custom_metadata on the parent schema, as before, and now sets schema->dictionary->metadata from DictionaryEncoding.custom_metadata.
  • Messages without the member decode exactly as before. There is no fallback for older files, per the spec.
  • This covers dictionaries nested in struct/list children and footer decoding.
• Footer encoding: carries the member through footer.dictionaries. The dictionary file-writing support from feat: Don't re-emit unchanged dictionaries when writing streams; fix nested dictionary and file writing #928 is now on main.
• Integration JSON: the reader and writer support the optional "metadata" of a field's "dictionary" object. TestingJSONComparison now compares dictionary value metadata at any depth, treating empty as absent.

Tests:

• nanoarrow-only (run in CI):
  • encode/decode round trips that check the flatbuffer directly: extension and arbitrary keys, non-UTF-8 values, nesting in struct and list, alongside field metadata, empty metadata omitted;
  • a footer encode/decode round trip with the dictionary encodings filled in;
  • decoding hand-built messages with and without the member;
  • a writer/reader stream round trip;
  • JSON reader/writer/comparison tests.
• Arrow C++ interop: with NANOARROW_BUILD_TESTS_WITH_ARROW, dictionary(int32, uuid) cases are enabled, including Arrow C++ reading nanoarrow-encoded schemas.
  • They skip at runtime unless the linked Arrow C++ writes the new member, so they skip on released Arrow C++.
  • I ran them locally against an Arrow C++ build of GH-49704: [C++] Read and write DictionaryEncoding.custom_metadata for dictionary value types arrow#53994: 888/888 (886 tests plus the two integration tests), also under ASan/UBSan.

Archery: with the dictionary_extension case enabled for nanoarrow, C Data passes between C++ and nanoarrow in both directions. On main, nanoarrow→C++ fails because the value metadata is lost. Those Archery results predate the dictionary file-writing support merged in #928; the original #622 limitation is no longer the reason to hold this draft.

This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer and an AI reviewer.

🤖 Generated with Claude Code

Rebased onto current main on October 26. Combined dictionary-value metadata comparison with #993's recursive child-metadata comparison and retained both regression tests. Local macOS arm64 Debug validation with Arrow C++ 25.0.1, LZ4 and ZSTD: 100% tests passed out of 929; 175 cases skipped (including unavailable IPC fixture files and format support absent from released Arrow C++). Remains draft pending the Arrow format vote.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 13.63636% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.45%. Comparing base (08d23f6) to head (fda26a6).

Files with missing lines	Patch %	Lines
src/nanoarrow/ipc/encoder.c	9.09%	0 Missing and 10 partials ⚠️
src/nanoarrow/testing/testing.cc	20.00%	0 Missing and 8 partials ⚠️
src/nanoarrow/ipc/decoder.c	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #972      +/-   ##
==========================================
- Coverage   78.52%   78.45%   -0.08%     
==========================================
  Files         106      106              
  Lines       17195    17212      +17     
  Branches     2064     2083      +19     
==========================================
+ Hits        13502    13503       +1     
  Misses       2431     2431              
- Partials     1262     1278      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.