[PureGo] Build Arrow Flight ingestion (Beta)

[zlata-stefanovic-db]

Summary

Build Arrow Flight ingestion for the pure-Go SDK: an ArrowStream (Beta) that
ingests arrow.RecordBatch values or raw Arrow IPC bytes over Flight DoPut,
reusing the generic internal/stream core that proto and JSON already share.

Status: in progress, 2 of 10 steps done. The durability groundwork (#682) and
the Arrow IPC payload (#758) are merged; raw IPC input (#768) is open. The Flight
wire path, ack model, and public API remain. Nothing is reachable from a public
API yet.

This is the last protocol gap in #467, which released purego/v0.1.0 with proto
and JSON only.

Motivation

Every other SDK already has Arrow Flight — Rust (#282), Python (#122), Java
(#141), TypeScript (#127), Go/cgo (#134), C++ (#474). PureGo shipped v0.1.0
without it, so a caller who wants Arrow ingestion in Go still has to take the
cgo/FFI SDK and, with it, the prebuilt Rust archive and the ffi/v* release
cadence that #467 set out to escape. Arrow is also the throughput path: callers
who already hold columnar data pay a full row-by-row re-encode to use the proto
API.

The hard part is not Flight itself, it is that proto/JSON and Arrow cannot
share a wire path, and the ack semantics differ underneath:

	Proto / JSON	Arrow
RPC	EphemeralStream (one bidi gRPC stream)	Flight DoPut (separate FlightClient)
Wire unit	a record or a batch, atomic per offset — the server never acks partway through one	a whole RecordBatch, re-chunked by rows at 2 MiB, so one batch spans several wire messages
Ack model	durable up to offset N	durable up to record-count N
Recovery	re-send unacked records	re-send unacked batches, slicing partially acked ones

Because chunking lets a server ack land mid-batch, a by-batch ack model cannot
express "4,200 of this batch's 5,000 rows are durable" — on reconnect it would
replay all 5,000 and duplicate 4,200 already-durable rows.

The anti-pattern to avoid is the cgo SDK (go/), which answered this divergence
with two parallel stacks sharing no Go code (ZerobusStream vs
ZerobusArrowStream): duplicated Flush/Close/WaitForOffset, a repeated
interface{} type-switch, and two overlapping config structs. The Rust core has
a standing TODO to unify the same split. The bar for this work is that Arrow
plugs into the existing core without editing a single file under
internal/stream. If a core change turns out to be unavoidable, that is a
signal the seam is wrong and the seam should be fixed rather than special-casing
Arrow inside the core.

How the work was split

The whole path was first drafted as one branch — #681, +9,070/-319 across 35
files — mixing the durability semantics, the encoder, the Flight transport, the
ack model, and the public API. That was not reviewable, so it was closed on
2026-08-06 and is being landed as independent pieces, riskiest semantics first.

The split has held up: #682 let the durability change be reviewed against
proto/JSON behavior that must not change, rather than hiding inside a
9,000-line Arrow PR, and every Arrow PR since has been testable in-process
against Arrow with no Flight connection and no test double standing in for one.

The remaining encoder work lives in the #732 umbrella (+3,386/-16 across 10
files), which is being split into steps 3-6 below.

Checklist

• 2026-08-17 — 1. Protocol-neutral durability in the core — [PureGo] Generalize stream durability model #682.
  Generalizes the core from "one offset = one atomic ack" to durability units,
  and turns every protocol seam into an injected hook so Arrow can live in its
  own package. Arrow-free by construction; proto and JSON behavior unchanged.
• 2026-08-20 — 2. Arrow IPC payload — [PureGo] Add Arrow IPC payload #758. internal/arrowproto with
  the Payload and Protocol types and the typed encoding path: eager
  materialization into a canonical self-contained IPC stream, a genuine
  row-range Slice for partial-ack replay, and slice-aware admission sizing.
• 3. Raw Arrow IPC bytes as an input — [PureGo] Accept Arrow IPC bytes as ingest input #768 (open). EncodeIPC for
  callers who already hold IPC bytes, plus the flatbuffer preflight that sizes
  a compressed stream by its declared uncompressed buffers rather than its
  wire length.
• 4. Flight frame encoding. The 2 MiB chunk plan and the frame emitter.
  Chunking is measured, not estimated: frames are capped by binary
  searching actual encoded protobuf size inside an exponentially grown
  bracket, because compressed bytes per row cannot be predicted from a row
  count. StampOffset stays a no-op — Arrow's wire offset lives in each
  frame's app_metadata and is stamped by the emitter, so a payload is
  offset-independent and needs no re-stamp when replayed on a new connection.
• 5. Flight acknowledgment and batch metadata parsing in
  internal/transport: strict JSON parsing of app_metadata for offset_id,
  ack_up_to_records, and the pause/rotation duration.
• 6. stream.EncoderHooks and core wiring, including reconciling the
  admission reservation against the payload's actual retained size once
  materialization has happened.
• 7. transport.FlightStream. A Flight DoPut wire stream embedding the
  existing rawStream[Req, Resp] with the two Arrow handshake hooks —
  sendSetup sends the schema, confirmReady waits for the ready sentinel —
  mirroring how the EphemeralStream path already uses it. Exposed as an
  OpenFunc, so wirestream.go needs no change.
• 8. AckModelHooks.Resolve, mapping the server's cumulative
  ack_up_to_records onto the submitted ranges the core hands it, reusing the
  core's exported ResolveAcknowledgedUnits range validation rather than
  reimplementing it.
• 9. Public Beta ArrowStream API. Typed and IPC ingestion, Arrow-only
  options scoped to the Arrow surface rather than silently accepted by a
  proto/JSON stream, and unacked-batch access for recovery. Arrow callers need
  a typed batch input, so this is a type-appropriate wrapper over the shared
  core rather than forcing arrow.RecordBatch through the byte API.
• 10. Recovery, tests, docs, and release. Partial replay, pause/rotation,
  auth retry, and Arrow error mapping; a mock Flight server and the full
  lifecycle matrix; README, examples, and changelog; then purego/v0.2.0.

Resolved design decisions

• Wire path — Flight DoPut with its own FlightClient, matching Rust,
  rather than adding an Arrow arm to the existing EphemeralStream RPC. Additive,
  client-only, no server-contract change, and the Arrow dependency stays contained
  in this module.
• Partial-ack recovery — client-side slicing, the same choice Rust made. The
  core does not rely on the server deduplicating a replayed prefix, so no
  service-team confirmation is needed.
• Where record counts enter the ack model — the core passes the outstanding
  submitted ranges at resolution time (resolve(resp, AckState)) rather than the
  ack model accumulating a history of what was sent. The ack model stays
  stateless and the receiver's send ledger remains the single owner of send
  history.
• Eager materialization — payloads hold serialized IPC bytes, never live
  caller-owned Arrow objects, so the core can hold a payload across a reconnect
  without pinning caller-owned Arrow arrays. Matches the proto/JSON encoders and
  the Rust core.
• Package boundary — Arrow lives in internal/arrowproto and plugs in
  through hooks; no file under internal/stream is edited.
• Dependencies — github.com/apache/arrow-go/v18 and
  github.com/google/flatbuffers, the module's first beyond gRPC and protobuf.
  arrow-go raises the gRPC minimum from v1.81.1 to v1.82.0.

Open design questions

Differences from the Rust core, all current implementation choices rather than
regressions, to settle before the Beta is called done:

• Backpressure model. PureGo admission is byte-based (size estimate plus
  reservation); the Rust Arrow stream is count-based
  (max_inflight_batches). Under large or highly compressible input the two
  apply backpressure at different points. Byte-based admission is also what makes
  the estimate's accuracy load-bearing: reserve hard-rejects any single weight
  above MaxBufferedPayloadBytes, so an over-estimate is refused outright rather
  than corrected by the reconciliation that follows materialization.
• Compressed IPC expansion guard. PureGo preflights declared uncompressed
  sizes; Rust materializes IPC directly in ingest_ipc_batch. PureGo will reject
  some compressed payloads that Rust accepts.
• Dictionary wire policy. Rust's Flight encoding defaults to dictionary
  hydration, whereas the PureGo payload contract preserves canonical IPC with
  schema, dictionary, and batch messages. Once Flight framing is compared
  end to end these may need explicit alignment.

Related

• [PureGo] Design a pure-Go SDK (no cgo/FFI) #467 — the pure-Go SDK, where Arrow is the last remaining protocol
• [PureGo] Add Arrow Flight ingestion #681 — the original monolithic Arrow draft, closed and split
• [PureGo] Generalize stream durability model #682, [PureGo] Add Arrow IPC payload #758, [PureGo] Accept Arrow IPC bytes as ingest input #768, [PureGo] Add Arrow IPC and Flight frame encoder #732 — the pieces above
• [Rust] Arrow Flight Beta #282, [Python] Add experimental Arrow Flight support #122, [Java] Add experimental Arrow Flight support #141, [TypeScript] Add experimental Arrow Flight support #127, [Go] Add experimental Arrow Flight ingestion support #134, [C++] Arrow Flight stream (Beta) #474 — Arrow Flight in the other SDKs