[Rust] Add pluggable stats exporter for Arrow streams - #820
Open
flaviofcruz wants to merge 1 commit into
Open
Conversation
flaviofcruz
force-pushed
the
rust-stats-exporter
branch
from
September 1, 2026 18:59
8e6dd83 to
9ff19e0
Compare
Add a StatsExporter trait and StreamStat events so callers can route Arrow
Flight stream telemetry into their own metrics systems:
- BatchSent { offset, stats } is emitted when a batch is encoded and sent,
where stats is a BatchStats { records, wire_bytes, uncompressed_bytes }.
wire_bytes is the actual on-wire size (after IPC compression);
uncompressed_bytes is the codec-independent Arrow payload size. It counts
retransmits and fires even if the batch later fails to ack.
- BatchAcked { offset } is a durability-only signal.
- Reconnected { attempt } fires on recovery.
Byte sizes are held in a single current-batch accumulator in the Flight
encoder and flushed when the next batch is pulled, at natural end-of-stream,
and on graceful close. A batch cut off by recovery/rotation is dropped and
re-sent (and re-emitted) on the next connection. A built-in
ChannelExporter/channel_exporter forwards events to a bounded channel,
dropping and counting when full so a slow consumer never stalls ingestion.
Arrow-only (arrow-flight feature); gRPC keeps its AckCallback. No FFI/JNI/PyO3.
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Flavio Cruz <flavio.cruz@databricks.com>
flaviofcruz
force-pushed
the
rust-stats-exporter
branch
from
September 1, 2026 19:50
9ff19e0 to
aa3e66b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
This proposes a different implementation compared to #790. Instead of adding a new
take_offset_details, we introduce a mechanism where a user of the SDK can set a stats exporter. The stats exporter can receive different kinds of SDK events such as BatchSent/BatchAcked/Reconnected which can be consumed by the exporter. As a basic exporter we have a channel exporter which can be used by clients to drain the events and decide what to do with them.The only major complication on the PR is how we handle
BatchSent: because we need uncompressed and wired bytes, these are computed in different places so they need to be coordinated. Ditto for the graceful shutdown case, we have to pushBatchSenton that scenario.Non arrow flight path doesn't support stats exporting for now.
Why: this is required so that clients can understand how many bytes are put on the wire.
How is this tested?
Added a few tests.