[Rust] Add Avro record format (avro feature)

[irinatomic-db]

PR stack

• avro/proto-surface [MERGED]
  • avro/purego
  • avro/rust-core <- this PR
    • avro/ffi-c-abi
      • avro/go
      • avro/cpp
      • avro/dotnet
    • avro/java
    • avro/python
    • avro/typescript

What changes are proposed in this pull request?

Adds the Avro record format to the Rust SDK, behind the off-by-default avro feature (Beta). Ephemeral streams only; server support is pending. Default builds are unchanged.

Declare the writer schema at stream creation with .avro(schema_json), then ingest either a record object the stream encodes against the schema (AvroRecord(AvroValue)) or a pre-encoded datum (AvroBytes) — both via the existing ingest_record_offset / ingest_records_offset. AvroValue is apache_avro::types::Value (re-exported); it can represent every Avro type, so the record path is exhaustive.

Implementation notes:

• apache-avro is an optional dep, pulled only by the avro feature. It requires Rust 1.85; default builds are unaffected.
• Ingest accepts impl Into<PreparedInput> (a #[doc(hidden)] payload type). A blanket From<T: Into<EncodedRecord>> keeps every existing caller unchanged; AvroRecord carries an un-encoded AvroValue.
• The writer schema is parsed once at build() and stored in TableProperties next to the JSON wire form, so it's reused on ingest and survives stream recovery.
• CI compiles, lints, and tests the feature via --features avro steps in ci-rust.yml (the default --workspace / make lint runs don't touch it).

How is this tested?

cd rust

# With the feature (unit + doctests incl. AvroRecord/AvroBytes cases)
cargo test  -p databricks-zerobus-ingest-sdk --features avro
cargo clippy -p databricks-zerobus-ingest-sdk --features avro --all-targets

# Default (feature off) — no regressions
cargo test -p databricks-zerobus-ingest-sdk

# Whole workspace builds/tests (ffi/jni not broken)
cargo build --workspace
cargo test  --workspace --no-run

cargo fmt --all --check

# Excluded example builds standalone
cd examples/avro && cargo build --examples

[danilonajkov-db]

weird that we need this

[elenagaljak-db]

+1 on the weird

[irinatomic-db]

It's a side effect of examples/avro being temporarily excluded from the workspace: the excluded crate builds its own lockfile that we don't track. It goes away once avro is ungated and the crate rejoins members (the root lock takes over). Added a comment explaining it.

[teodordelibasic-db]

We actually probably shouldn't ever "ungate" Avro the same way we didn't "ungate" Arrow. The features flag isn't necessarily for gating a feature, but instead just lets users not pull in parts of the binary they are not interested in. Ideally we would have done this for more parts of the SDK.

[elenagaljak-db]

How do we resolve this problem for other features?

[irinatomic-db]

I clarified the comment.

• The avro feature is permanent (never ungated) - it just lets users who don't need Avro skip apache-avro and its Rust 1.85 floor.
• The workspace exclusion (and this untracked lockfile) is the genuinely temporary part. ffi/jni currently build the sdk without avro and match EncodedRecord/EncodedBatch exhaustively, so making examples/avro a member would unify avro onto them and break those matches (EncodedRecord::Avro(_) not covered).

It goes away when ffi/jni gain Avro support (#830): they'll enable avro on the sdk dependency just like they already do for arrow-flight, add the match arms, and examples/avro rejoins members - at which point this lockfile is tracked like any other.

[teodordelibasic-db]

Not sure if this is planned for a future PR, but cargo test --workspace and make lint (clippy --all) use default features. Every Avro unit test is #[cfg(feature = "avro")]. rust/tests/Cargo.toml enables arrow-flight but not avro. examples/avro is excluded from the workspace, so it has no CI step either. The green checks do not compile this feature.

[teodordelibasic-db]

StreamConfigurationOptions.record_type still lists Proto/Json/Unspecified only.

[teodordelibasic-db]

Let's also fix ingest_record / ingest_records in multiplexed_stream.rs as they still take Into<EncodedRecord>.

pub async fn ingest_record(
    &self,
    payload: impl Into<PreparedInput>,
) -> ZerobusResult<MessageId> {
    self.check_closed()?;
    let idx = self.pick_substream();
    let stream = &self.streams[idx];
    let encoded_batch = stream.prepare_record(payload)?;
    self.enqueue_reserved(stream, idx, encoded_batch).await
}

[teodordelibasic-db]

LGTM once the remaining three comments are addressed and we double check CI. We should also probably not link internal Google Docs in the PR description, but no harm done I guess. Feel free to ping for reapproval.

[irinatomic-db]

@teodordelibasic-db

I've rebased onto main and removed my step from the CI job. The avro tests are running as part of the CI since you've added the --all-features arg. Also cleaned up the README and fixed the examples to encode the record with apache_avro instead of hand-written bytes.

[teodordelibasic-db]

LGTM