[All] Add avro record type to proto surface

[irinatomic-db]

PR stack

• avro/proto-surface <- this PR
  • avro/purego
  • avro/rust-core
    • avro/ffi-c-abi
      • avro/go
      • avro/cpp
      • avro/dotnet
    • avro/java
    • avro/python
    • avro/typescript

What changes are proposed in this pull request?

Adds the Avro record type to the proto surface (wire contract only) so later PRs can build Avro ingestion on ephemeral streams. No public SDK API and no behavior change yet.

• Proto (rust/sdk/zerobus_service.proto):
  • reserve 3 (ARROW_IPC on the service - this SDK models Arrow as a separate Arrow Flight stream, not a RecordType)
  • add AVRO = 4;
  • add AvroRecordBatch, CreateIngestStreamRequest.avro_schema_json, IngestRecordRequest.avro_encoded_record, and IngestRecordBatchRequest.avro_batch.
• purego: regenerate internal/zerobuspb bindings from the schema. The commit also resyncs pre-existing drift (proto doc-comments + gofmt) and adds a generate CI job that fails if the committed bindings are stale = the gate purego was missing.
• Rust: set avro_schema_json: None at the one CreateIngestStreamRequest construction site so codegen compiles.

How is this tested?

# Rust - proto compiles + no regressions
cd rust
cargo build -p databricks-zerobus-ingest-sdk
cargo test  -p databricks-zerobus-ingest-sdk

# purego - bindings match the schema (the new CI gate), plus build/vet/test
cd ../purego/internal/zerobuspb
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.6.2
go generate ./...
cd ../../..
git diff --exit-code -- purego/internal/zerobuspb/   # clean = bindings in sync

cd purego
go build ./...
go vet ./...
go test ./...

Code diff

Most of the code diff is generated (purego/internal files): +1005 -87

[danilonajkov-db]

LGTM, just check with Teodor about the CI

[teodordelibasic-db]

Ah, this is a preexisting issue but has only now surfaced/come to my attention. Adding this is a potential breaking change for some clients. The code that would break is not useful in any way I think, but it's still breaking, for example:

use databricks_zerobus_ingest_sdk::databricks::zerobus::RecordType;
use databricks_zerobus_ingest_sdk::StreamConfigurationOptions;
fn describe(opts: &StreamConfigurationOptions) -> &'static str {
  match opts.record_type {
    RecordType::Proto => "proto",
    RecordType::Json => "json",
    RecordType::Unspecified => "unspecified",
  }
}

After this PR this match would break. This is true also for some interactions with other generated types from this file. Due to this you had to change wrapper SDKs in this change as well.

IMO we shouldn't treat this as a regular API breaking change that requires a major version bump, but instead just note it in NEXT_CHANGELOG.md and then handle it later the proper way during a major version bump so that we don't have to think about it anymore. I've created an issue for this long term fix: [Rust] Hide generated gRPC types so additive proto changes are not crate-breaking.

Let's discuss with other folks offline as well. cc: @davidtosovic-db

[teodordelibasic-db]

LGTM