Skip to content

[PureGo] Add Avro record format (avro build tag) - #828

Draft
irinatomic-db wants to merge 2 commits into
mainfrom
irina-tomic_data/avro/purego
Draft

irinatomic-db wants to merge 2 commits into
mainfrom
irina-tomic_data/avro/purego

Conversation

@irinatomic-db

@irinatomic-db irinatomic-db commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

PR stack

What changes are proposed in this pull request?

Adds the Avro record format to the pure-Go SDK, behind the avro build tag. Ephemeral streams only; feature in development. Default builds are unchanged — tagged files aren't compiled and no Avro dependency is pulled in.

Declare the writer schema once at stream creation with WithAvro(schemaJSON); it is parsed and validated there, so a malformed schema fails fast. Records are then ingested in one of two ways: pre-encoded bytes via IngestRecordOffset([]byte) / IngestRecordsOffset([][]byte) (raw Avro datums the caller already encoded), or record objects the SDK encodes via IngestAvroRecordOffset(AvroRecord) / IngestAvroRecordsOffset([]AvroRecord), where AvroRecord is a map[string]any.

AvroRecord covers every Avro type in the Avro-to-Delta mapping, including unions (via the Union("branch", v) helper), fixed, decimal, and logical types, using hamba's native value conventions documented on the AvroRecord type.

Implementation notes: object encoding uses github.com/hamba/avro/v2, imported only in //go:build avro files, so the pre-encoded bytes path needs no Avro library and stays available without the tag. The writer schema is parsed once at stream creation and reused across recovery, with no per-record parsing. Object-encode failures return non-retryable errors.

How is this tested?

cd purego

# Default (feature off) — tagged files skipped, no Avro dep
gofmt -l . && go build ./... && go vet ./... && go test ./...

# With the avro tag (all-types encode test, fail-fast test, gated WithAvro tests, e2e)
go build -tags avro ./... && go vet -tags avro ./... && go test -tags avro ./...

# Examples are a separate module; build the avro example with the tag
cd examples && go build -tags avro ./...

CI: ci-purego.yml has an avro job covering all of the above.

Comment thread purego/zerobus/stream.go Outdated
Comment thread purego/zerobus/avro_test.go
Comment thread purego/zerobus/avro_test.go
Signed-off-by: Irina Tomic <irina.tomic@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants