[PureGo] Add Avro record format (avro build tag) - #828
Draft
irinatomic-db wants to merge 2 commits into
Draft
irinatomic-db wants to merge 2 commits into
irinatomic-db wants to merge 2 commits into
Conversation
This was referenced Sep 3, 2026
irinatomic-db
force-pushed
the
irina-tomic_data/avro/purego
branch
from
September 3, 2026 15:20
e2b0465 to
cc415fa
Compare
irinatomic-db
force-pushed
the
irina-tomic_data/avro/purego
branch
from
September 3, 2026 16:55
cc415fa to
c997412
Compare
This was referenced Sep 3, 2026
irinatomic-db
marked this pull request as ready for review
September 4, 2026 08:09
irinatomic-db
force-pushed
the
irina-tomic_data/avro/purego
branch
from
September 4, 2026 11:58
489c8f9 to
d380dac
Compare
irinatomic-db
force-pushed
the
irina-tomic_data/avro/purego
branch
from
September 4, 2026 15:55
d380dac to
896b63d
Compare
irinatomic-db
requested review from
danilonajkov-db,
elenagaljak-db and
teodordelibasic-db
September 4, 2026 16:28
irinatomic-db
marked this pull request as draft
September 8, 2026 12:57
Signed-off-by: Irina Tomic <irina.tomic@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
irinatomic-db
force-pushed
the
irina-tomic_data/avro/purego
branch
from
September 9, 2026 15:56
896b63d to
1dde577
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR stack
What changes are proposed in this pull request?
Adds the Avro record format to the pure-Go SDK, behind the
avrobuild tag. Ephemeral streams only; feature in development. Default builds are unchanged — tagged files aren't compiled and no Avro dependency is pulled in.Declare the writer schema once at stream creation with
WithAvro(schemaJSON); it is parsed and validated there, so a malformed schema fails fast. Records are then ingested in one of two ways: pre-encoded bytes viaIngestRecordOffset([]byte)/IngestRecordsOffset([][]byte)(raw Avro datums the caller already encoded), or record objects the SDK encodes viaIngestAvroRecordOffset(AvroRecord)/IngestAvroRecordsOffset([]AvroRecord), whereAvroRecordis amap[string]any.AvroRecordcovers every Avro type in the Avro-to-Delta mapping, including unions (via theUnion("branch", v)helper),fixed,decimal, and logical types, using hamba's native value conventions documented on theAvroRecordtype.Implementation notes: object encoding uses
github.com/hamba/avro/v2, imported only in//go:build avrofiles, so the pre-encoded bytes path needs no Avro library and stays available without the tag. The writer schema is parsed once at stream creation and reused across recovery, with no per-record parsing. Object-encode failures return non-retryable errors.How is this tested?
CI:
ci-purego.ymlhas anavrojob covering all of the above.