fix(spec): avoid unbounded memory growth in Record::serialize by interning field names - #2951
Open
moribundfish-bit wants to merge 1 commit into
Open
fix(spec): avoid unbounded memory growth in Record::serialize by interning field names#2951moribundfish-bit wants to merge 1 commit into
moribundfish-bit wants to merge 1 commit into
Conversation
…rning field names Record::serialize satisfied serde's `&'static str` field-name bound with Box::leak(k.clone().into_boxed_str()) on every field of every record, permanently leaking one allocation per field per serialized record. Workloads that serialize many manifest entries (large fast_append batches, manifest rewrites during maintenance) leak O(records x fields) heap that is never reclaimed for the lifetime of the process. Fix: intern each distinct field name once in a process-lifetime table and reuse the 'static reference thereafter. Total leaked memory drops from O(records x fields) to O(distinct field names) - a handful of schema field names, leaked once, reused forever. The interned name is content-identical to the previously leaked copy, so serialized output is unchanged. Regression tests pin content equality, pointer stability, and bounded growth across 10k records.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No existing issue — the bug is described in full below.
What changes are included in this PR?
Record::serializeincrates/iceberg/src/spec/values/serde.rssatisfies serde'sSerializeStruct::serialize_field(key: &'static str, ...)bound with:This permanently leaks a fresh clone of the field-name string for every field of
every record serialized — memory growth of O(records × fields) that is never
reclaimed for the lifetime of the process. Any long-running process that commits
manifests continuously (e.g. a streaming writer using
fast_append, or a maintenancejob rewriting manifests) exhibits a monotonically increasing heap.
We hit this in production: jemalloc heap profiling attributed ~100% of a steady
multi-hundred-MB/hour heap growth to this one call site, via
write_manifest_file → apache_avro::to_value → RawLiteralEnum::serialize.An offline reproduction (in-memory catalog + local filesystem, no object store)
driving repeated commits showed:
The leak scales with fields-per-commit, not snapshot count or table size.
Box::leakis present and identical in 0.8.0, 0.9.0, 0.10.0 andmain.Note: only
Record::serializeis affected —StringMap::serializecorrectly usesborrowed
&strkeys.The fix
Intern each distinct field name once in a process-lifetime table and reuse the
'staticreference thereafter:Total leaked memory drops from O(records × fields) to O(distinct field names) — a
handful of schema field names, leaked once at warmup, reused forever.
Design notes:
'staticrequirement comes from serde's API, so some leak is unavoidablewithout changing the serialization shape (switching to
serialize_mapwould changethe Avro output from record to map). Interning is the minimal fix.
serialized bytes are unchanged. Validated by writing 16,000 entries with the fix
and reading them all back through the unmodified deserialize path — all entries
round-tripped intact.
write_manifest_filecall, and after warmup every lookup is an O(1) read holdingthe mutex for nanoseconds. The lock is poison-safe (
unwrap_or_else(|e| e.into_inner())).Are these changes tested?
Yes — new
interning_testsmodule pins the two properties that together guaranteeboth the bounded leak and unchanged output:
unchanged)
10,000 simulated records (bounded leak)
Plus: all existing
spec::valuestests pass (115 passed),cargo fmtandcargo clippy --all-targets -- -D warningsare clean.