Skip to content

fix(spec): avoid unbounded memory growth in Record::serialize by interning field names - #2951

Open
moribundfish-bit wants to merge 1 commit into
apache:mainfrom
moribundfish-bit:fix-record-serialize-leak
Open

fix(spec): avoid unbounded memory growth in Record::serialize by interning field names#2951
moribundfish-bit wants to merge 1 commit into
apache:mainfrom
moribundfish-bit:fix-record-serialize-leak

Conversation

@moribundfish-bit

Copy link
Copy Markdown

Which issue does this PR close?

No existing issue — the bug is described in full below.

What changes are included in this PR?

Record::serialize in crates/iceberg/src/spec/values/serde.rs satisfies serde's
SerializeStruct::serialize_field(key: &'static str, ...) bound with:

record.serialize_field(Box::leak(k.clone().into_boxed_str()), &v)?;

This permanently leaks a fresh clone of the field-name string for every field of
every record serialized
— memory growth of O(records × fields) that is never
reclaimed for the lifetime of the process. Any long-running process that commits
manifests continuously (e.g. a streaming writer using fast_append, or a maintenance
job rewriting manifests) exhibits a monotonically increasing heap.

We hit this in production: jemalloc heap profiling attributed ~100% of a steady
multi-hundred-MB/hour heap growth to this one call site, via
write_manifest_file → apache_avro::to_value → RawLiteralEnum::serialize.

An offline reproduction (in-memory catalog + local filesystem, no object store)
driving repeated commits showed:

files/commit heap slope (stock) heap slope (this fix)
2000 +32 KB/commit, monotonic flat (−36 B/commit)
50 flat (leak below noise) flat

The leak scales with fields-per-commit, not snapshot count or table size.
Box::leak is present and identical in 0.8.0, 0.9.0, 0.10.0 and main.
Note: only Record::serialize is affected — StringMap::serialize correctly uses
borrowed &str keys.

The fix

Intern each distinct field name once in a process-lifetime table and reuse the
'static reference thereafter:

record.serialize_field(intern_field_name(k), &v)?;

Total leaked memory drops from O(records × fields) to O(distinct field names) — a
handful of schema field names, leaked once at warmup, reused forever.

Design notes:

  • The 'static requirement comes from serde's API, so some leak is unavoidable
    without changing the serialization shape (switching to serialize_map would change
    the Avro output from record to map). Interning is the minimal fix.
  • The interned name is content-identical to the previously leaked copy, so the
    serialized bytes are unchanged. Validated by writing 16,000 entries with the fix
    and reading them all back through the unmodified deserialize path — all entries
    round-tripped intact.
  • Lock contention is negligible: manifest serialization is sequential within a
    write_manifest_file call, and after warmup every lookup is an O(1) read holding
    the mutex for nanoseconds. The lock is poison-safe (unwrap_or_else(|e| e.into_inner())).

Are these changes tested?

Yes — new interning_tests module pins the two properties that together guarantee
both the bounded leak and unchanged output:

  1. content equality — interned name equals the original text (serialized bytes
    unchanged)
  2. pointer stability — repeated names reuse a single allocation; verified across
    10,000 simulated records (bounded leak)

Plus: all existing spec::values tests pass (115 passed), cargo fmt and
cargo clippy --all-targets -- -D warnings are clean.

…rning field names

Record::serialize satisfied serde's `&'static str` field-name bound with
Box::leak(k.clone().into_boxed_str()) on every field of every record,
permanently leaking one allocation per field per serialized record.
Workloads that serialize many manifest entries (large fast_append
batches, manifest rewrites during maintenance) leak O(records x fields)
heap that is never reclaimed for the lifetime of the process.

Fix: intern each distinct field name once in a process-lifetime table
and reuse the 'static reference thereafter. Total leaked memory drops
from O(records x fields) to O(distinct field names) - a handful of
schema field names, leaked once, reused forever.

The interned name is content-identical to the previously leaked copy,
so serialized output is unchanged. Regression tests pin content
equality, pointer stability, and bounded growth across 10k records.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant