Skip to content

feat(storage): read-only direct CompactStore container mmap (G-EM0.1) - #7

Open
jarmen423 wants to merge 1 commit into
integration/g-em0-1-base-f0-r0from
feature/g-em0-1-direct-mapped-container
Open

feat(storage): read-only direct CompactStore container mmap (G-EM0.1)#7
jarmen423 wants to merge 1 commit into
integration/g-em0-1-base-f0-r0from
feature/g-em0-1-direct-mapped-container

Conversation

@jarmen423

Copy link
Copy Markdown

Summary

Implements G-EM0.1 — read-only direct mapped-container ownership for CompactStore reopen.

  • Lifetime-safe MmapSectionBytes::from_owner bridge (into_bytes)
  • Read-only open uses mmap_section + deserialize_from_mapped_bytes (no read_section_data / full-section Vec copy)
  • Query-visible CompactBacking::ContainerMmap { artifact_id, payload_version, mapped_bytes }
  • Mapping retained on CompactStore until last reader drains
  • Fail-closed StorageError::DirectMmapUnavailable for encrypted / non-mmap-able / empty sections
  • Writable reopens report LegacyEager (not Milestone R evidence)

Packet

  • Packet: G-EM0.1
  • Source base: 06b318e90e96ecb73d8278a6e02c84c6a4d0cece (integration/g-em0-1-base-f0-r0, accepted F0 + R0)
  • Head: see branch tip after push
  • Handoff: /home/jfrie/agent-memory-hosted/.hermes/handoffs/grafeo-disk-backed/G-EM0.1.md

Verification

cargo test -p grafeo-storage --lib --features grafeo-file mmap_ -- --nocapture
cargo test -p grafeo-storage --lib --features grafeo-file bytes_views_retain -- --nocapture
cargo test -p grafeo-engine --features compact-store --test compact_store_direct_mmap -- --nocapture
cargo test -p grafeo-engine --features compact-store \
  --test compact_store_integration \
  --test compact_store_large_string_persistence \
  --test compact_store_allocation_inventory -- --nocapture
cargo test -p grafeo-core --lib --features compact-store graph::compact::section:: -- --nocapture

All green. Pre-existing: cargo clippy -D warnings fails on base missing_errors_doc in buffer/manager.rs (unrelated).

Residual (explicit non-claims)

  • G-EM0.2 still owns v5 mapped proportional graph structures; v4 mapped open still rebuilds some heap dictionaries (accounted separately from file-backed mapped_bytes).
  • Leaf does not merge this PR; parent integration only.

Wire open_read_only to mmap the CompactStore section via a lifetime-safe
Bytes owner, report CompactBacking::ContainerMmap, and fail closed for
encrypted or non-mmap-able layouts without silent eager fallback.
Writable reopens keep LegacyEager and are not Milestone R evidence.
jarmen423 added a commit that referenced this pull request Aug 2, 2026
Replace ColumnEncoder's Vec<Option<Value>> (~40B/row) with family-specific
primitive accumulators seeded from pre-computed geometry. Anonymous retention
drops to packed output width (1 bit/bool, bits/uint, 8B/i64, 8B/f64, 4B/dict-code,
4B/vector-component).

StreamingColumnEncoder:
- from_geometry() locks accumulator family from ColumnGeometry
- push() appends to primitive accumulator (no Value buffering)
- push_placeholder() emits family-neutral placeholder
- finish() builds identical ColumnCodec → byte parity preserved

N-vs-4N acceptance test (250K vs 1M nodes):
- Delta RssAnon ratio: 0.75x (must be < 2.0) ✓
- Payload ratio: 4.01x (expect ~4x) ✓
- Memory does NOT scale with input — pipeline is bounded

GATES:
- grafeo-core: 2130 passed
- grafeo-storage: 240 passed
- grafeo-engine: 1076 passed
- n_vs_4n_peak_memory: PASSED
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant