Skip to content

Add runtime process memory monitoring to timsseek #150

Description

@jspaezp

Problem

Search slowdowns are hard to attribute: memory pressure and swapping can dominate wall time even when the changed code is not slower. The existing track-alloc feature reports allocator churn and live bytes per phase, but it does not show whether process memory is resident, compressed, or swapped out. It also misses memory outside the tracked allocator, including mapped files.

Proposed work

Add an opt-in, low-overhead runtime memory monitor for timsseek. Sample process memory periodically and at search phase boundaries. Put timestamps and phase names next to the samples so a slowdown can be compared with memory pressure during the same interval. Keep a summary in run_report.json and make live samples available in logs or a simple machine-readable file.

Report at least current and peak resident memory. Where the OS supports it, also report compressed/swapped-out memory and page-in or swap-in activity. Label each metric by its actual meaning; do not present committed or compressed bytes as on-disk swap. On macOS, investigate task_vm_info and proc_pidinfo region counters; on Linux, /proc provides RSS and VmSwap. Mark unavailable metrics as unavailable.

Acceptance criteria

  • A developer can enable sampling for a run and see memory growth or swap activity alongside phase timings.
  • Samples cover library loading and each search phase, including multi-file runs.
  • The run report records peak resident memory and the phase/time of that peak. Live samples have timestamps and phase labels.
  • Metrics document platform semantics and distinguish allocator bytes, resident memory, compressed memory, and swap.
  • Monitoring has bounded storage and modest overhead when enabled; it is off by default.

Related existing feature: timsseek_cli --features track-alloc reports allocation deltas, but cannot diagnose OS memory pressure on its own.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions