Skip to content

fix(s3): prevent per-shard client and cache growth - #87

Open
durson wants to merge 2 commits into
mainfrom
piotr/memfix
Open

durson wants to merge 2 commits into
mainfrom
piotr/memfix

Conversation

@durson

@durson durson commented Aug 9, 2026

Copy link
Copy Markdown
  • Reuse aiobotocore clients per process and S3 configuration instead of creating a retained client context for every shard. Reset inherited client state in DataLoader workers so clients remain tied to the worker event loop.

  • Bound FileReader's async range cache to a configurable per-reader memory budget and expose presigned retry count for tail-latency tuning. Preserve the previous default of four total presigned request attempts.

  • Bound presigned S3 request timeouts to avoid DataLoader stalls

Added env vars:

  • WSDS_ASYNC_CACHE_MAX_BYTES (max cache size per thread)
  • WSDS_S3_PRESIGNED_RETRIES (configurable number of re-tries)
  • WSDS_S3_PRESIGNED_TIMEOUT_SECONDS
  • WSDS_S3_PRESIGNED_CONNECT_TIMEOUT_SECONDS

Reuse aiobotocore clients per process and S3 configuration instead of creating a retained client context for every shard. Reset inherited client state in DataLoader workers so clients remain tied to the worker event loop.

Bound FileReader's async range cache to a configurable per-reader memory budget and expose presigned retry count for tail-latency tuning. Preserve the previous default of four total presigned request attempts.
@durson
durson requested review from jpc and shahbaz-humeai August 9, 2026 22:47
Add configurable total and connect timeouts, defaulting to 30s and
10s, so stuck range requests fail promptly and allow retries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant