fix(data): bound backfill concurrency

[Faysk]

Summary

Fixes #107.

• scope PostgreSQL connection leases to short read/write operations instead of slow provider I/O;
• bound /backfill/bars upstream fetch concurrency to 8 per data worker by default;
• preserve an admitted request's capacity permit until provider work actually exits after caller cancellation;
• drop requests cancelled while still waiting for admission instead of running their provider work later;
• add focused regression coverage for DB-lease lifetime, admission bounds, admitted cancellation, queued cancellation, and provider failures;
• run the new backfill admission regression tests in the data-service CI job.

Why

Under sustained concurrent factor and live-runner traffic, /backfill/bars could hold PostgreSQL pool leases across provider I/O. Releasing those leases removed the DB-starvation mechanism, but also exposed excessive upstream provider fan-out.

A process-local admission gate keeps provider concurrency bounded while leaving database connections available for short DB operations.

Validation

Controlled issue-107 benchmarks and topology tests showed:

• zero health/control failures in production-like two-worker runs;
• 40/40 concurrent backfills completed successfully;
• each data worker stayed capped at 8 provider fetches;
• the selected B8 limit materially reduced factor tail latency and DB-pool waiting versus the reproduced baseline;
• direct cancellation testing reproduced physical thread amplification with a plain semaphore and verified the final ownership/shielding design keeps admitted physical work within the configured cap.

Focused regression coverage verifies:

• no DB lease is held during provider I/O;
• provider concurrency respects the configured gate;
• admitted thread-backed work keeps its permit after caller cancellation;
• callers cancelled while waiting for admission do not execute later;
• provider failures do not leak DB leases.

Operational note

BACKFILL_MAX_CONCURRENCY is process-local and defaults to 8. Set it to 0 to disable the gate. Production currently runs two data workers, so the container-wide provider peak can be roughly twice the configured per-worker value.

Known follow-up

Concurrent same-key backfills can still duplicate provider work and complete writes out of order. That behavior predates this fix and is intentionally left out of #107 to avoid expanding the saturation fix into cross-worker single-flight/write-order coordination.

[mirror29]

Thanks @Faysk for the thoughtful fix and the detailed benchmarks! The cancellation handling and focused regression tests are especially helpful. I reviewed the changes and ran the five new admission tests plus the existing backfill router/API tests locally: all 27 passed, and Ruff passed for the changed Python files. I found no blocking issues in the review. I have approved the pending CI workflow for this fork PR so the full checks can run before merging. Thanks again for contributing to Inalpha!