feat: prolong individual request queue locks - #74
Conversation
|
Thanks for your contribution @atirna! Could you please also add request lock prolongation support to the python wrapper? |
Mirrors the node binding on the python wrapper, in the wrapper's own idiom: a datetime.timedelta rather than numeric seconds, Arc clone into future_into_py, and the existing storage_err mapping, matching the persist_state and set_expected_request_processing_time methods beside it. Core behaviour is unchanged; the stub is regenerated.
|
done, python wrapper has it now. went with async def prolong_request_lock(self, request_id: builtins.str, duration: datetime.timedelta) -> builtins.bool: ...7 new cases in all 7 fail without the binding and pass with it. python suite 26 passed, node suite 60 passed, core untouched. the PR title still says |
Why
Crawlee can extend a request handler timeout, but the native filesystem queue currently has no way to extend the matching on-disk
orderNolock. A long handler can therefore outlive its reservation and be fetched by another shared consumer.This adds per-request prolongation at the native lock owner. The client records the exact lock value it wrote, extends that expiry, and returns
falseif the lock expired or another client has since reacquired the request. The Node and Python bindings expose the method. The Node binding supports the companion change in apify/crawlee#4041.Verification
cargo test(91 passed)npm test(60 passed)npm run lintnpm run fmt:checkRelated to apify/crawlee#3961.