Skip to content

feat: prolong individual request queue locks - #74

Merged
janbuchar merged 4 commits into
apify:mainfrom
atirna:fix/prolong-request-lock
Sep 17, 2026
Merged

janbuchar merged 4 commits into
apify:mainfrom
atirna:fix/prolong-request-lock

Conversation

@atirna

@atirna atirna commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Why

Crawlee can extend a request handler timeout, but the native filesystem queue currently has no way to extend the matching on-disk orderNo lock. A long handler can therefore outlive its reservation and be fetched by another shared consumer.

This adds per-request prolongation at the native lock owner. The client records the exact lock value it wrote, extends that expiry, and returns false if the lock expired or another client has since reacquired the request. The Node and Python bindings expose the method. The Node binding supports the companion change in apify/crawlee#4041.

Verification

  • cargo test (91 passed)
  • npm test (60 passed)
  • npm run lint
  • npm run fmt:check

Related to apify/crawlee#3961.

@janbuchar

Copy link
Copy Markdown
Collaborator

Thanks for your contribution @atirna! Could you please also add request lock prolongation support to the python wrapper?

Mirrors the node binding on the python wrapper, in the wrapper's own idiom:
a datetime.timedelta rather than numeric seconds, Arc clone into
future_into_py, and the existing storage_err mapping, matching the
persist_state and set_expected_request_processing_time methods beside it.
Core behaviour is unchanged; the stub is regenerated.
@atirna

atirna commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

done, python wrapper has it now.

went with datetime.timedelta rather than the numeric seconds the node side takes, since that's what set_expected_request_processing_time right next to it already takes and it'd be odd to have two duration conventions in one class. same Arc clone into future_into_py and the existing storage_err mapping as the neighbouring methods. stub regenerated with cargo run --bin stub_gen.

async def prolong_request_lock(self, request_id: builtins.str, duration: datetime.timedelta) -> builtins.bool: ...

7 new cases in tests/test_conversions.py, patterned on the timedelta test already there: regular and forefront queues, a peer client getting False for a lock it doesnt own, expiry through advance_clock_for_testing, refetch after expiry, and a table of bad durations (bare int, float, zero, negative, sub-millisecond).

all 7 fail without the binding and pass with it. python suite 26 passed, node suite 60 passed, core untouched.

the PR title still says feat(node): but it covers both bindings now. happy to retitle to feat: or split the python side into its own PR if you'd rather review them separately.

@atirna atirna changed the title feat(node): prolong individual request queue locks feat: prolong individual request queue locks Sep 17, 2026

@janbuchar janbuchar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@janbuchar
janbuchar merged commit d528cfd into apify:main Sep 17, 2026
10 checks passed
@atirna
atirna deleted the fix/prolong-request-lock branch September 17, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants