fix!: Don't empty the request queue between run() calls

[janbuchar]

Repeated run() calls on the same crawler no longer empty the request queue — the second run continues where the first left off, instead of re-crawling everything locally and silently crawling nothing on Apify. Supersedes the drop-and-recreate approach this PR started with. crawlee-python will need the same change.

• purge() keeps its place on IRequestManager and RequestQueueBackend, it just loses its only automatic caller. A backend that cannot empty a queue in place can throw, and only someone who explicitly called purge() ever sees it.
• The purgeRequestQueue option of run() goes with it, as does the "cannot decide what to purge" arbitration that sameDomainDelaySecs over a supplied manager needed.
• A crawl that processes nothing while its manager holds only handled requests now warns and says why. Catches more than a repeated run() — a second crawler on a shared queue, or a queue a previous process worked through. Starting against handled requests stays silent; that is what resuming looks like.
• Adds Dataset.purge() and KeyValueStore.purge(), which crawlee-python has on Storage and we didn't.
• BasicCrawler.hasFinishedBefore is a read-only accessor now — same value, no longer writable from outside.
• Obsoletes the SDK counterpart (fix: Remove storage purge() methods so that the storage frontends can fill in the functionality apify-sdk-js#694).

[barjin]

One note (related to my comment under the SDK PR), otherwise lgtm ⬇️

[barjin]

This condition looks different for the rest of the storages

crawlee/packages/core/src/storages/dataset.ts

Line 811 in 8cb6a53

if (openContext?.alias === undefined || openContext.alias === DEFAULT_STORAGE_ALIAS) {

Does this mean we can drop run's default queue? Is there an intentional reason for this?

If so, what happens in the case of a migration? Apify Worker will still supply the old queue id through the ACTOR_DEFAULT_REQUEST_QUEUE_ID envvar (nobody told it the queue got deleted) and any subsequent RequestQueue.open() will fail, right?

---

In a way, this is complementary to my comment here .

[janbuchar]

Well, yeah, and crawlee-python already does that. It has downsides for sure, for example that the new queue won't show up correctly in the Apify console.

All this makes me think that we should just drop the purgeRequestQueue flag from BasicCrawler.run, that was the whole motivation for the purge method backfilling anyway. What do you think? 😁

[barjin]

Imo purging the storages on crawler start makes sense only with local projects (where the storage folder can contain previous run's data).

On Apify Platform where each 'script invocation' gets its own set of (run-linked) storages, it makes little sense to me to ever attempt doing this automatically.

Perhaps the purging should be fs-storage's concern in some initialization step?

[janbuchar]

Brace yourselves, this is going to be long.

Two mechanisms here, and we should make sure we're not conflating them:

1. purgeOnStart / CRAWLEE_PURGE_ON_START (purgeDefaultStorages()) — lets you re-run a script against filesystem storage without wiping ./storage by hand. Process-scoped, fires once. Pointless on Apify, where every run gets a fresh set of default storages, but harmless.
2. Purge on repeated crawler.run() — fires from the second run() on the same instance, so the second call doesn't silently crawl nothing.

Does this mean we can drop run's default queue? Is there an intentional reason for this?

Yes, but it was probably not the right decision in this PR.

How it got there: v3 did drop() + reopen too, gated on requestQueue.name === 'default' — and the platform's default queue is unnamed, so it never fired there. v4 on master removed that path in favour of purging in place, which is why a second run() currently (v4) throws on Apify.

Side note — a non-default queue named "default" would get purged on Apify (v3).

This PR brings drop+recreate back as a fallback and re-expresses the gate as "opened by alias", which is true for the default queue on Apify, so the accidental protection went with it. Your migration scenario follows directly — the Worker keeps handing out the old ACTOR_DEFAULT_REQUEST_QUEUE_ID, now pointing at a queue we deleted.

It follows that this PR, at the current state, is a dead end, though. Mechanism 2 (purge on repeated run()) has been inconsistent since 3.x anyway — locally two run() calls crawl twice, on Apify the second does nothing — and I'd rather stop doing it than keep teaching backends to recreate storages behind the user's back.

• purge() stays on IRequestManager and RequestQueueBackend. It just loses its only automatic caller, the hasFinishedBefore block. The Apify implementation can throw, so someone who explicitly calls requestQueue.purge() there gets a clear error instead of silent id churn.
• With nothing calling it automatically, the optional purge?() and the drop-and-recreate fallback aren't needed either — recreateStorage() and StorageOpenContext go, and the inconsistency you spotted goes with the code path it lived in. No SDK counterpart needed after all.
• the purgeRequestQueue option to run() goes away.
• For the case mechanism 2 was protecting against: warn when a repeated run() finishes having crawled nothing, and say why — everything is already handled, a failed request counts as handled, purge the queue or open a fresh one (possibly via alias) if you meant to re-crawl it.

Mechanism 1 stays as-is.

CC @B4nan @vladfrangu @vdusek (if we go through with this, crawlee-python will need to change too)

[janbuchar]

Confirmed with @B4nan and @barjin AFK, let's proceed.

[janbuchar]

Done in fbc54ce

[barjin]

Thank you @janbuchar !

I'm alright with the behaviour, just some notes regarding the docs ⬇️

[barjin]

I'm inclined to log a warning(Once)? on the (>=2)nd crawler start, regardless of the crawl statistics and RQ contents.

People might get partial results (some requests have been processed before, some not), which might, imo, still be upsetting - arguably more than no results at all (harder to spot).

[barjin]

How do you "use a fresh one" on subsequent .run() calls? Can we swap requestManager in an existing crawler instance?

[janbuchar]

Good point. It's possible in v3, but we removed it in v4. Perhaps a setter would be appropriate here, if we prohibited calling it when a run() is in progress.