pool: preserve running jobs across repeated starts

[raphael]

A repeated recovery delivery could remove a running job's ownership and leave a scheduler unable to finish its plan. The worker called JobHandler.Start again for a key it already handled. When a handler rejected that duplicate, failed-start cleanup deleted the original job's saved ownership and payload even though its handler was still running. A local reproduction then showed the scheduler repeatedly failing on that key, never starting the following job or advancing its next planning time.

The pool now handles repeated delivery before calling the handler again. A matching delivery keeps the accepted handler running once. Requests carrying a dispatch ID—the identifier for one admitted submission—still validate their original key and payload. A different pending submission or conflicting recovery payload is rejected without replacing the running job. Replaying a completed submission returns its retained outcome without modifying the current job at that key.

Recovery also restores missing ownership or payload records for an accepted handler. One Redis operation checks the active pool generation, node and worker registrations, pending dispatch, existing payload and other owners before restoring only missing entries. It cannot take ownership from another worker. A failed repair leaves the handler and its local record intact so delivery can be retried.

Starts, stops, dispatch settlement, moves and shutdown requeue now share the worker's existing coordination lock. Incoming recovery deliveries check their current destination after waiting for that lock. Internal restoration after a failed move remains able to restart the original handler; this preserves recovery when either ownership removal or publication of the move fails.

No public types, method signatures, stored fields, resource names or Redis migrations change. Existing callers still submit the same job keys and payloads. JobHandler documentation now states how repeated accepted delivery is handled. Applications adopt the fix by updating Pulse and replacing their workers normally; no ownership reset is required.

Validation performed:

• The deployed base reproduces two handler-start attempts, a still-running handler with no saved owner or payload, and a blocked following scheduler job.
• Focused race tests cover matching and conflicting replay, exact dispatch identity, concurrent starts, missing owner/payload recovery, conflicting stored owners and dispatches, stop/rebalance ordering, failed move restoration, failed-handler cleanup, and settlement during worker removal and node closure.
• The existing orphan-recovery tests pass without restarting accepted handlers.
• go test -race -p 1 ./... -count=1 -timeout=8m passed in 244.76 seconds.
• staticcheck ./... passed.

The main contracts to inspect are Worker.startJob and startJobLocked for delivery versus internal restoration, acceptRunningJob for replay identity, and restoreRunningJobScript for conditional ownership repair. The regression tests follow the complete path through a successful subsequent scheduler plan.

An independent correctness review rechecked the failed-move and orphan-recovery corrections and found no remaining blocker on the tested four-file patch. CI, dependency review and all CodeQL checks passed on 2b43504ee2865b2d95d361a4c4716767443acf6f. Copilot was requested on that same commit, but could not run because the requesting account had reached its review quota; it left no inline findings.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.