Skip to content

Retry scripts on transient gRPC and pruned-state errors - #57

Merged
janezpodhostnik merged 2 commits into
mainfrom
janez/retry-unavailable-backpressure
Aug 13, 2026
Merged

Retry scripts on transient gRPC and pruned-state errors#57
janezpodhostnik merged 2 commits into
mainfrom
janez/retry-unavailable-backpressure

Conversation

@janezpodhostnik

Copy link
Copy Markdown
Collaborator

Description

Makes the script runner resilient to failures seen during long-running scans:

  • Retry on codes.Unavailable gRPC errors (connection drops); gRPC reconnects on the next attempt, so these are transient.
  • When a batch's reference block state is pruned on the execution nodes, resubmit the whole batch at the latest scanned height once it advances past the batch's height, instead of retrying or splitting at a height that can never succeed. If the batch is within PrunedStateFatalHeightGap of the latest scanned height (or nothing has been scanned yet), the error stays fatal.
  • All other resubmissions (retry, split, exclude) are also moved to the latest scanned height when it is higher, via AddressBatch.WithBlockHeight, which shares done-tracking with the original batch.
  • Reduce the script result channel buffer from 10000 to 500 to apply backpressure to producers earlier and bound memory usage.
  • Skip the empty address in event candidate scanning: it is not a real account (e.g. TokensWithdrawn during the initial FLOW mint has no source) and can never be scanned.

Most critical to review: scanner/script_runner.go (new retry-at-latest-height path and fatal-gap check) and scanner/address_batch.go (WithBlockHeight).


For contributor use:

  • Targeted PR against main branch
  • Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
  • Code follows the standards mentioned here.
  • Updated relevant documentation
  • Re-reviewed Files changed in the Github PR explorer
  • Added appropriate labels

Retry when the connection drops (codes.Unavailable); gRPC reconnects
on the next attempt so these errors are transient.

Reduce the script result channel buffer from 10000 to 500 to apply
backpressure to producers earlier and bound memory usage.
…pruned

When execution nodes prune state for a batch's reference block, retrying
or splitting at the same height can never succeed. Add
ScriptErrorActionRetryAtLatestHeight: the batch is resubmitted whole at
the latest scanned height once it advances past the batch's height.

If the batch is within PrunedStateFatalHeightGap of the latest scanned
height (or nothing has been scanned yet), the error is fatal instead:
state that close to the tip should never be pruned.

All other resubmissions (retry, split, exclude) are also moved to the
latest scanned height when it is higher, so retries never re-execute
against potentially pruned state. AddressBatch.WithBlockHeight returns a
height-adjusted copy that shares done-tracking with the original.

Also skip the empty address in event candidate scanning: it is not a
real account (e.g. TokensWithdrawn during the initial FLOW mint has no
source) and can never be scanned.
@janezpodhostnik
janezpodhostnik merged commit 766714a into main Aug 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant