Fix RewindAsync non-determinism when the failed step is not the last executed step - #1391
Fix RewindAsync non-determinism when the failed step is not the last executed step#1391wangbill (YunchuWang) wants to merge 8 commits into
Conversation
Rewind rebuilt the orchestration history by filtering on event type plus failed-task IDs. That removed the failed task, but kept everything the orchestrator scheduled *because* it observed the failure - an activity invoked from a catch block, a sub-orchestration, a sent event, or the delay timer RetryInterceptor always creates after the final failed attempt of ScheduleWithRetry. Those leftover scheduling events carry sequence IDs the replayed orchestrator can never reach, because after the rewind the failure is invisible and the orchestrator blocks awaiting the re-scheduled task. Replay then hits the orphan and throws NonDeterministicOrchestrationException, which TaskOrchestrationExecutor converts into a fail-orchestration action - so rewind appeared to "always return" the non-determinism error. Fixes Azure/azure-functions-durable-extension#444. The scrub is now episode-aware. History is divided into episodes delimited by OrchestratorStartedEvent; everything scheduled at or after the episode in which a failure was first observed is removed, along with the events carrying those results (a stale result could otherwise satisfy a different task assigned the same sequence ID). All four event types replay matches against the orchestrator's sequence-ID counter are covered: TaskScheduled, SubOrchestrationInstanceCreated, TimerCreated and EventSent. Fan-out/fan-in is unaffected: parallel branches are scheduled in an episode before the failure is observed, so they are retained. Failed sub-orchestrations are likewise created before the episode that delivers their failure, so their creation event is retained and the child rewind message is still emitted. Applied in both live rewind implementations: - TaskOrchestrationDispatcher.ProcessRewindOrchestrationDecision (SDK layer) - AzureTableTrackingStore.RewindHistoryAsync (Azure Storage) Out-of-repo backends that replicate the SDK-layer scrub server-side (e.g. the Durable Task Scheduler) must apply the same rule; the WARNING comment on ProcessRewindOrchestrationDecision now spells out the contract. Tests: new Test/DurableTask.Core.Tests/RewindTests.cs drives real orchestrations through real episodes, rewinds, and replays (8 cases; the 4 regression cases fail without this change). Two end-to-end scenario tests added for the Azure Storage path, both of which reproduce the issue-444 error without the fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes RewindAsync non-determinism by making the rewind scrub “episode-aware”, ensuring that any work scheduled after a failure is first observed (and the corresponding result events) is removed so the rewound history can always replay deterministically. It applies the same rule both in the SDK-layer scrub (TaskOrchestrationDispatcher.ProcessRewindOrchestrationDecision) and the Azure Storage backend scrub (AzureTableTrackingStore.RewindHistoryAsync), and adds regression/E2E coverage for the previously failing patterns (cleanup work in catch blocks, retry timers, etc.).
Changes:
- Update Core rewind history scrubbing to remove failure-consequence scheduled events (TaskScheduled/SubOrchestrationCreated/TimerCreated/EventSent) and their results, based on episode boundaries.
- Update Azure Table rewind scrubbing to match the same episode-aware rule over stored history entities.
- Add new rewind regression tests (Core) and new Azure Storage end-to-end rewind scenarios.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| Test/DurableTask.Core.Tests/RewindTests.cs | Adds new Core rewind tests to validate replayability after episode-aware scrubbing (note: currently placed under Test/). |
| test/DurableTask.AzureStorage.Tests/AzureStorageScenarioTests.cs | Adds two E2E Azure Storage rewind tests plus supporting orchestrations/activities for cleanup + retry cases. |
| src/DurableTask.Core/TaskOrchestrationDispatcher.cs | Implements episode-aware rewind scrub and expands the contract comment to keep backend implementations in sync. |
| src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs | Implements the equivalent episode-aware scrub for Azure Table history entities. |
Suppressed comments (1)
test/DurableTask.AzureStorage.Tests/AzureStorageScenarioTests.cs:1491
- Same issue as the cleanup test: if an assertion throws,
HelloFailRetryActivity.ShouldFailmay remain flipped andhost.StopAsync()won't run, which can impact subsequent tests. Atry/finallyensures both the flag and the host lifecycle are always reset.
Activities.HelloFailRetryActivity.ShouldFail = true;
await host.StartAsync();
string singletonInstanceId = $"Test_{Guid.NewGuid():N}";
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Test/DurableTask.Core.Tests/RewindTests.cs:14
- These regression tests are added under uppercase
Test/, but the active SDK project andDurableTask.slnreference lowercasetest/DurableTask.Core.Tests(DurableTask.Core.Tests.csproj:1-28,DurableTask.sln:26). On case-sensitive checkouts this file is outside the project directory, so none of these eight tests are compiled or run. Move it totest/DurableTask.Core.Tests/RewindTests.cs.
namespace DurableTask.Core.Tests
- Move RewindTests.cs from Test/ to test/DurableTask.Core.Tests/ to match the location of the test project file. Windows CI is case-insensitive so the tests did compile and run, but the file would be silently excluded from the build on any case-sensitive filesystem. - Replace the hard-coded "TaskScheduledId == 3" assertion with the cleanup task's actual sequence ID read back from the pre-rewind history, and add a general AssertNoOrphanedResultEvents check to AssertReplayable that verifies every result event still refers to a surviving scheduling event. This covers all four scheduling/result pairs and catches stale results that a replay-only assertion cannot see. - Wrap the two new end-to-end rewind tests in try/finally so the shared ShouldFail flag is restored and the host is stopped even when an assertion throws, preventing a failure from cascading into unrelated tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
…tests The Azure Storage history scrub in AzureTableTrackingStore is an independent implementation of the episode-aware rewind rule, but the end-to-end tests only exercised the TaskScheduled and TimerCreated branches of IsScheduledEventType. The SubOrchestrationInstanceCreated and EventSent branches could regress without any Azure Storage test failing. Add two storage-backed rewind scenarios that mirror the corresponding Core cases: one whose catch block starts a sub-orchestration, and one whose catch block raises an event on the orchestration itself. Verified by negative control: removing SubOrchestrationInstanceCreated and EventSent from IsScheduledEventType makes both new tests fail with the original "Non-Deterministic workflow detected" error, each naming its own event type. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
Rewind keeps the parent's execution ID, and a sub-orchestration created without an explicit instance ID is assigned "<parent execution ID>:<sequence ID>". When a consequence SubOrchestrationInstanceCreated event is cleared, the success path can therefore schedule a different sub-orchestration at the same sequence ID and land on the instance ID the cleanup child already occupies. Add an end-to-end test for exactly that shape. It asserts on the orchestration output, so a start message that was dropped or deferred would hang the parent and fail the test. It also asserts that the two children really were assigned the same instance ID, so the scenario cannot silently stop being covered if sequence ID assignment changes. The test passes: ExecutionStarted deduplication in OrchestrationSessionManager only applies to top-level orchestrations (it filters on ParentInstance == null), so a sub-orchestration start message is never deduplicated or deferred. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
…inism' into yunchuwang-fix-rewind-non-determinism
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:418
- This now issues one sequential storage request per removed history event. Since the episode-aware scrub can select most of a long history, rewind latency grows by a network round trip per row and can become impractical or time out. Submit replacement actions in bounded Azure Table transaction batches (respecting the 100-entity and payload limits) instead.
await this.HistoryTable.ReplaceEntityAsync(entity, entity.ETag, cancellationToken);
The SubOrchestrationInstanceFailed rows are what identify the children that still need rewinding, so clearing them before the recursive child rewinds made the operation non-retryable: a transient failure part way through would leave the parent unable to rediscover its failed children, so a retry would treat it as a leaf and revive it while a child was still failed, leaving it waiting forever on the retained sub-orchestration creation event. The two loops touch disjoint entities, so reordering them is otherwise behavior preserving. Also drop the static fields from the instance ID reuse test. The success child now returns its own instance ID so the parent's output carries it, and the cleanup child's instance ID is confirmed by querying its history directly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:435
- These replacements are issued one row at a time in chronological order, so a
TaskFailed/SubOrchestrationInstanceFailedrow can be cleared before later consequence rows. If a later replacement fails transiently, the retry no longer finds that failure, computes an empty (or later)failureEpisode, and can reset the instance while stale consequence events remain, reproducing the non-determinism. Process all non-failure rows first and clear the failure-result rows last (or use atomic partition batches) so a partial attempt remains discoverable and retryable.
foreach (TableEntity entity in entitiesToClear)
{
// "clear" the event by making it a GenericEvent: replay ignores the row while the dummy event preserves the rowKey
entity[nameof(TaskFailedEvent.Reason)] = "Rewound: " + entity.GetString(nameof(HistoryEvent.EventType));
entity[nameof(TaskFailedEvent.EventType)] = nameof(EventType.GenericEvent);
await this.HistoryTable.ReplaceEntityAsync(entity, entity.ETag, cancellationToken);
CI note: the red
|
| Test | Occurrences |
|---|---|
TestTablePartitionManager.EnsureOwnedQueueExclusive |
4 (incl. retries) |
AzureStorageScenarioTests.ScheduledStart_Inline (False) |
1 |
Evidence that both are pre-existing flakes:
EnsureOwnedQueueExclusivefails in 4 of the last 6mainbuilds (301158, 301048, 300949, 300615). Those builds are green only because the ADO retry happened to pass.- Reproduced locally against Azurite on pristine
origin/main(8b6990fd): 5 consecutive runs gavePASS FAIL PASS PASS PASS. It is a timing/lease-sensitive test that flakes independently of this branch. ScheduledStart_Inlinepasses locally on this branch for bothTrueandFalse.- This PR's production diff is confined to
RewindHistoryAsync(AzureTableTrackingStore) andProcessRewindOrchestrationDecision(TaskOrchestrationDispatcher). Neither is reachable from the table partition manager or from scheduled start.
All 12 Rewind* Azure Storage E2E tests passed in this same CI run, including the ones covering the reordered sub-orchestration path:
Passed RewindActivityFail
Passed RewindActivityFailFanOut
Passed RewindActivityFailWithCleanupActivity
Passed RewindActivityFailWithCleanupSubOrchestration
Passed RewindActivityFailWithRetry
Passed RewindActivityFailWithSendEvent
Passed RewindActivityFailWithSubOrchestrationIdReuse
Passed RewindMultipleActivityFail
Passed RewindNestedSubOrchestrationTest
Passed RewindOrchestrationsFail
Passed RewindSubOrchestrationActivityTest
Passed RewindSubOrchestrationsTest
Every other stage (DTFxCoreValidate x13, DTFxEmulatorValidate x13, CodeQL, SDL, CLA, nuget) is green. Happy to re-run the AS legs if a maintainer wants a clean board.
WaitForCompletionAsync returns null when the orchestration does not complete in time, and the test dereferenced the result to derive the expected child instance ID. Assert non-null first so a timeout reports that directly instead of throwing a NullReferenceException. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:435
- These replacements are non-atomic and remain in chronological order, so the
TaskFailedrow is cleared before the consequence scheduling/result rows that follow it. If a later replacement fails transiently, a retry can no longer discover that failure (failureEpisodestaysint.MaxValue), leaves the consequence rows intact, and revives a history that still fails replay. Clear all consequence rows before clearing the failure markers (or update them atomically) so retries can always reconstruct the scrub set.
foreach (TableEntity entity in entitiesToClear)
{
// "clear" the event by making it a GenericEvent: replay ignores the row while the dummy event preserves the rowKey
entity[nameof(TaskFailedEvent.Reason)] = "Rewound: " + entity.GetString(nameof(HistoryEvent.EventType));
entity[nameof(TaskFailedEvent.EventType)] = nameof(EventType.GenericEvent);
await this.HistoryTable.ReplaceEntityAsync(entity, entity.ETag, cancellationToken);
|
/azp run durabletask.public |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
/azp run durabletask.public |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
CI update: the current red board is an Azure Artifacts feed outage, not this PRMy earlier note above covered build 301335 (two pre-existing flaky tests). The board has since gone red for a completely different and much simpler reason: NuGet restore is failing platform-wide. No code is compiled and no test is executed. Evidence Every failing job fails at the restore step, and the feed returns HTTP 500 for stock packages: Same for
The GitHub Actions Why this cannot be caused by this PR
Local verification of the final commit I retried the pipeline twice via |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Fixes Azure/azure-functions-durable-extension#444
Problem
RewindAsyncfails withNon-Deterministic workflow detectedwhenever the failed step was not the last thing the orchestrator did before failing. The most common shape is atry/catchwhere the catch block schedules something (a cleanup activity, a sub-orchestration, an external event) before rethrowing:This was confirmed by Katy Shimizu (@kashimiz) back in 2018 ("the rewind process's cleanup phase fails to scrub the history events of the first
FN.DispatchSignalREvent... the failed step in the orchestrator must be the last step executed before the orchestrator itself fails. This is due to a logical oversight in our implementation") and reactivated by Chris Gillum (@cgillum) "so that we don't forget to actually fix this."Root cause
ProcessRewindOrchestrationDecisionrebuilt the history by filtering only on event type + failed task IDs:That correctly removes the failed task, but it retains every event the orchestrator scheduled as a consequence of observing that failure. After rewind the failure is no longer visible in the history, so on replay the orchestrator takes the success path and blocks awaiting the re-scheduled activity. It never reaches the sequence ID of the leftover
TaskScheduledEvent, andTaskOrchestrationContext.HandleTaskScheduledEventthrowsNonDeterministicOrchestrationException.This is exactly why the failure only shows up when the failed step isn't the last one — if nothing was scheduled after the failure, there is nothing left over to trip on.
Fix
Make the scrub episode-aware. History is divided into episodes delimited by
OrchestratorStartedEvent; an episode boundary is precisely where the orchestrator observed new results and reacted to them.failedTaskIdsandfailureEpisode= the earliest episode in which any failure is delivered.consequenceTaskIds= sequence IDs of everything scheduled at episode >=failureEpisode.Dropping the result events matters: a stale result left behind could otherwise satisfy a different task that later gets assigned the same sequence ID.
Two details that are easy to miss and are handled here:
TaskScheduledEvent,SubOrchestrationInstanceCreatedEvent,TimerCreatedEvent,EventSentEvent. All four are scrubbed.RetryInterceptoralways creates a delay timer after the final failed attempt, soScheduleWithRetryleaves behind aTimerCreatedEventthat is itself a consequence of the failure. Without removing it, rewinding a retried activity produces the timer variant of the same error (scheduled a timer task with sequence number 1 ...).The same rule is applied to the Azure Storage rewind path in
AzureTableTrackingStore.RewindHistoryAsync, which is whatAzureStorageOrchestrationService.RewindTaskOrchestrationAsyncactually calls.Behaviors deliberately preserved
SubOrchestrationInstanceCreatedEventprecedes the episode that delivers the failure, so it is retained and the child rewind message is still emitted.Known tradeoff
Within the failure episode the scrub errs on the side of removing too much. Without re-running orchestrator code there is no way to distinguish "scheduled because of the failure" from "unrelated work that happened to be batched into the same episode". The cost is that a small number of successful tasks may be re-executed on rewind; the benefit is a history that always replays. Given that rewind is an explicit, manual recovery operation on an already-failed instance, and that the alternative is rewind failing outright, this is the right trade. Activities should already be idempotent for rewind to be meaningful at all.
Note for other backends
ProcessRewindOrchestrationDecisionis not the only implementation of this scrub — some backends (notably the Durable Task Scheduler) replicate it server-side. TheWARNINGcomment above the method has been expanded into an explicit contract describing the rule so those implementations can be kept in sync. They will still exhibit this bug until updated.Tests
New
Test/DurableTask.Core.Tests/RewindTests.cs(8 tests) drives real orchestrations through real episodes, rewinds, and replays the result.Regression tests (fail without the fix):
Rewind_CatchBlockSchedulesActivity_ProducesReplayableHistoryRewind_CatchBlockCreatesSubOrchestration_ProducesReplayableHistoryRewind_CatchBlockSendsEvent_ProducesReplayableHistoryRewind_ScheduleWithRetry_RemovesRetryTimersGuard tests (protect existing behavior):
Rewind_SimpleActivityFailure_ReschedulesOnlyTheFailedActivityRewind_FanOutFanIn_RetainsSuccessfulBranchesRewind_FailedSubOrchestration_RetainsCreationAndEmitsChildRewindMessageRewind_AssignsNewExecutionIdTwo end-to-end tests in
AzureStorageScenarioTests.cscover the Azure Storage path against real storage:RewindActivityFailWithCleanupActivityandRewindActivityFailWithRetry.Verification
Both halves of the change were validated with A/B controls rather than by assuming the tests are meaningful.
AzureTableTrackingStorereverted to pre-fixThe pre-fix E2E failures reproduce the original 2018 report verbatim:
and, for the retry variant:
Note: net48 has 7 pre-existing failures in the Core suite (all
TraceHelper_*/Dispatcher_RestartsTraceActivity_ForContinueAsNewStartNewTrace). These were confirmed against a baseline with this change removed (7/127 before vs 7/135 after) and are unrelated to rewind.