Skip to content

Fix RewindAsync non-determinism when the failed step is not the last executed step - #1391

Draft
wangbill (YunchuWang) wants to merge 8 commits into
mainfrom
yunchuwang-fix-rewind-non-determinism
Draft

Fix RewindAsync non-determinism when the failed step is not the last executed step#1391
wangbill (YunchuWang) wants to merge 8 commits into
mainfrom
yunchuwang-fix-rewind-non-determinism

Conversation

@YunchuWang

Copy link
Copy Markdown
Member

Fixes Azure/azure-functions-durable-extension#444

Problem

RewindAsync fails with Non-Deterministic workflow detected whenever the failed step was not the last thing the orchestrator did before failing. The most common shape is a try/catch where the catch block schedules something (a cleanup activity, a sub-orchestration, an external event) before rethrowing:

try
{
    await context.ScheduleTask<string>(nameof(Foo), "");   // fails
}
catch (Exception)
{
    await context.ScheduleTask<string>(nameof(Cleanup), ""); // <-- scheduled *because* of the failure
    throw;
}

This was confirmed by Katy Shimizu (@kashimiz) back in 2018 ("the rewind process's cleanup phase fails to scrub the history events of the first FN.DispatchSignalREvent ... the failed step in the orchestrator must be the last step executed before the orchestrator itself fails. This is due to a logical oversight in our implementation") and reactivated by Chris Gillum (@cgillum) "so that we don't forget to actually fix this."

Root cause

ProcessRewindOrchestrationDecision rebuilt the history by filtering only on event type + failed task IDs:

if (!(evt is TaskScheduledEvent ts && failedTaskIds.Contains(ts.EventId))
    && evt is not TaskFailedEvent
    && evt is not SubOrchestrationInstanceFailedEvent
    && evt is not ExecutionCompletedEvent)

That correctly removes the failed task, but it retains every event the orchestrator scheduled as a consequence of observing that failure. After rewind the failure is no longer visible in the history, so on replay the orchestrator takes the success path and blocks awaiting the re-scheduled activity. It never reaches the sequence ID of the leftover TaskScheduledEvent, and TaskOrchestrationContext.HandleTaskScheduledEvent throws NonDeterministicOrchestrationException.

This is exactly why the failure only shows up when the failed step isn't the last one — if nothing was scheduled after the failure, there is nothing left over to trip on.

Fix

Make the scrub episode-aware. History is divided into episodes delimited by OrchestratorStartedEvent; an episode boundary is precisely where the orchestrator observed new results and reacted to them.

  1. Collect failedTaskIds and failureEpisode = the earliest episode in which any failure is delivered.
  2. Collect consequenceTaskIds = sequence IDs of everything scheduled at episode >= failureEpisode.
  3. Rebuild, dropping the failed/consequence scheduling events and their corresponding result events.

Dropping the result events matters: a stale result left behind could otherwise satisfy a different task that later gets assigned the same sequence ID.

Two details that are easy to miss and are handled here:

  • Replay matches the orchestrator's sequence-ID counter for four event types, not just task scheduling: TaskScheduledEvent, SubOrchestrationInstanceCreatedEvent, TimerCreatedEvent, EventSentEvent. All four are scrubbed.
  • RetryInterceptor always creates a delay timer after the final failed attempt, so ScheduleWithRetry leaves behind a TimerCreatedEvent that is itself a consequence of the failure. Without removing it, rewinding a retried activity produces the timer variant of the same error (scheduled a timer task with sequence number 1 ...).

The same rule is applied to the Azure Storage rewind path in AzureTableTrackingStore.RewindHistoryAsync, which is what AzureStorageOrchestrationService.RewindTaskOrchestrationAsync actually calls.

Behaviors deliberately preserved

  • Fan-out/fan-in: parallel branches are all scheduled in an episode before the failure is observed, so they are retained and not re-executed.
  • Failed sub-orchestrations: the SubOrchestrationInstanceCreatedEvent precedes the episode that delivers the failure, so it is retained and the child rewind message is still emitted.

Known tradeoff

Within the failure episode the scrub errs on the side of removing too much. Without re-running orchestrator code there is no way to distinguish "scheduled because of the failure" from "unrelated work that happened to be batched into the same episode". The cost is that a small number of successful tasks may be re-executed on rewind; the benefit is a history that always replays. Given that rewind is an explicit, manual recovery operation on an already-failed instance, and that the alternative is rewind failing outright, this is the right trade. Activities should already be idempotent for rewind to be meaningful at all.

Note for other backends

ProcessRewindOrchestrationDecision is not the only implementation of this scrub — some backends (notably the Durable Task Scheduler) replicate it server-side. The WARNING comment above the method has been expanded into an explicit contract describing the rule so those implementations can be kept in sync. They will still exhibit this bug until updated.

Tests

New Test/DurableTask.Core.Tests/RewindTests.cs (8 tests) drives real orchestrations through real episodes, rewinds, and replays the result.

Regression tests (fail without the fix):

  • Rewind_CatchBlockSchedulesActivity_ProducesReplayableHistory
  • Rewind_CatchBlockCreatesSubOrchestration_ProducesReplayableHistory
  • Rewind_CatchBlockSendsEvent_ProducesReplayableHistory
  • Rewind_ScheduleWithRetry_RemovesRetryTimers

Guard tests (protect existing behavior):

  • Rewind_SimpleActivityFailure_ReschedulesOnlyTheFailedActivity
  • Rewind_FanOutFanIn_RetainsSuccessfulBranches
  • Rewind_FailedSubOrchestration_RetainsCreationAndEmitsChildRewindMessage
  • Rewind_AssignsNewExecutionId

Two end-to-end tests in AzureStorageScenarioTests.cs cover the Azure Storage path against real storage: RewindActivityFailWithCleanupActivity and RewindActivityFailWithRetry.

Verification

Both halves of the change were validated with A/B controls rather than by assuming the tests are meaningful.

Check Result
New Core tests, net8.0 + net48 8/8 pass
Core tests with the new logic surgically neutralized 4 fail — the regression tests genuinely catch the bug
Full Core suite, net8.0 147/147 pass
E2E tests with AzureTableTrackingStore reverted to pre-fix both fail with the exact issue #444 message
E2E tests with the fix both pass

The pre-fix E2E failures reproduce the original 2018 report verbatim:

Non-Deterministic workflow detected: A previous execution of this orchestration scheduled
an activity task with sequence ID 1 and name '...Hello' (version ''), but the current replay
execution hasn't (yet?) scheduled this task.

and, for the retry variant:

Non-Deterministic workflow detected: A previous execution of this orchestration scheduled
a timer task with sequence number 1 but the current replay execution hasn't (yet?) scheduled this task.

Note: net48 has 7 pre-existing failures in the Core suite (all TraceHelper_* / Dispatcher_RestartsTraceActivity_ForContinueAsNewStartNewTrace). These were confirmed against a baseline with this change removed (7/127 before vs 7/135 after) and are unrelated to rewind.

Rewind rebuilt the orchestration history by filtering on event type plus
failed-task IDs. That removed the failed task, but kept everything the
orchestrator scheduled *because* it observed the failure - an activity
invoked from a catch block, a sub-orchestration, a sent event, or the
delay timer RetryInterceptor always creates after the final failed
attempt of ScheduleWithRetry.

Those leftover scheduling events carry sequence IDs the replayed
orchestrator can never reach, because after the rewind the failure is
invisible and the orchestrator blocks awaiting the re-scheduled task.
Replay then hits the orphan and throws
NonDeterministicOrchestrationException, which TaskOrchestrationExecutor
converts into a fail-orchestration action - so rewind appeared to
"always return" the non-determinism error.

Fixes Azure/azure-functions-durable-extension#444.

The scrub is now episode-aware. History is divided into episodes
delimited by OrchestratorStartedEvent; everything scheduled at or after
the episode in which a failure was first observed is removed, along with
the events carrying those results (a stale result could otherwise
satisfy a different task assigned the same sequence ID). All four event
types replay matches against the orchestrator's sequence-ID counter are
covered: TaskScheduled, SubOrchestrationInstanceCreated, TimerCreated
and EventSent.

Fan-out/fan-in is unaffected: parallel branches are scheduled in an
episode before the failure is observed, so they are retained. Failed
sub-orchestrations are likewise created before the episode that delivers
their failure, so their creation event is retained and the child rewind
message is still emitted.

Applied in both live rewind implementations:
  - TaskOrchestrationDispatcher.ProcessRewindOrchestrationDecision (SDK layer)
  - AzureTableTrackingStore.RewindHistoryAsync (Azure Storage)

Out-of-repo backends that replicate the SDK-layer scrub server-side
(e.g. the Durable Task Scheduler) must apply the same rule; the WARNING
comment on ProcessRewindOrchestrationDecision now spells out the
contract.

Tests: new Test/DurableTask.Core.Tests/RewindTests.cs drives real
orchestrations through real episodes, rewinds, and replays (8 cases; the
4 regression cases fail without this change). Two end-to-end scenario
tests added for the Azure Storage path, both of which reproduce the
issue-444 error without the fix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 25, 2026 20:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes RewindAsync non-determinism by making the rewind scrub “episode-aware”, ensuring that any work scheduled after a failure is first observed (and the corresponding result events) is removed so the rewound history can always replay deterministically. It applies the same rule both in the SDK-layer scrub (TaskOrchestrationDispatcher.ProcessRewindOrchestrationDecision) and the Azure Storage backend scrub (AzureTableTrackingStore.RewindHistoryAsync), and adds regression/E2E coverage for the previously failing patterns (cleanup work in catch blocks, retry timers, etc.).

Changes:

  • Update Core rewind history scrubbing to remove failure-consequence scheduled events (TaskScheduled/SubOrchestrationCreated/TimerCreated/EventSent) and their results, based on episode boundaries.
  • Update Azure Table rewind scrubbing to match the same episode-aware rule over stored history entities.
  • Add new rewind regression tests (Core) and new Azure Storage end-to-end rewind scenarios.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
Test/DurableTask.Core.Tests/RewindTests.cs Adds new Core rewind tests to validate replayability after episode-aware scrubbing (note: currently placed under Test/).
test/DurableTask.AzureStorage.Tests/AzureStorageScenarioTests.cs Adds two E2E Azure Storage rewind tests plus supporting orchestrations/activities for cleanup + retry cases.
src/DurableTask.Core/TaskOrchestrationDispatcher.cs Implements episode-aware rewind scrub and expands the contract comment to keep backend implementations in sync.
src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs Implements the equivalent episode-aware scrub for Azure Table history entities.
Suppressed comments (1)

test/DurableTask.AzureStorage.Tests/AzureStorageScenarioTests.cs:1491

  • Same issue as the cleanup test: if an assertion throws, HelloFailRetryActivity.ShouldFail may remain flipped and host.StopAsync() won't run, which can impact subsequent tests. A try/finally ensures both the flag and the host lifecycle are always reset.
                Activities.HelloFailRetryActivity.ShouldFail = true;
                await host.StartAsync();

                string singletonInstanceId = $"Test_{Guid.NewGuid():N}";


💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/DurableTask.Core.Tests/RewindTests.cs
Comment thread test/DurableTask.Core.Tests/RewindTests.cs
Comment thread test/DurableTask.AzureStorage.Tests/AzureStorageScenarioTests.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Test/DurableTask.Core.Tests/RewindTests.cs:14

  • These regression tests are added under uppercase Test/, but the active SDK project and DurableTask.sln reference lowercase test/DurableTask.Core.Tests (DurableTask.Core.Tests.csproj:1-28, DurableTask.sln:26). On case-sensitive checkouts this file is outside the project directory, so none of these eight tests are compiled or run. Move it to test/DurableTask.Core.Tests/RewindTests.cs.
namespace DurableTask.Core.Tests

- Move RewindTests.cs from Test/ to test/DurableTask.Core.Tests/ to match the
  location of the test project file. Windows CI is case-insensitive so the tests
  did compile and run, but the file would be silently excluded from the build on
  any case-sensitive filesystem.

- Replace the hard-coded "TaskScheduledId == 3" assertion with the cleanup task's
  actual sequence ID read back from the pre-rewind history, and add a general
  AssertNoOrphanedResultEvents check to AssertReplayable that verifies every
  result event still refers to a surviving scheduling event. This covers all four
  scheduling/result pairs and catches stale results that a replay-only assertion
  cannot see.

- Wrap the two new end-to-end rewind tests in try/finally so the shared
  ShouldFail flag is restored and the host is stopped even when an assertion
  throws, preventing a failure from cascading into unrelated tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
Copilot AI review requested due to automatic review settings August 27, 2026 21:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs
…tests

The Azure Storage history scrub in AzureTableTrackingStore is an independent
implementation of the episode-aware rewind rule, but the end-to-end tests only
exercised the TaskScheduled and TimerCreated branches of IsScheduledEventType.
The SubOrchestrationInstanceCreated and EventSent branches could regress without
any Azure Storage test failing.

Add two storage-backed rewind scenarios that mirror the corresponding Core
cases: one whose catch block starts a sub-orchestration, and one whose catch
block raises an event on the orchestration itself.

Verified by negative control: removing SubOrchestrationInstanceCreated and
EventSent from IsScheduledEventType makes both new tests fail with the original
"Non-Deterministic workflow detected" error, each naming its own event type.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
Copilot AI review requested due to automatic review settings August 27, 2026 22:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

@YunchuWang
wangbill (YunchuWang) marked this pull request as draft August 31, 2026 14:29
Copilot AI review requested due to automatic review settings August 31, 2026 14:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs
wangbill (YunchuWang) and others added 2 commits August 31, 2026 10:42
Rewind keeps the parent's execution ID, and a sub-orchestration created without
an explicit instance ID is assigned "<parent execution ID>:<sequence ID>". When
a consequence SubOrchestrationInstanceCreated event is cleared, the success path
can therefore schedule a different sub-orchestration at the same sequence ID and
land on the instance ID the cleanup child already occupies.

Add an end-to-end test for exactly that shape. It asserts on the orchestration
output, so a start message that was dropped or deferred would hang the parent
and fail the test. It also asserts that the two children really were assigned
the same instance ID, so the scenario cannot silently stop being covered if
sequence ID assignment changes.

The test passes: ExecutionStarted deduplication in OrchestrationSessionManager
only applies to top-level orchestrations (it filters on ParentInstance == null),
so a sub-orchestration start message is never deduplicated or deferred.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
…inism' into yunchuwang-fix-rewind-non-determinism
Copilot AI review requested due to automatic review settings August 31, 2026 14:44
Comment thread test/DurableTask.AzureStorage.Tests/AzureStorageScenarioTests.cs Fixed
Comment thread test/DurableTask.AzureStorage.Tests/AzureStorageScenarioTests.cs Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:418

  • This now issues one sequential storage request per removed history event. Since the episode-aware scrub can select most of a long history, rewind latency grows by a network round trip per row and can become impractical or time out. Submit replacement actions in bounded Azure Table transaction batches (respecting the 100-entity and payload limits) instead.
                await this.HistoryTable.ReplaceEntityAsync(entity, entity.ETag, cancellationToken);

Comment thread src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs Outdated
The SubOrchestrationInstanceFailed rows are what identify the children that
still need rewinding, so clearing them before the recursive child rewinds made
the operation non-retryable: a transient failure part way through would leave
the parent unable to rediscover its failed children, so a retry would treat it
as a leaf and revive it while a child was still failed, leaving it waiting
forever on the retained sub-orchestration creation event. The two loops touch
disjoint entities, so reordering them is otherwise behavior preserving.

Also drop the static fields from the instance ID reuse test. The success child
now returns its own instance ID so the parent's output carries it, and the
cleanup child's instance ID is confirmed by querying its history directly.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
Copilot AI review requested due to automatic review settings August 31, 2026 15:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:435

  • These replacements are issued one row at a time in chronological order, so a TaskFailed/SubOrchestrationInstanceFailed row can be cleared before later consequence rows. If a later replacement fails transiently, the retry no longer finds that failure, computes an empty (or later) failureEpisode, and can reset the instance while stale consequence events remain, reproducing the non-determinism. Process all non-failure rows first and clear the failure-result rows last (or use atomic partition batches) so a partial attempt remains discoverable and retryable.
            foreach (TableEntity entity in entitiesToClear)
            {
                // "clear" the event by making it a GenericEvent: replay ignores the row while the dummy event preserves the rowKey
                entity[nameof(TaskFailedEvent.Reason)] = "Rewound: " + entity.GetString(nameof(HistoryEvent.EventType));
                entity[nameof(TaskFailedEvent.EventType)] = nameof(EventType.GenericEvent);

                await this.HistoryTable.ReplaceEntityAsync(entity, entity.ETag, cancellationToken);

Comment thread test/DurableTask.AzureStorage.Tests/AzureStorageScenarioTests.cs Fixed
@YunchuWang

Copy link
Copy Markdown
Member Author

CI note: the red DTFxASValidate legs are a pre-existing flake, not a regression

All 13 DTFxASValidate legs report failed, but that is an artifact of the sharded legs sharing a single ADO test run (1767317) — a failure anywhere in the run fails every leg's Run tests task. Only two tests actually failed, and neither is touched by this PR:

Test Occurrences
TestTablePartitionManager.EnsureOwnedQueueExclusive 4 (incl. retries)
AzureStorageScenarioTests.ScheduledStart_Inline (False) 1

Evidence that both are pre-existing flakes:

  • EnsureOwnedQueueExclusive fails in 4 of the last 6 main builds (301158, 301048, 300949, 300615). Those builds are green only because the ADO retry happened to pass.
  • Reproduced locally against Azurite on pristine origin/main (8b6990fd): 5 consecutive runs gave PASS FAIL PASS PASS PASS. It is a timing/lease-sensitive test that flakes independently of this branch.
  • ScheduledStart_Inline passes locally on this branch for both True and False.
  • This PR's production diff is confined to RewindHistoryAsync (AzureTableTrackingStore) and ProcessRewindOrchestrationDecision (TaskOrchestrationDispatcher). Neither is reachable from the table partition manager or from scheduled start.

All 12 Rewind* Azure Storage E2E tests passed in this same CI run, including the ones covering the reordered sub-orchestration path:

Passed  RewindActivityFail
Passed  RewindActivityFailFanOut
Passed  RewindActivityFailWithCleanupActivity
Passed  RewindActivityFailWithCleanupSubOrchestration
Passed  RewindActivityFailWithRetry
Passed  RewindActivityFailWithSendEvent
Passed  RewindActivityFailWithSubOrchestrationIdReuse
Passed  RewindMultipleActivityFail
Passed  RewindNestedSubOrchestrationTest
Passed  RewindOrchestrationsFail
Passed  RewindSubOrchestrationActivityTest
Passed  RewindSubOrchestrationsTest

Every other stage (DTFxCoreValidate x13, DTFxEmulatorValidate x13, CodeQL, SDL, CLA, nuget) is green. Happy to re-run the AS legs if a maintainer wants a clean board.

WaitForCompletionAsync returns null when the orchestration does not complete in
time, and the test dereferenced the result to derive the expected child instance
ID. Assert non-null first so a timeout reports that directly instead of throwing
a NullReferenceException.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9b6bb49d-7dc5-4abf-86f5-d453fcfc28a4
Copilot AI review requested due to automatic review settings August 31, 2026 15:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:435

  • These replacements are non-atomic and remain in chronological order, so the TaskFailed row is cleared before the consequence scheduling/result rows that follow it. If a later replacement fails transiently, a retry can no longer discover that failure (failureEpisode stays int.MaxValue), leaves the consequence rows intact, and revives a history that still fails replay. Clear all consequence rows before clearing the failure markers (or update them atomically) so retries can always reconstruct the scrub set.
            foreach (TableEntity entity in entitiesToClear)
            {
                // "clear" the event by making it a GenericEvent: replay ignores the row while the dummy event preserves the rowKey
                entity[nameof(TaskFailedEvent.Reason)] = "Rewound: " + entity.GetString(nameof(HistoryEvent.EventType));
                entity[nameof(TaskFailedEvent.EventType)] = nameof(EventType.GenericEvent);

                await this.HistoryTable.ReplaceEntityAsync(entity, entity.ETag, cancellationToken);

@YunchuWang

Copy link
Copy Markdown
Member Author

/azp run durabletask.public

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@YunchuWang

Copy link
Copy Markdown
Member Author

/azp run durabletask.public

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@YunchuWang

Copy link
Copy Markdown
Member Author

CI update: the current red board is an Azure Artifacts feed outage, not this PR

My earlier note above covered build 301335 (two pre-existing flaky tests). The board has since gone red for a completely different and much simpler reason: NuGet restore is failing platform-wide. No code is compiled and no test is executed.

Evidence

Every failing job fails at the restore step, and the feed returns HTTP 500 for stock packages:

Failed to download package 'Microsoft.Bcl.AsyncInterfaces.8.0.0' from
  https://pkgs.dev.azure.com/azfunc/.../_packaging/.../flat2/microsoft.bcl.asyncinterfaces/8.0.0/...
Response status code does not indicate success: 500 (Internal Server Error - Forbidden ...)

Same for Microsoft.SourceLink.GitHub, Microsoft.Build.Tasks.Git, System.Reactive.Compatibility, System.Diagnostics.DiagnosticSource, System.Runtime.InteropServices.WindowsRuntime, and others.

Build Branch Jobs failed at restore
301344 refs/pull/1391/merge 39
301371 refs/pull/1391/merge (after /azp run) 35
301226 / 301116 / 301002 refs/heads/durabletask-core-v2 27 each

The GitHub Actions CodeQL / Analyze (csharp) jobs fail at the same step — its failing step is literally Restore dependencies, and re-running it reproduced the failure.

Why this cannot be caused by this PR

  • This PR adds zero package references; the diff is 2 production files and 2 test files.
  • The immediately preceding commit on this same branch (14c1633c, build 301335) restored fine and ran all 12 Rewind* Azure Storage E2E tests green.
  • The only delta between 14c1633c and f5e4ef4f is adding one Assert.IsNotNull(statusFail, ...) line to a single test.
  • Another branch (durabletask-core-v2) hits the identical restore failure.

Local verification of the final commit f5e4ef4f (Azurite, net8.0):

DurableTask.AzureStorage.Tests  -> Passed: 12, Failed: 0   (all Rewind* E2E)
DurableTask.Core.Tests          -> Passed:  8, Failed: 0   (RewindTests)

I retried the pipeline twice via /azp run; the feed is still returning 500. This needs a re-run once the Azure Artifacts feed recovers — no change to this branch will help.

@YunchuWang

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RewindAsync always returns "Non-Deterministic workflow detected"

2 participants