Skip to content

feat: Retry data sources on terminal errors instead of stopping - #533

Merged
beekld merged 12 commits into
v11from
bklimt/SDK-2802/retry-conformance
Sep 30, 2026
Merged

beekld merged 12 commits into
v11from
bklimt/SDK-2802/retry-conformance

Conversation

@beekld

@beekld beekld commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Previously, the streaming and polling data sources stopped permanently on failures classified as terminal. They now retry those failures, after a longer backoff than other errors.

Implementation notes:

  • Streaming: The SDK now manages reconnection itself. The underlying EventSource is told never to reconnect on its own, and each reconnect is scheduled with an LDTimer. LDTimer was reused so timer behavior around sleep does not change.
  • Polling: LDTimer was extended to support one-shot (non-repeating) timers, so the poll delay can be adjusted per attempt based on the error response.

Notes for reviewers:

  • On a persistent unrecoverable error, the client stays uninitialized and keeps retrying. As a result, the deprecated no-timeout start(completion:) and identify(completion:) can block forever. The timeout-based variants still return.
  • setOnline(true) reconnects immediately but does not reset the backoff. Only identify and a background/foreground transition start a fresh data source and reset the backoff.
  • On a terminal failure, the connection mode is now reported as establishing (streaming) or polling instead of offline. A dedicated interrupted mode was not added because a new ConnectionMode case would be source-breaking for FDv1. FDv2 will add interrupted.
  • TLS and certificate failures are still treated as recoverable.
  • The event processor is unchanged. The RETRY spec currently binds only data sources, so it has no conformance requirement yet, and will be revisited when the spec adds one. The diagnostic reporter already conforms and is unchanged.

Contract tests declare the retry-conformance-fdv1-streaming and retry-conformance-fdv1-polling capabilities. Long-running client-side retry tests are not implemented in the harness yet, and there is no client-side polling retry suite, so the polling capability is forward-looking.


Note

Overview
Streaming and polling flag data sources now keep retrying after “terminal” sync failures instead of shutting down the synchronizer, marking the client initialized, or reporting connection mode as offline.

RetryState adds jittered exponential backoff for streaming reconnects and per-poll delays, with a longer “extended” regime after unexpected (e.g. 4xx) failures. FlagSynchronizer owns reconnection: stream errors return .shutdown to EventSource, then tear down and schedule reconnect via LDTimer; polling reschedules one-shot polls through pollDidComplete. A ~60s healthy stream resets streaming backoff.

LDClient no longer calls flagSynchronizer.isOnline = false on terminal errors. ConnectionInformation records failures but leaves mode as establishing streaming or polling on terminal errors.

LDSwiftEventSource is bumped to 3.4.0 (pod, SPM, Xcode). Contract tests advertise retry-conformance-fdv1-streaming and retry-conformance-fdv1-polling.

Reviewed by Cursor Bugbot for commit 147d849. Bugbot is set up for automated code reviews on this repo. Configure here.

@beekld
beekld marked this pull request as ready for review September 24, 2026 17:12
@beekld
beekld requested a review from a team as a code owner September 24, 2026 17:12

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread LaunchDarkly/LaunchDarkly/ServiceObjects/FlagSynchronizer.swift

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4d62600. Configure here.


public func onClosed() {
os_log("%s EventSource closed", log: service.config.logger, type: .debug, typeName(and: #function))
connectedAt = nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Close wipes healthy-stream backoff marker

Medium Severity

Clearing connectedAt in onClosed can run before eventSourceErrorHandler when an open stream fails. The error handler then sees a nil marker and skips the healthy-stream reset, so a connection that stayed up long enough still keeps the previous backoff.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 4d62600. Configure here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is actually a bug. swift-eventsource calls our error handler first and onClosed() only after it returns (LDSwiftEventSource.swift:282-285, both on one serial queue), so the healthy check always reads connectedAt before it's cleared. And the clear is needed anyway, to stop a stale marker leaking across an offline/online cycle.

Comment thread LaunchDarkly/LaunchDarkly/ServiceObjects/RetryState.swift
Comment thread LaunchDarkly/LaunchDarkly/ServiceObjects/FlagSynchronizer.swift
Comment thread LaunchDarkly/LaunchDarkly/ServiceObjects/FlagSynchronizer.swift
@jsonbailey

Copy link
Copy Markdown

There is a possible gap in iOS: When the server closes the stream cleanly (didCompleteWithError with error == nil, in LDSwiftEventSource.swift), it just logs "Connection unexpectedly closed", calls onClosed(), and reconnects on its own 1s-to-30s backoff. So on iOS a server close skips the SDK's retry state, and it does not keep the extended backoff after a 401. Ruby needed StreamClosedByServerError for the same case.

@tanderson-ld

Copy link
Copy Markdown
Contributor

Informal approval, but @jsonbailey left a comment that I don't want to approve and risk accidental merge.

@beekld
beekld requested a review from tanderson-ld September 29, 2026 17:45
@beekld

beekld commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Sorry, I rerequested review on this before seeing the latest comments. Looking into that now.

@beekld

beekld commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

There is a possible gap in iOS: When the server closes the stream cleanly (didCompleteWithError with error == nil, in LDSwiftEventSource.swift), it just logs "Connection unexpectedly closed", calls onClosed(), and reconnects on its own 1s-to-30s backoff. So on iOS a server close skips the SDK's retry state, and it does not keep the extended backoff after a 401. Ruby needed StreamClosedByServerError for the same case.

I opened swift-eventsource#118 to address this. I think it's cleaner to address this there (similar to ruby), rather than doing it in this repo.

@beekld

beekld commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

There is a possible gap in iOS: When the server closes the stream cleanly (didCompleteWithError with error == nil, in LDSwiftEventSource.swift), it just logs "Connection unexpectedly closed", calls onClosed(), and reconnects on its own 1s-to-30s backoff. So on iOS a server close skips the SDK's retry state, and it does not keep the extended backoff after a 401. Ruby needed StreamClosedByServerError for the same case.

I opened swift-eventsource#118 to address this. I think it's cleaner to address this there (similar to ruby), rather than doing it in this repo.

Fixed

@beekld
beekld merged commit fd2f397 into v11 Sep 30, 2026
18 checks passed
@beekld
beekld deleted the bklimt/SDK-2802/retry-conformance branch September 30, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants