Conversation
Bring together the best aspects of the swiftlang GitHub Actions workflows in this repository and the SwiftNIO ones at apple/swift-nio, through a new reusable workflow that sits beside the existing one and a hierarchy of supporting infrastructure. Key design features: 1) GitHub-release-based versioning 2) No skipped jobs 3) Scripts are cloned not curled 4) Custom matrix builds 5) Minimum Swift version detection `package_test.yml` runs a range of common test and build configurations on a variety of platforms against multiple Swift versions. It sits atop a matrix generation layer which takes inputs and produces a canonical work definition in YAML or JSON; that definition is then expanded into one job per entry, each executing one slice of the work. Benchmarking is a further consumer of the same layer rather than part of the recommended test workflow. The design is layered, so an adopter may use the whole stack or customize portions beyond the level the workflow allows: * Use `package_test.yml` for the full suite of conveniences. * Use a custom workflow which calls `toolchain_matrix.yml` for the matrix and `execute_matrix.yml` to run its own command across it. * Generate the work definition by some other means, or hard-code it into a workflow, and pass it to `execute_matrix.yml`. The matrix generation layer is a Swift script, `generate-matrix.swift`, run by the matrix job on `ubuntu-latest`, which comes with the Swift, jq and yq it needs. It reads the workflow's inputs from the environment, applies the defaults and the minimum version filter, and writes the work definition to standard output. In SwiftNIO it was a bash script, which executes faster, but the unified version is much more complex, so Swift seemed like the more maintainable option. A test package under `tests/MatrixGeneratorValidator` covers the inputs, their defaults and failure modes. Execution times are reduced by defaulting to running natively on the runner rather than inside a container. The toolchain is installed with swiftly, which was previously used only for macOS toolchains, so using it on Linux is new to both upstreams. Containers remain available for a package that needs a particular distribution or system dependencies, through `linux_use_docker` or by naming a distribution in `linux_os`. The existing workflows are untouched, so a repository migrates when it suits it. The most visible change on migrating is that the `*_exclude_swift_versions` inputs are gone, replaced by explicit version lists and minimum-version detection which drops anything the package manifest cannot support. A repository that was only excluding old versions can delete those inputs entirely. Migrating from the SwiftNIO workflows is a change in spelling rather than a reduction in flexibility: per-version inputs become one map, which removes the churn of adding an input per Swift release. Functionality new to this repository, some inspired by SwiftNIO and some new to both: * Simulator testing. iOS was build-only; xcodebuild now builds and tests on simulators, with tvOS, watchOS and visionOS alongside it. Mac Catalyst is new to both upstreams. * Benchmarks across the matrix. `benchmarks.yml` checks committed thresholds under `Thresholds/<swift-version>/` on every platform and Swift version, failing the job and printing a diff when one moves. This differs from the existing `performance_test.yml`, which compares a pull request against its merge base in one container and posts the table as a comment: that reports what a change did to performance, this holds a package to a standard it has written down. * A C++ interoperability check. A release-configuration build needs no dedicated input: it is a second labeled command, so `Release-Build Linux Swift 6.3` runs alongside `Debug-Test Linux Swift 6.3` from one call. * A semantic version pull request label check, `pull_request_label.yml`. * An importable Swift version matrix, `toolchain_matrix.yml`, which gives a custom workflow the current supported matrix as plain YAML, so it can be filtered or extended with yq before it is executed. * Per-version overrides, so one Swift version can take extra arguments or a different command without splitting the matrix by hand. Known limitation: a pinned tag still does not pin the scripts. This is inherited behavior. `swift_package_test.yml` and `soundness.yml` check this repository out with no `ref:`, so they resolve to its default branch, and one variant hardcodes `ref: main`. A caller on `@0.0.14` has therefore always run that tag's YAML against the default branch's scripts, and the unified workflows inherit that default. Resolving it is possible but would require stamping versions as part of the release process.
Member
|
Overall this looks good. Thanks for working on it. I left some comments inline. One thing I would like to see added is the definition of the JSON/YML used by the matrix job so if somebody wants to hand-roll a matrix they have a reference to write it against.
We should create an issue to track this since we ran into problems with this in the past. |
FranzBusch
reviewed
Sep 14, 2026
| @@ -0,0 +1,47 @@ | |||
| # Swift Package Test Workflow Architecture | |||
|
|
|||
| ## Aims | |||
Member
There was a problem hiding this comment.
Nit:
Suggested change
| ## Aims | |
| ## Goals |
|
|
||
| The Swift Package Test workflow is intended to give maintainers of open-source Swift packages a low-friction way to give contributors confidence in the code they write. This means making testing against a range of Swift versions and platforms as simple as possible, and making the results as accessible as possible. | ||
|
|
||
| This workflow builds on the GitHub Actions workflows from [apple/swift-nio](https://github.com/apple/swift-nio/tree/main/.github/workflows) and [swiftlang/github-workflows](https://github.com/swiftlang/github-workflows/tree/main/.github/workflows), taking from both repositories the key features listed below, which the unified architecture treats as requirements. |
Member
There was a problem hiding this comment.
I would drop this since it won't stand the test of time once NIO removes its workflows.
Comment on lines
+37
to
+41
| windows_job_timeout: | ||
| type: number | ||
| description: "Timeout in minutes for Windows jobs. 0 uses job_timeout. Windows installs a toolchain and Visual Studio Build Tools before building, so it often needs longer." | ||
| default: 0 | ||
| freebsd_job_timeout: |
Member
There was a problem hiding this comment.
Why do we need different timeouts per platform? If we do should this be configured in the matrix instead?
| exit 1 | ||
| fi | ||
|
|
||
| # Reject unexpected platforms early. |
Member
There was a problem hiding this comment.
Suggested change
| # Reject unexpected platforms early. | |
| # Reject unexpected host platforms early. |
| path: github-workflows | ||
| persist-credentials: false | ||
|
|
||
| - name: Check out linked PRs (macOS) |
Member
There was a problem hiding this comment.
Why does this step only run on macOS and the others don't need it? Can we at least add a comment on the why?
Member
There was a problem hiding this comment.
Could we separate the entire Semver check out into a separate PR? It is not tied to the matrix based builds at all.
Comment on lines
+9
to
+13
| benchmark_package_path: | ||
| type: string | ||
| description: "Path to the directory containing the benchmarking package. Used only when benchmark_package_paths is empty." | ||
| default: "." | ||
| benchmark_package_paths: |
Member
There was a problem hiding this comment.
Do we need both or would just the plural be enough?
Comment on lines
+28
to
+30
| static let releases = ["6.1", "6.2", "6.3"] | ||
| /// The nightlies every platform runs: the next release's branch, and main. | ||
| static let nightlies = ["nightly-release", "nightly-main"] |
Member
There was a problem hiding this comment.
So there are defaults in the yml files and here. Do we need to update both? Do we need both to default? It would be great if we had just 1 place to update.
Member
There was a problem hiding this comment.
This file would benefit from splitting up into multiple files.
Member
There was a problem hiding this comment.
This might be better added in a second PR since it is a net-new capability for this repo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Swift GitHub Actions Workflows Unification.
This PR aims to bring the best aspects of the swiftlang GitHub Actions workflows (this repository) and the SwiftNIO (apple/swift-nio) workflows together. It does this chiefly through a new re-usable workflow which sits beside the existing workflow, and a hierarchy of supporting infrastructure.
Some key design features:
The unified design provides one workflow which can be adopted to run a range of common test and build configurations on a variety of platforms against multiple Swift versions. This workflow sits atop a "matrix generation" layer which takes inputs and produces a canonical work definition (in YAML or JSON). That definition is then expanded into one job per entry, each executing one slice of the work. Benchmarking is a further consumer of the same layer rather than part of the recommended test workflow.
The design is intended to be layered, so that adopters may use the whole stack, or customize portions beyond the level which the workflow allows. Adopters may:
package_test.ymlworkflow for the full suite of conveniencestoolchain_matrix.ymlfor the matrix andexecute_matrix.ymlto run its own command across itexecute_matrix.ymlThe matrix generation layer is a Swift script,
generate-matrix.swift, run by the matrix job onubuntu-latest, which comes with the Swift, jq and yq it needs. It reads the workflow's inputs from the environment, applies the defaults and the minimum version filter, and writes the work definition to standard output. In SwiftNIO it was a bash script, which executes faster, but the unified version is much more complex, so Swift seemed like the more maintainable option. A test package undertests/MatrixGeneratorValidatorcovers the inputs, their defaults and failure modes.As part of this work an effort was also made to reduce execution times, defaulting to running natively on the runner rather than inside a container where possible. The unified workflows install the toolchain with swiftly on the runner rather than pulling a full Docker image. Containers remain available for a package that needs a particular distribution or system dependencies, either through
linux_use_dockeror by naming a distribution inlinux_os. swiftly was previously used only for macOS toolchains, so using it on Linux is new to both upstreams.Migrating from this repository's existing workflow
The new workflow is currently available alongside existing infrastructure so a repository can control when they perform the migration.
When adopting the new workflow the most visible change is that the
*_exclude_swift_versionsinputs are gone, replaced with explicit supported Swift version lists and minimum-version detection which drops anything the package manifest cannot support. A repository that was only excluding old versions based on its Swift tools version can simplify its config.Before:
After:
The new architecture means that jobs which are not intended to be run will no longer appear as skipped:
Before:
After:
Migrating from the SwiftNIO workflows
Changing from the SwiftNIO workflows means a change in spelling but no reduction in flexibility. Per-version inputs are now supplied in a map which reduces churn and synchronization problems when a new Swift version is released.
Before:
After:
Functionality new to this repository
The unification also brings some new functionality to this repository. Some inspired by SwiftNIO, some entirely new:
xcodebuildnow builds and tests on simulators, and tvOS, watchOS and visionOS join it. Mac Catalyst is new to both upstreams.benchmarks.ymlmeasures performance with Swift Package Benchmark, checking committed thresholds underThresholds/<swift-version>/on every platform and Swift version in the matrix, failing the job and printing a diff when one moves. This differs from the existingperformance_test.yml, which compares a PR against its merge base in one container and posts the table as a PR comment. The existing workflow tells you what a change did to performance; the new one holds a package to a standard it has written down.Release-Build Linux Swift 6.3runs alongsideDebug-Test Linux Swift 6.3from one call.pull_request_label.yml), which fails a PR that carries no SemVer label.toolchain_matrix.yml, provides the current supported Swift version matrix for custom workflows to utilize. The output is plain YAML, so it can be filtered or extended withyqbefore it is executed.Known limitations
swift_package_test.ymlandsoundness.ymlcheck this repository out with noref:, so they resolve to its default branch, and one variant hardcodesref: main; this means a caller on a tag e.g.@0.0.14has always run that tag's YAML against the default branch's scripts. The unified workflows inherit that default. Resolving this should be possible but would require stamping versions as part of the release process.