chore: Use CircleCI native timing-based test splitting for acceptance shards

[danskmt]

Pull Request Submission Checklist

• Follows CONTRIBUTING guidelines
• Commit messages are release-note ready, emphasizing what was changed, not how.
• Includes detailed description of changes
• Contains risk assessment (Low | Medium | High)
• Highlights breaking API changes (if applicable)
• Links to automated tests covering new functionality
• Includes manual testing instructions (if necessary)
• Updates relevant GitBook documentation (PR link: ___)
• Includes product update to be announced in the next stable release notes

What does this PR do?

Replaces Jest's --shard with CircleCI's native circleci tests run --split-by=timings so acceptance test shards are automatically balanced by historic runtime data — no manually maintained weight files needed.

• Replaces --shard=<index>/<total> with circleci tests glob | circleci tests run --split-by=timings --timings-type=file in the acceptance-tests job.
• Adds addFileAttribute: true to jest-junit config so JUnit XML includes file="..." on each <testsuite>, which CircleCI needs for file-based timing matching.
• Switches the Windows test step to bash.exe (from PowerShell) since the command uses xargs and Unix pipe syntax. Sources the bash-compatible env script (snyk-env.sh) instead of the PowerShell one.

Where should the reviewer start?

• .circleci/config.yml — the acceptance-tests job definition (replaced shard_calc_cmd parameter with test_shell, new circleci tests run command)
• package.json — addFileAttribute addition in jest-junit config

How should this be manually tested?

1. Push branch and trigger CI.
2. Check that acceptance tests run on all platforms (linux, macOS, Windows).
3. Compare shard wall times — after 1-2 runs, CircleCI accumulates timing data and balances shards automatically.

What's the product update that needs to be communicated to CLI users?

None. CI-only change; no CLI behavior change for end users.

Risk assessment (Low | Medium | High)?

Medium — changes how tests are distributed across CI shards on all platforms including Windows (shell change). First run may use name-based splitting until CircleCI accumulates timing data from JUnit reports.

Any background context you want to provide?

CircleCI stores timing data per job name across all branches, so existing store_test_results uploads from main should provide immediate timing data. The addFileAttribute flag makes jest-junit emit the file attribute on each <testsuite>, enabling CircleCI to match timing history to spec files. See CircleCI docs: Test splitting and parallelism.

What are the relevant tickets?

CLI-1482

Screenshots (if appropriate)

Before (main) X Now (chore/CLI-1482-circleci-native-test-splitting):

Before (feat/CLI-1482-balance-acceptance-test-shards) X Now (chore/CLI-1482-circleci-native-test-splitting)

[snyk-io]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[robertolopezlopez]

LGTM! Good job

[danskmt]

We need this addFileAttribute: true to jest-junit config so JUnit XML includes file="..." on each , which CircleCI needs for file-based timing matching.

[j-luong]

I think introducing bash as the shell for Windows causes more problems than it solves, let's see if we can do this without needing bash for Windows. That way we can clean up the extra scripts you introduced

[danskmt]

@j-luong it seems we do need the path thing... Needs both for saving as forward slash, and for normalizing when fetching
https://app.circleci.com/pipelines/gh/snyk/cli/37803/workflows/6ad62f25-703d-42a3-aad8-8401d73bb71d/jobs/595269

[j-luong]

nitpick: Would be nice to have this written in bash/PowerShell to remove the JavaScript dependency

[robertolopezlopez]

My vote for python, much easier to craft+read+review.
And multi platform

issue: the windows acceptance tests seem to be taking longer than before (> 30mins) which means this PR actually degrades CI runtimes. Let's look into what's going on here

[graphite-app]

Missing error handling for file operations could fail the entire CI job even when tests pass. If fs.readFileSync() or fs.writeFileSync() throws an error (e.g., due to permissions or file locks), the unhandled exception will cause the normalization step to fail. Since this step runs with when: always in CircleCI, it will execute even after test failures, but any crash here will override the test results and fail the job.

Add try-catch error handling:

for (const entry of fs.readdirSync(reportsDir)) {
  if (!entry.endsWith('.xml')) {
    continue;
  }

  const filePath = path.join(reportsDir, entry);
  try {
    const original = fs.readFileSync(filePath, 'utf8');
    const normalized = original.replace(fileAttrPattern, (match) =>
      match.split('\\').join('/'),
    );

    if (normalized !== original) {
      fs.writeFileSync(filePath, normalized);
    }
  } catch (err) {
    console.error(`Failed to process ${filePath}:`, err.message);
    // Continue processing other files
  }
}

Suggested change

	for (const entry of fs.readdirSync(reportsDir)) {
	if (!entry.endsWith('.xml')) {
	continue;
	}
	const filePath = path.join(reportsDir, entry);
	const original = fs.readFileSync(filePath, 'utf8');
	const normalized = original.replace(fileAttrPattern, (match) =>
	match.split('\\').join('/'),
	);
	if (normalized !== original) {
	fs.writeFileSync(filePath, normalized);
	}
	for (const entry of fs.readdirSync(reportsDir)) {
	if (!entry.endsWith('.xml')) {
	continue;
	}
	const filePath = path.join(reportsDir, entry);
	try {
	const original = fs.readFileSync(filePath, 'utf8');
	const normalized = original.replace(fileAttrPattern, (match) =>
	match.split('\\').join('/'),
	);
	if (normalized !== original) {
	fs.writeFileSync(filePath, normalized);
	}
	} catch (err) {
	console.error(`Failed to process ${filePath}:`, err.message);
	// Continue processing other files
	}

Spotted by Graphite



Is this helpful? React 👍 or 👎 to let us know.

[snyk-pr-review-bot]

PR Reviewer Guide 🔍

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Environment Incompatibility 🟠 [major] The new test_cmd for Windows acceptance tests uses a PowerShell pipe (|) to send file paths into circleci tests run, which then executes xargs. While Get-ChildItem is a PowerShell cmdlet, xargs is a Unix utility not natively available in PowerShell. If the win-server2022-amd64 executor does not have a Unix-like environment (like Git Bash) in its system PATH, this command will fail immediately. Additionally, the PR description mentions switching to bash.exe, but the code in the diff uses PowerShell syntax (Get-ChildItem, ForEach-Object, -replace). test_cmd: | Get-ChildItem -Recurse -Path test/jest/acceptance,ts-binary-wrapper/test/acceptance -Filter *.spec.ts | ForEach-Object { (Resolve-Path -Relative $_.FullName) -replace '\\', '/' -replace '^\./', '' } | circleci tests run --command "xargs npx jest --maxWorkers=1 --selectProjects coreCli --reporters=jest-junit --reporters=default --" --split-by=timings --timings-type=file Fragile Regex 🟡 [minor] The fileAttrPattern regex /file="[^"]*"/g is used to find and replace paths in the JUnit XML. If a test failure message or a system-out block contains the string file="some\path", it will be incorrectly modified by this script. While acceptance tests typically don't output this specific pattern in logs, a more robust approach would be to use a proper XML parser to target only the file attribute on <testsuite> or <testcase> elements. const fileAttrPattern = /file="[^"]*"/g; for (const entry of fs.readdirSync(reportsDir)) { if (!entry.endsWith('.xml')) { continue; } const filePath = path.join(reportsDir, entry); const original = fs.readFileSync(filePath, 'utf8'); const normalized = original.replace(fileAttrPattern, (match) => match.split('\\').join('/'),

📚 Repository Context Analyzed This review considered 7 relevant code sections from 6 files (average relevance: 0.81) dangerfile.js (lines 1-111) CONTRIBUTING.md (2 segments, lines 1-231) scripts/normalize-junit-file-paths.js (lines 1-46) package-lock.json (lines 6-266) packages/snyk-fix/jest.config.js (lines 1-4) package.json (lines 1-229)

🤖 Repository instructions applied (from AGENTS.md)

[j-luong]

The new test_cmd for Windows acceptance tests uses a PowerShell pipe (|) to send file paths into circleci tests run, which then executes xargs. While Get-ChildItem is a PowerShell cmdlet, xargs is a Unix utility not natively available in PowerShell. If the win-server2022-amd64 executor does not have a Unix-like environment (like Git Bash) in its system PATH, this command will fail immediately. Additionally, the PR description mentions switching to bash.exe, but the code in the diff uses PowerShell syntax (Get-ChildItem, ForEach-Object, -replace).

this seems a valid concern, and I'm not sure why it's not failing, but it's worth fixing