Skip to content

feat: write a run manifest for CI consumers - #414

Open
FRSgit wants to merge 3 commits into
feat/major-5from
feat/manifest
Open

FRSgit wants to merge 3 commits into
feat/major-5from
feat/manifest

Conversation

@FRSgit

@FRSgit FRSgit commented Sep 21, 2026

Copy link
Copy Markdown
Member

Every matchImage comparison now ends up in a JSON manifest, by default at <screenshotsFolder>/cp-visual-regression-diff-manifest.<testingType>.json. The idea is that whatever runs after Cypress in CI (a PR comment bot, a review dashboard, an approval tool) can read one file instead of parsing logs or diffing the snapshots folder.

Each entry has:

  • the test it came from (spec file, title path, retry number)
  • status: passed, failed, missing-baseline, created, updated or approved
  • diff ratio and threshold
  • project-relative paths (and sizes) of the baseline, .actual.png and .diff.png, null when the file isn't on disk
  • baselineWritten, so you know whether the working tree changed regardless of status
  • platform (os, arch, browser incl. family/headless), viewport, and the resolved matchImage options it ran with (imagesPath keeps its tokens unexpanded), plus a recordedAt timestamp

And the file itself now says where and how the run happened, which is what a bot needs to post on the right PR and grab the right artifact:

  • createdAt / updatedAt
  • platform: os, arch, os version of the machine
  • ci: auto-detected on GitHub Actions and GitLab CI - repository, sha, ref, branch, pull request number (with head sha/ref, since GITHUB_SHA is the merge commit on PR events), run id/attempt, job, workflow, run URL, and the checkout dir (workspace, handy in monorepos where projectRoot isn't the repo root). Other CI systems get { provider: null }, local runs null. Only those allow-listed variables are ever copied.
  • options: every pluginVisualRegression* option from expose/env, prefix stripped, values as configured
  • runner: cypress version, testing type, run/open mode, config file, the launched browser, spec list (run mode only), spec pattern, base url, viewport, retries, Cypress Cloud run url/group/tag when recording. This is the only Cypress-specific block; everything else is meant to be shared with a future Playwright client.

A few details worth knowing:

  • The file is rewritten after every comparison (write to .tmp, then rename), so it's complete even if the run gets killed halfway.
  • before:run wipes it and re-seeds the run data, and re-running a spec in cypress open drops that spec's entries first, so you don't get leftovers from old counters. In cypress open there's no before:run, so the run data comes from initPlugin and before:browser:launch (which is now one handler together with the device-scale-factor flags, since Cypress only calls one per event).
  • A retry purges the entries of earlier attempts of the same test.
  • "Replace image" in the headed review UI flips the entry to approved.
  • pluginVisualRegressionManifestPath moves the file, false turns it off.
  • version is still 1. Nothing has shipped, so the run metadata went in without a bump.
  • The Manifest* types are exported from ./plugins.
  • CI now uploads the example manifest next to the PNGs in the test artifact, so we can dogfood tooling on this repo's own PRs.

Docs are in the README (new "Run manifest" section with a field table and consumer notes, including how to reproduce a run from runner + options) and MIGRATION.md.

Related: #416 removes stale .diff.png files when a comparison passes. Without it a passed entry can still point at a diff image left by a previous failed run, which contradicts the status table in the README. Best to land that one first or together.

馃 Generated with Claude Code

Every `matchImage` comparison is now recorded in a JSON manifest, by
default `<screenshotsFolder>/cp-visual-regression-diff-manifest.<testingType>.json`,
so tooling that runs after Cypress (PR comments, review dashboards,
approval bots) can find out what passed, failed, created or updated its
baseline without parsing logs or diffing the snapshots folder.

Each entry carries the originating test (file, title path, retry), the
status, diff ratio and threshold, project-relative paths and sizes of the
baseline, `.actual.png` and `.diff.png` files, whether the baseline was
written, and browser and viewport. Approving a screenshot from the headed
review UI flips its entry to `approved`.

The file is rewritten (write-then-rename) after every comparison, so it is
complete even when the run is aborted; `before:run` starts from scratch,
and re-running a spec in `cypress open` drops that spec's entries first.
Retries purge the entries left by earlier attempts of the same test.

The format is runner-agnostic: only the `runner` block is Cypress-specific.
`pluginVisualRegressionManifestPath` moves the file or, with `false`,
disables it. The `Manifest*` types are exported from `./plugins`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
FRSgit and others added 2 commits September 22, 2026 02:04
Resolve the two task.hook.ts hunks by keeping both sides: the manifest
status bookkeeping and the stale .diff.png cleanup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The manifest now carries what a CI consumer needs to find the run and to
reproduce it, not only the comparison results:

- run-level `createdAt`/`updatedAt`, `platform` (os, arch, os version),
  `ci` detected from GitHub Actions / GitLab CI env vars (repository, sha,
  ref, pull request incl. head sha, run id/attempt, workspace), the global
  `pluginVisualRegression*` options with the prefix stripped, and a richer
  `runner` block (mode, config file, launched browser, specs, spec pattern,
  base url, viewport, retries, Cypress Cloud fields)
- per-entry `recordedAt`, `platform` (os, arch, browser; replaces the
  top-level `browser`) and the resolved `matchImage` `options`

Run data is seeded at `initPlugin`, the launched browser is recorded from
`before:browser:launch` (now a single handler shared with the device scale
factor flags) and `before:run` merges its details. The format version stays
1 because nothing shipped yet. CI uploads the example manifest with the
screenshot artifact.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant