docs(contributing): update screenshot commands and Docker usage (#31441)

Updated the contributing guide to replace the outdated commands (`npm
run test.screenshot`) with the current commands: `npm run
test.e2e.docker.update-snapshots` and `npm run test.e2e.docker`.

Also moved most of the explanation into the proper testing files and
link to it instead of having it in the main contributing guide.
Restructures the steps for running tests and removes old Docker
instructions.

---------

Co-authored-by: Brandy Smith <6577830+brandyscarney@users.noreply.github.com>

Author: brandyscarney

docs/CONTRIBUTING.md

@@ -86,7 +86,7 @@ Without a reliable code reproduction, it is unlikely we will be able to resolve
 
 To contribute on Windows, do the following:
 
-- Configure VS Code to read/save files using line breaks (LF) instead of carriage returns (CRLF). Set it globally by navigating to: Settings -> Text Editor -> Files -> Eol. Set to `\n`.
+- Configure VS Code to read/save files using line breaks (LF) instead of carriage returns (CRLF). Set it globally by navigating to: Settings → Text Editor → Files → Eol. Set to `\n`.
 
   - You can optionally use the following settings in your `.vscode/settings.json`:
     ```json
@@ -290,20 +290,29 @@ npm install file:/~/ionic-vue-router-7.0.1.tgz
 
 1. Locate the test to modify inside the `test/` folder in the component's directory.
 2. If a test exists, modify the test by adding an example to reproduce the problem fixed or feature added.
-3. If a new test is needed, the easiest way is to copy the `basic/` directory from the component's `test/` directory, rename it, and edit the content in both the `index.html` and `e2e.ts` file (see [Screenshot Tests](#screenshot-tests) for more information on this file).
+3. If a new test is needed, the easiest way is to copy the `basic/` directory from the component's `test/` directory, rename it, and edit the content in both the `index.html` and `*.e2e.ts` file (see [Screenshot Tests](#screenshot-tests) for more information on this file).
 4. The `preview/` directory is used in the documentation as a demo. Only update this test if there is a bug in the test or if the API has a change that hasn't been updated in the test.
 
-Refer to [Ionic's E2E testing guide](/core/src/utils/test/playwright/docs/README.md) for information regarding the tools you can use to test Ionic.
+Refer to [Ionic's E2E testing guide](/docs/core/testing/README.md) for information regarding the tools you can use to test Ionic.
 
 ##### Screenshot Tests
 
-1. If the test exists in screenshot, there will be a file named `e2e.ts` in the directory of the test.
-2. A screenshot test can be added by including this file and adding one or more `test()` calls that include a call to `page.compareScreenshot()`. See [Stencil end-to-end testing](https://stenciljs.com/docs/end-to-end-testing) and existing tests in `core/` for examples.
-3. **Important:** each `test()` should have only one screenshot (`page.compareScreenshot()`) call **or** it should check the expect at the end of each test. If there is a mismatch it will fail the test which will prevent the rest of the test from running, i.e. if the first screenshot fails the remaining screenshot calls would not be called _unless_ they are in a separate test or all of the expects are called at the end.
-4. To run screenshot locally, use the following command: `npm run test.screenshot`.
-    - To run screenshot for a specific test, pass the path to the test or a string to search for.
-    - For example, running all `alert` tests: `npm run test.screenshot alert`.
-    - Or, running the basic `alert` tests: `npm run test.screenshot src/components/alert/test/basic/e2e.ts`.
+Screenshot tests live in the same `*.e2e.ts` files as a component's other E2E tests and assert with `toHaveScreenshot()`. They compare against ground truth screenshots that are generated in Docker, so both generating and running them use the Docker commands from the `core` directory:
+
+```shell
+# Generate or update the ground truths for a component
+npm run test.e2e.docker.update-snapshots src/components/alert/
+
+# Run the tests against the committed ground truths
+npm run test.e2e.docker src/components/alert
+```
+
+To learn more:
+
+- [Managing Screenshots](/docs/core/testing/usage-instructions.md#managing-screenshots) covers why Docker is required, which screenshots are committed, and how Ionic team members update ground truths on CI.
+- [Best Practices](/docs/core/testing/best-practices.md) covers the conventions screenshot tests follow, including using one screenshot assertion per test.
+- [Playwright Test Utils](/docs/core/testing/api.md) documents `configs`, `screenshot`, and the other helpers.
+- [Playwright Visual Comparisons](https://playwright.dev/docs/test-snapshots) documents Playwright's screenshot comparison APIs, including `toHaveScreenshot()` and its options.
 
 
 #### Building Changes

docs/core/testing/best-practices.md

@@ -14,6 +14,7 @@ This guide details best practices that should be followed when writing E2E tests
 - [Break up large or slow-running tests across multiple files](#practice-slow-tests)
 - [Use standard viewport sizes](#practice-viewport)
 - [Avoid using screenshots as a way of verifying functionality](#practice-screenshot-functionality)
+- [Use one screenshot assertion per test](#practice-one-screenshot)
 - [Avoid tests that compare computed values](#practice-test-computed)
 - [Test for positive and negative cases](#practice-positive-negative)
 - [Start your test with the configuration or layout in place if possible](#practice-test-config)
@@ -247,6 +248,51 @@ configs().forEach(({ config, title }) => {
 });
 ```
 
+<h2 id="practice-one-screenshot">Use one screenshot assertion per test</h2>
+
+A failed `toHaveScreenshot()` assertion ends the test, so anything after it never runs. When a test takes several screenshots, only the first mismatch is reported and the remaining screenshots are never compared. An intentional visual change then takes several runs of the suite to fully verify.
+
+Give each screenshot its own `test()`. If the screenshots must share setup, take them all and assert at the end of the test.
+
+❌ Incorrect
+
+A mismatch on `button-solid` means `button-outline` is never compared.
+
+```typescript
+configs().forEach(({ config, screenshot, title }) => {
+  test.describe(title('button: fill'), () => {
+    test('should not have visual regressions', async ({ page }) => {
+      await page.goto('/src/components/button/test/fill', config);
+
+      await expect(page.locator('#solid')).toHaveScreenshot(screenshot('button-solid'));
+      await expect(page.locator('#outline')).toHaveScreenshot(screenshot('button-outline'));
+    });
+  });
+});
+```
+
+✅ Correct
+
+Each screenshot is compared independently, and a failure names the state that changed.
+
+```typescript
+configs().forEach(({ config, screenshot, title }) => {
+  test.describe(title('button: fill'), () => {
+    test('should not have visual regressions for solid buttons', async ({ page }) => {
+      await page.goto('/src/components/button/test/fill', config);
+
+      await expect(page.locator('#solid')).toHaveScreenshot(screenshot('button-solid'));
+    });
+
+    test('should not have visual regressions for outline buttons', async ({ page }) => {
+      await page.goto('/src/components/button/test/fill', config);
+
+      await expect(page.locator('#outline')).toHaveScreenshot(screenshot('button-outline'));
+    });
+  });
+});
+```
+
 <h2 id="practice-test-computed">Avoid tests that compare computed values</h2>
 
 All browsers render web content in slightly different manners. Instead of testing computed values such as exact pixel values, screenshots are a great way to ensure that elements are being rendered in a consistent manner across browsers.

docs/core/testing/usage-instructions.md

@@ -80,12 +80,14 @@ macOS uses [XQuartz](https://www.xquartz.org) to use XServer on macOS.
 
 1. Install [Homebrew](https://brew.sh) if not already installed. You can run `brew --version` to check if Homebrew is installed.
 2. Install XQuartz: `brew install --cask xquartz`
-3. Open XQuartz, go to `Preferences > Security`, and check "Allow connections from network clients".
+3. Open XQuartz, go to `Settings → Security`, and check "Allow connections from network clients".
 4. Restart your computer.
 5. Start XQuartz from the command line: `xhost +localhost`
-6. Open Docker Desktop and edit settings to give access to `/tmp/.X11-unix` in `Preferences > Resources > File sharing`.
-7. In the `core` directory run `echo host.docker.internal:0 > docker-display.txt`. This information is used to set the `DISPLAY` environment variable which tells Playwright how to render a headed UI from the Docker container.
-8. In the `core` directory run `echo /tmp/.X11-unix:/tmp/.X11-unix > docker-display-volume.txt`. This information is used to make XServer available inside of the Docker container.
+6. In the `core` directory run `echo host.docker.internal:0 > docker-display.txt`. This information is used to set the `DISPLAY` environment variable which tells Playwright how to render a headed UI from the Docker container.
+7. In the `core` directory run `echo /tmp/.X11-unix:/tmp/.X11-unix > docker-display-volume.txt`. This information is used to make XServer available inside of the Docker container.
+
+> [!NOTE]
+> Unlike Docker Desktop, Rancher Desktop needs no file sharing configuration for this. It shares `/private/tmp` by default, which is where `/tmp` points on macOS.
 
 #### Windows
 
@@ -99,44 +101,58 @@ Windows has a native XServer called [WSLg](https://github.com/microsoft/wslg#rea
 
 ## Running Tests
 
-### Running All Test Files
+Tests are run from the `core` directory with `npm run test.e2e.docker`, which runs them inside the Docker environment provided by the Ionic team through [Rancher Desktop](#installing-rancher-desktop). Any test that takes a screenshot must be run this way so that it compares against the ground truths committed to the repository. See [Managing Screenshots](#managing-screenshots) for more information.
 
-All E2E tests can be run using the following command:
+This command builds a Docker image before tests run. It will also re-build the Docker image in the event that a Playwright update was merged into the repo.
 
-```shell
-npm run test.e2e
-```
+Note that the Playwright report will not automatically open in your web browser when tests are complete because the tests were run in Docker. Run `npx playwright show-report` outside of Docker to open the most recent test report.
 
 > [!NOTE]
-> This command is a wrapper for `npx playwright test`. All data passed to `npm run test.e2e` can also be passed to `npx playwright test`.
+> Additional setup is needed to run Playwright tests with headed mode in Docker. See [Configuring Docker for Headed Tests](#configuring-docker-for-headed-tests-optional) for more information.
 
 ### Running Specific Test Files
 
-Specific test files can be run by passing the file paths or a directory that contains multiple test files. See [Managing Screenshots](#managing-screenshots) for generating ground truths before running screenshot tests.
+Scope each run to the tests you are working on by passing file paths, a directory that contains multiple test files, or a component name.
 
 **Specific Test Files**
 
 ```shell
-npm run test.e2e src/components/button/test/basic/button.e2e.ts src/components/button/test/a11y/button.e2e.ts
+npm run test.e2e.docker src/components/button/test/basic/button.e2e.ts src/components/button/test/a11y/button.e2e.ts
 ```
 
 **Test Directory with Multiple Files**
 
 ```shell
 # Will run all the test files in the `test` directory
-npm run test.e2e src/components/button/test
+npm run test.e2e.docker src/components/button/test
 ```
 
-### Running Tests Inside Docker
+**Component Names**
 
-While `npm run test.e2e` can be used to run tests in the same environment that you are developing in, `npm run test.e2e.docker` can be used to run tests in a Docker environment provided by the Ionic team through [Rancher Desktop](#installing-rancher-desktop). This command supports all the same features as `npm run test.e2e` detailed in the previous section.
+The argument is a Playwright filter, so a bare component name matches every test file whose path contains it.
 
-This command builds a Docker image before tests run. It will also re-build the Docker image in the event that a Playwright update was merged into the repo.
+```shell
+npm run test.e2e.docker checkbox radio toggle
+```
 
-Note that the Playwright report will not automatically open in your web browser when tests are complete because the tests were run in Docker. Run `npx playwright show-report` outside of Docker to open the most recent test report.
+### Running All Test Files
+
+Omitting the filter runs every E2E test file:
+
+```shell
+npm run test.e2e.docker
+```
+
+There are over 400 E2E test files, which CI runs in parallel across 20 shards. A single machine runs them one shard at a time, so prefer scoping a local run to the component you changed and let CI cover the rest.
+
+### Running Tests Outside of Docker
+
+`npm run test.e2e` runs the tests directly in the environment you are developing in. It accepts all of the same arguments as `npm run test.e2e.docker`.
 
 > [!NOTE]
-> Additional setup is needed to run Playwright tests with headed mode in Docker. See [Configuring Docker for Headed Tests](#configuring-docker-for-headed-tests-optional) for more information.
+> This command is a wrapper for `npx playwright test`. All data passed to `npm run test.e2e` can also be passed to `npx playwright test`.
+
+Use this only for tests that take no screenshots. Because screenshots are resolved per platform, a screenshot test run outside of Docker compares against a ground truth that is not in the repository. See [Managing Screenshots](#managing-screenshots) for why this passes locally and fails on CI.
 
 ### Headed vs. Headless Tests
 
@@ -146,14 +162,14 @@ No additional steps are needed in order to run the tests in headless mode:
 
 ```shell
 # Will run tests in headless mode
-npm run test.e2e src/components/chip
+npm run test.e2e.docker src/components/chip
 ```
 
  Playwright supports the `--headed` flag to run in headed mode which causes the visual representation of the browser to appear:
 
  ```shell
  # Will run tests in headed mode
- npm run test.e2e src/components/chip -- --headed
+ npm run test.e2e.docker src/components/chip -- --headed
  ```
 
 ### Debugging Tests
@@ -205,11 +221,18 @@ This is especially useful when CI reports a failure you cannot reproduce on your
 **Example:**
 
 ```shell
-npm run test.e2e.docker.update-snapshots src/components/radio/test/a11y/radio.e2e.ts -- --repeat-each=10
+npm run test.e2e.docker src/components/radio/test/a11y/radio.e2e.ts -- --repeat-each=10
 ```
 
 This runs the test 10 times, increasing the chance of catching the flaky behavior.
 
+> [!WARNING]
+> Reproduce a flaky failure with `test.e2e.docker`, not
+> `test.e2e.docker.update-snapshots`. On a mismatch the update variant overwrites
+> the ground truth and reports the test as **passing**, so the run goes green with
+> no diff images and the flaky screenshot is left in your working tree. Check
+> `git status` if you suspect this happened.
+
 #### 4. Pausing Test Execution
 
 Additionally, you can pause execution of a test by using the `page.pause()` method. This pauses the script execution and allows you to manually inspect the page in the browser.
@@ -238,6 +261,10 @@ test('example test', async ({ page }) => {
 
 If you are running a test that takes a screenshot, you must first generate the reference screenshot from your reference branch. This is known as generating a "ground truth screenshot". All other screenshots will be compared to this ground truth.
 
+Playwright appends the browser and platform to every screenshot name, so the same test resolves a different file per operating system. Example: `button-expand-md-ltr-Mobile-Chrome-linux.png`. The ground truths committed to the repository are the `-linux.png` files generated in Docker, and `.gitignore` excludes every other platform's.
+
+This is why screenshot tests should be run with `npm run test.e2e.docker`. Running them natively on macOS or Windows looks for a `-darwin.png` or `-win32.png` ground truth that is not in the repository. Playwright writes that file, fails the test once, and passes on every run afterward against a baseline that git ignores and CI never sees. The result is a test that passes locally and fails on CI.
+
 ### Generating or Updating Ground Truths With Docker (Local Development)
 
 We recommend generating ground truths inside of [Docker](https://www.docker.com) using [Rancher Desktop](#installing-rancher-desktop). This allows anyone contributing to Ionic Framework to create or update ground truths in a consistent environment.
@@ -319,7 +346,7 @@ test-results-[current shard]-[total shards]
 
 Example:
 
-test-results-2-5 --> Test results from job runner 2 out of 5.
+test-results-2-5 → Test results from job runner 2 out of 5.
 ```
 
 Download the appropriate artifact and unzip the file.