Skip to content

test(snapshot): add cross-runtime presentation conformance - #1973

Open
thymikee wants to merge 4 commits into
fix/android-snapshot-quality-contractsfrom
fix/android-snapshot-conformance
Open

test(snapshot): add cross-runtime presentation conformance#1973
thymikee wants to merge 4 commits into
fix/android-snapshot-quality-contractsfrom
fix/android-snapshot-conformance

Conversation

@thymikee

@thymikee thymikee commented Aug 22, 2026

Copy link
Copy Markdown
Member

Summary

Add one shared RawAXNode-to-presented conformance fixture for Android XML presentation and Swift SnapshotPresentation.

The vectors cover cumulative nested-scroll clipping, raw geometry, viewport-edge clipping, scope re-rooting, and preservation of acquired actionability. The nested-scroll case now intersects two distinct scroll bounds so neither runtime can accidentally pass with only the nearest ancestor.

Part of #1832 Step 9. C4/C5/C6 are owned by stacked #1968/#1972; drawing-order pruning and corpus work remain evidence-gated.

Validation

The earlier planted red established the Swift regression: presentation upgraded acquired raw-false nodes to hittable. The owning visibility fold now preserves node.hittable, and both runtime projections consume the same expected values.

Exact-head pnpm check:affected --run passed at c772c16: 633 files / 5,028 tests plus format, lint, typecheck, layering, fallow, build, and XCTest lane-selection validation (0 methods unreachable by all lanes).

Live Android validation on this exact head produced a healthy android-helper 0.20.10 snapshot across two windows in 142 ms with nested scroll depth preserved; the purpose-specific session was closed. GitHub remains authoritative for host/native XCTest and required CI.

5 files changed, +476/-9 against #1972; the full stack is 28 files, +2,159/-450 against main. Scope stayed within the shared snapshot-presentation contract and its Android/Swift consumers.

@github-actions

github-actions Bot commented Aug 22, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 2.37 MB 2.37 MB 0 B
JS gzip 796.4 kB 796.4 kB 0 B
npm tarball 918.2 kB 918.2 kB +10 B
npm unpacked 3.18 MB 3.18 MB +41 B

npm unpacked components

Component Base Current Diff
JS / dist source 2.52 MB 2.52 MB 0 B
Apple runner source/project 564.2 kB 564.3 kB +41 B
macOS helper source 54.5 kB 54.5 kB 0 B
Android helper artifacts 0 B 0 B 0 B
Other package files 44.5 kB 44.5 kB 0 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.8 ms 28.1 ms +0.3 ms
CLI --help 79.1 ms 79.0 ms -0.1 ms

Top changed chunks: no changes in the largest emitted chunks.

Top changed packed files

Packed file Base Current Diff
dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SnapshotVisibilityFold.swift 11.7 kB 11.7 kB +41 B

@thymikee

Copy link
Copy Markdown
Member Author

Not ready at 5a619b515. The new conformance gate is doing useful work and exposes a real owner-contract drift: Swift's SnapshotVisibilityFold recomputes positive non-root nodes as hittable, so the shared cases emit true for the acquired-non-hittable Outer and scoped Settings ScrollViews, while Android preserves false. Per the presentation contract, fix the owning Swift fold so presentation only narrows acquired actionability (or explicitly revise the shared policy); don't paper over this by changing one runtime's expected values. Separately, the branch is DIRTY/conflicting because #1972's base was rewritten from 34bed0fa to b9b997a5; rebase the single conformance commit. That rebase should remove the inherited file-size Coverage failure, but the Swift host red is PR-owned and must be fixed.

@thymikee
thymikee force-pushed the fix/android-snapshot-conformance branch from 5a619b5 to c8e4718 Compare August 22, 2026 15:04
@thymikee

Copy link
Copy Markdown
Member Author

Addressed on c8e47181b (rebased onto 6943369b9 / #1972).

  • Rebased only the conformance commit; stale inherited feat(android): bound snapshot presentation quality #1972 history is gone.
  • Swift SnapshotVisibilityFold now preserves acquired node.hittable and only narrows it with enabled/effective geometry, so it cannot upgrade raw false.
  • Updated the existing unit assertion to prove raw-false covered/labeled nodes stay false, raw-true positive overlay stays true, and disabled stays false.
  • Android shared conformance remains green; a fresh unit-test-enabled macOS runner build is green; xctest selection is green.
  • The final affected gate passed: 637 files and 5,044 tests.
  • Host XCTest remains locally blocked by LaunchServices worker materialization after 111s, so that exact CI lane remains authoritative. No 33-screen corpus is claimed.

@thymikee

Copy link
Copy Markdown
Member Author

The one-line Swift actionability fix is correct and its focused assertion is load-bearing, but three blockers remain. (1) Replace Closes #1832 Step 9 with Part of #1832/advances Step 9: GitHub will close the whole issue even though C4/C5/C6 and Step 9’s API-level differential legs remain. (2) The shared “cumulative scroll clip” fixture has only one scroll ancestor, so it proves direct clipping—not cumulative nested clipping. Add outer+inner scroll containers whose intersection is narrower than either individual clip and require both Swift and Android presentation to emit that effective rect. (3) Exact rewritten heads for #1973 and base #1972 currently have only CodeQL; obtain authoritative Swift Runner Host XCTests/full CI after the base stabilizes. Also describe the Android leg accurately as XML parser + presentation conformance; it reconstructs hierarchy from depth/order rather than consuming the same RawAXNode seam.

@thymikee
thymikee force-pushed the fix/android-snapshot-conformance branch from c8e4718 to c772c16 Compare August 22, 2026 16:45
@thymikee

Copy link
Copy Markdown
Member Author

Addressed on c772c16.

The published branch is mergeable; required CI is running.

@thymikee

Copy link
Copy Markdown
Member Author

Exact-head Android Smoke inherits #1972's owner-action regression; it is not infrastructure and not caused by #1973's incremental conformance/Swift diff. Separate #1972/#1973 jobs both successfully click the native-alert control, then time out waiting for Automation confirmation after 20/18 readable helper captures. The PR body's healthy two-window Settings snapshot does not exercise this alert route. Diagnose/fix #1972 with an immediate post-click tree/image showing whether the click lands and whether the dialog survives multi-window presentation; obtain green Android Smoke on #1972, then restack/rerun #1973. Every other #1973 lane, including Host XCTest and iOS smoke, is green; do not label while Android is red.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant