feat(verify): borrow an emulator on a CI runner when the machine cannot run one

[mikepitre]

What & why

• The end-to-end tests in feat(verify): add the end-to-end test package and the verify-clerk-android skill #1046 run on an emulator on the machine that runs their CLI. An agent in a cloud sandbox often cannot run one. This PR adds a second backend to that CLI. On a machine that cannot run the emulator, up starts a session on a GitHub Actions runner, and the CLI drives the emulator there through a tunnel, with the same verbs, tests, and evidence.
• No session has been started from the present head of this branch, so the remote backend is covered by unit tests only.
• The tests and the CLI are the package e2e-tests/, and the verify-clerk-android skill is the instructions that point an agent at it.
• It sits on ci: run the end-to-end tests in the release workflow, and remove the Trailblaze suite #1052, and ci: publish the evidence a borrowed-device session hands off #1079, which adds the upload workflow, sits on it. Nothing in test(e2e): configure the e2e host from verify launch extras #1041, feat(verify): add the end-to-end test package and the verify-clerk-android skill #1046, or ci: run the end-to-end tests in the release workflow, and remove the Trailblaze suite #1052 needs it.
• --backend remote on doctor, up, and run forces the remote backend. With no flag the CLI picks the local backend where the machine can run the emulator and the remote one anywhere else, and its first line says which and why.
• .github/workflows/verify-remote.yml runs one session. It builds the pushed commit, never the working tree, boots the emulator, records the video, and ends on down, after 15 idle minutes, or at 60 minutes.
• --runner <label> names the runner for a new session. The default is ubuntu-24.04.
• With the remote backend, doctor checks that git can fetch and push the branch, that GitHub has HEAD, and that this machine reaches GitHub, the tunnel host, and Clerk. It starts nothing without --live. doctor --live starts one short session and stops it.
• down ends the runner job, and down --stale also ends a session that a crashed run of the same checkout left running.
• When the machine's gh cannot attach files, attach hands the video and screenshots to the session's runner through the tunnel, and the runner keeps them as the artifact verify-evidence.
• In the skill directory, references/remote.md is new and SKILL.md describes the remote backend. Outside it and e2e-tests/, the PR adds verify-remote.yml and one line in android-test.yml, and it changes one sentence in AGENTS.md.

To try it, on any machine with Node 24.8 or newer, the team's Clerk Platform API key, and a GitHub token that can push to this repository and start a workflow in it:

$ npm ci --prefix e2e-tests
$ git push
$ e2e-tests/bin/control-clerk-android doctor --backend remote
$ e2e-tests/bin/control-clerk-android run auth-start --backend remote
$ e2e-tests/bin/control-clerk-android down

On a machine that cannot run the emulator, leave out --backend remote. Allow about ten minutes for the first up, most of it the first Gradle build.

What to focus on

Five commits in dependency order, then six commits added after review.

Commit	Lines	What it does
1. Evidence in the description	397 added, 100 removed	Makes attach write a run's video and screenshots into the pull request description, in one block that a later attach replaces. In #1046 it posts a comment.
2. The remote session	2,111 added	Adds the workflow that boots an emulator on a GitHub runner for one session, the code on the runner that drives the emulator, and the code on your machine that talks to it through a tunnel.
3. The choice of backend	293 added, 82 removed	Teaches the CLI to choose between an emulator on this machine and one on a runner, and tells the tests' support code how to reach a borrowed one. Outside the unit tests, commits 1 and 3 and two small edits in commit 2 hold every edit to code that #1046 added.
4. Unit tests	2,078 added, 7 removed	Tests of the borrowed-device code.
5. Docs	158 added, 9 removed	Tells an agent how a session works, what it costs, and how to end it.
6. Proxy port	19 added, 6 removed	Probes an https: proxy on port 443 when its URL names no port, with the same rule the doctor's check uses. Added after review.
7. Free ports in the tests	17 added, 3 removed	Gives each session agent the unit tests start a port that is free. The guessed ports were inside the range Linux hands out for outgoing connections, and one CI run failed on a taken port. Added after review.
8. Renewing a stopped session	33 added, 7 removed	When the held session has stopped, asks the session that replaces it for the pushed HEAD, not for the commit the stopped one built, which a run refuses once the branch has moved on. Added after review.
9. Skill fixes	7 added, 5 removed	Fixes what an agent with no prior knowledge found when it followed these directions on clerk-ios: a verb missing from the list, the order of steps on a local run, how to assert on a label, and when a spec becomes a golden spec.
10. Order of the verbs	1 added, 1 removed	Lists attach before down in AGENTS.md, the order a remote session needs.
11. Build limit and tunnel wait	175 added, 13 removed	Raises the limit on a remote build from 30 to 45 minutes and reports a build that runs out of time as timed out, not as a build error. The step that waits for the tunnel stops when the session agent has exited. Came from a failed Expo session on the free macOS runner; the code is shared.

Commit 1 has its own unit tests. The unit tests for commits 2 and 3 arrive in commit 4.

Everything under e2e-tests/src/core/ and e2e-tests/specs/support/, and most of the unit tests, are the same files, byte for byte, in clerk/clerk-ios#661 and clerk/javascript#10131. src/core/MANIFEST lists every shared file with its hash, and a unit test fails when one drifts. Review them once.

Not proven:

• A session started from this head. No session has run from these commits, and none has run the present 19 tests.
• doctor --live, the idle stop, and the 60-minute cap, at this head.

What limits the risk:

• A session starts by workflow_dispatch only, so it takes write access to start one and a fork cannot. The workflow has no push or pull_request trigger and uses no repository or environment secret. Its one permissions block is contents: read, and no job has a write permission. The step that reads the request uses the job's own read-only token to check the requested commit. A session starts only for the commit the run was dispatched on, or for a later commit of that branch.
• Every route on the tunnel answers 403 without a bearer token that the CLI made. The token stays under e2e-tests/.verify/remote/ on the machine that started the session. The workflow gets only its SHA-256.
• The tunnel is a Cloudflare quick tunnel, so Cloudflare can read the traffic, which includes sign-in tickets for test users. Clerk's keys stay on the machine that drives the session, and down deletes the throwaway Clerk application.
• A session costs nothing on the default ubuntu-24.04 label. A Blacksmith label is billed by the minute.
• A run on the machine's own emulator takes the same path as before this PR. Apart from attach, the new code runs only when the CLI picks the remote backend. The release gate in ci: run the end-to-end tests in the release workflow, and remove the Trailblaze suite #1052 does not use it.

npm test --prefix e2e-tests passes 672 of 672 at this head with no network, key, or emulator, and npm run typecheck --prefix e2e-tests reports nothing.

Screenshots / video

None attached.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Android verification can run locally or on a remote GitHub Actions runner, with automatic backend selection and optional runner selection.
  • Remote sessions support app builds, video recording and collection, device logs, and session health checks.
  • Remote verification includes environment checks, session cleanup, and automatic use of a network proxy when needed.
  • Test evidence can be added to the pull request description or handed off to the remote runner when direct publication is unavailable. Existing evidence for the same platform is replaced.
• Documentation
  • Added guidance for remote setup, runner selection, committed test changes, session cleanup, and evidence publishing.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

The change adds GitHub Actions-backed Android emulator sessions and remote device support. It adds remote session planning, authenticated device operations, lease and preflight handling, and evidence handoff. The CLI, workflows, documentation, and tests cover local and remote execution.

Changes: Remote Android verification

Layer / File(s)	Summary
Remote contracts and CLI .claude/skills/verify-clerk-android/*, AGENTS.md, e2e-tests/src/core/types.ts, e2e-tests/src/core/cli.ts, e2e-tests/src/core/remote/{protocol,settings,tunnel}.ts, e2e-tests/specs/support/*, e2e-tests/src/core/launch.*, e2e-tests/src/host.ts	The CLI accepts remote and --runner. Shared types describe remote requests, leases, builds, and borrowed devices. Device configuration carries remote daemon credentials. Runtime startup selects direct or proxy egress.
Runner workflow and Android session .github/workflows/{verify-remote,android-test}.yml, e2e-tests/src/core/remote/{session-agent,recipe}.ts, e2e-tests/src/platform/android/*, e2e-tests/testing/*	The workflow plans and starts sessions, prepares the runner and optional emulator, starts the tunnel and session agent, waits for completion, cleans up, and uploads evidence. The session agent validates commits, enforces bearer access, builds eligible revisions, and exposes device operations.
Remote backend and lease lifecycle e2e-tests/src/core/{devices,workspace,verbs,manifest,types}.ts, e2e-tests/src/core/remote/{backend,github,preflight,session,settings}.ts	The remote backend dispatches and monitors workflows, checks pushed commits and core versions, manages builds and leases, reports session health, and supports remote doctor checks.
Evidence handoff and publication e2e-tests/src/core/{publish,verbs,types}.ts, e2e-tests/src/core/remote/{backend,handoff,session,session-agent}.ts, e2e-tests/test/{evidence,handoff,session-agent,run-groups}.test.ts	Evidence publication edits a marked block in the pull-request description. If direct attachment is unavailable for a remote run, the CLI validates the session and pull request, then transfers evidence in validated chunks.

Priority: ➖ Normal

Estimated code review effort: 5 (Critical) | ~100 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant VerifyCLI
  participant GitHubActions
  participant SessionAgent
  participant AndroidEmulator
  VerifyCLI->>GitHubActions: dispatch remote session request
  GitHubActions->>SessionAgent: plan and start session
  SessionAgent->>AndroidEmulator: build and run device operations
  VerifyCLI->>SessionAgent: send authenticated session requests

Loading

sequenceDiagram
  participant AttachCommand
  participant PublishModule
  participant RemoteBackend
  participant SessionAgent
  AttachCommand->>PublishModule: edit pull-request description and attach files
  PublishModule-->>AttachCommand: return attachment result
  AttachCommand->>RemoteBackend: hand off evidence when direct attachment fails
  RemoteBackend->>SessionAgent: send manifest and evidence chunks

Loading

Merge Risk: 🟡 Moderate · up to f4eaf

A slow remote build can exceed its limit or appear successful after timing out, delaying verification and making retries unreliable. Fix the timeout handling before merging.

Pre-merge checks: 4 · 1

Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 0.69% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 145 functions across 45 files. (2 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly summarizes the main change: using a CI runner emulator when the local machine cannot run one.
Description check		The description includes the required What & why, What to focus on, and Screenshots / video sections. It explains the remote backend, testing status, risks, limitations, and review focus in sufficient…

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.69% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 145 functions across 45 files. (2 skipped: 2 unsupported.)

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[swolfand]

Pushed fixes for the review findings above:

• f6c66ff merge ci: run the end-to-end tests in the release workflow, and remove the Trailblaze suite #1052's fixes (conflicts in src/core/launch.mjs, src/host.ts, and MANIFEST: kept this branch's proxy egress code and remoteBackend, dropped the npx node@24 rerun, kept withoutClerkKeys for Gradle; MANIFEST regenerated)
• 2f17453 ci(verify): start a remote session only on the runner labels the repository uses (ubuntu-24.04, ubuntu-latest, blacksmith-4vcpu-ubuntu-2404; remote.md updated to match)

The comment near the end of verify-remote.yml was already gone after your f4eaf89.