Skip to content

fix(api): keep artifacts when browser tests fail - #356

Open
TheNaubit wants to merge 1 commit into
theam:mainfrom
TheNaubit:fix/browser-test-failure-artifacts
Open

TheNaubit wants to merge 1 commit into
theam:mainfrom
TheNaubit:fix/browser-test-failure-artifacts

Conversation

@TheNaubit

@TheNaubit TheNaubit commented Sep 9, 2026

Copy link
Copy Markdown

What changes

Save the screenshots, traces and reports a browser test writes, even when the test exits with an error. The service registers those files and records the redacted result event before returning the failure.

If saving the evidence also fails, keep the original test error so it doesn't get hidden by the second failure. The docs explain how to retrieve failed-test artifacts from the story, since a failed operation doesn't return the successful result's artifact list.

This is a nonbreaking fix using the existing artifact and event records. It needs no migration and doesn't change permissions, budgets or the API schema.

Why

Closes #352

When a browser test fails, the screenshots and traces are often what you need to work out why. Throwing before registering them leaves that evidence out of the story.

Verification

I ran the checks in a local Docker Sandbox microVM, with no host directories mounted.

  • pnpm --filter @facility/api exec vitest run test/project-environment.integration.test.ts --fileParallelism=false passed all 10 tests. The new cases check that failed tests keep their screenshot and trace, secrets are redacted, and an artifact scan error doesn't replace the original test error.
  • API typecheck, Biome checks, docs tests and the docs build passed.
  • pnpm verify passed. The dependency audit still has the repo's two existing ignored high advisories; this PR adds no exceptions.

I couldn't complete the Docker workspace checks. The published runner is linux/amd64, and the local sandbox is ARM64. It exits with exec /usr/local/bin/docker-entrypoint.sh: exec format error; the test then gets HTTP 409 because the container has stopped. Trying emulation inside the sandbox didn't fix it. These checks still need to pass on a compatible runner before merge.

  • pnpm verify passes locally
  • Behaviour verified beyond the test suite (say how)
  • Documentation updated, or no user-facing change

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4ba9c2252c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +389 to +392
const failure =
result.exitCode === 0
? undefined
: commandFailure("environment.browser_test", script, safeResult);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retain artifacts after timed-out browser tests

Handle rejected command executions as failed tests before returning: Docker's exec throws workspace_command_timeout when the configured 30-minute timeout expires (services/api/src/workspaces/docker.ts:147-172), so this exitCode-based branch is never reached and a browser test that writes a screenshot or trace and then hangs exits at the preceding await without scanning/inserting its artifacts or appending its result event. This leaves a common failed-test case without the evidence this change is intended to retain; collect the artifact directory in the timeout/error path while preserving the original timeout error.

Useful? React with 👍 / 👎.

@TheNaubit TheNaubit changed the title fix(api): retain browser test artifacts after failure fix(api): keep artifacts when browser tests fail Sep 9, 2026

@adrian-lorenzo adrian-lorenzo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this.

I confirmed the timeout case from the existing review: a screenshot written before timeout remains on disk but never reaches the story. Please collect artifacts when execution throws too, preserve the original error, and add a timeout regression test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failed browser tests leave screenshots and traces out of the story

2 participants