Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4ba9c2252c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const failure = | ||
| result.exitCode === 0 | ||
| ? undefined | ||
| : commandFailure("environment.browser_test", script, safeResult); |
There was a problem hiding this comment.
Retain artifacts after timed-out browser tests
Handle rejected command executions as failed tests before returning: Docker's exec throws workspace_command_timeout when the configured 30-minute timeout expires (services/api/src/workspaces/docker.ts:147-172), so this exitCode-based branch is never reached and a browser test that writes a screenshot or trace and then hangs exits at the preceding await without scanning/inserting its artifacts or appending its result event. This leaves a common failed-test case without the evidence this change is intended to retain; collect the artifact directory in the timeout/error path while preserving the original timeout error.
Useful? React with 👍 / 👎.
adrian-lorenzo
left a comment
There was a problem hiding this comment.
Thanks for fixing this.
I confirmed the timeout case from the existing review: a screenshot written before timeout remains on disk but never reaches the story. Please collect artifacts when execution throws too, preserve the original error, and add a timeout regression test.
What changes
Save the screenshots, traces and reports a browser test writes, even when the test exits with an error. The service registers those files and records the redacted result event before returning the failure.
If saving the evidence also fails, keep the original test error so it doesn't get hidden by the second failure. The docs explain how to retrieve failed-test artifacts from the story, since a failed operation doesn't return the successful result's artifact list.
This is a nonbreaking fix using the existing artifact and event records. It needs no migration and doesn't change permissions, budgets or the API schema.
Why
Closes #352
When a browser test fails, the screenshots and traces are often what you need to work out why. Throwing before registering them leaves that evidence out of the story.
Verification
I ran the checks in a local Docker Sandbox microVM, with no host directories mounted.
pnpm --filter @facility/api exec vitest run test/project-environment.integration.test.ts --fileParallelism=falsepassed all 10 tests. The new cases check that failed tests keep their screenshot and trace, secrets are redacted, and an artifact scan error doesn't replace the original test error.pnpm verifypassed. The dependency audit still has the repo's two existing ignored high advisories; this PR adds no exceptions.I couldn't complete the Docker workspace checks. The published runner is
linux/amd64, and the local sandbox is ARM64. It exits withexec /usr/local/bin/docker-entrypoint.sh: exec format error; the test then gets HTTP 409 because the container has stopped. Trying emulation inside the sandbox didn't fix it. These checks still need to pass on a compatible runner before merge.pnpm verifypasses locally