Skip to content

test(eval): exercise GitHub Actions reporter#265

Closed
Abhijeet Prasad (AbhiPrasad) wants to merge 3 commits into
abhi-reporter-testfrom
abhi-feat-test-github-actions-reporter
Closed

test(eval): exercise GitHub Actions reporter#265
Abhijeet Prasad (AbhiPrasad) wants to merge 3 commits into
abhi-reporter-testfrom
abhi-feat-test-github-actions-reporter

Conversation

@AbhiPrasad

Copy link
Copy Markdown
Member

Summary

  • add a self-contained eval that intentionally errors one case
  • add a GitHub Actions workflow that runs the eval with --reporter=github-actions and verifies the workflow annotation
  • capture case identity events from the JS SDK bridge so local/no-send-logs evals provide real case IDs instead of a degraded synthetic stream

User-facing test

The workflow prints a command like:

::error title=case 0 errored::intentional error from the GitHub Actions reporter example

GitHub renders that command as an annotation. The eval itself exits non-zero by design, while the workflow verifies both the annotation and expected exit status.

How to run locally

pnpm install --dir examples/eval-reporters --frozen-lockfile
cargo build --bin bt
cd examples/eval-reporters
../../target/debug/bt eval --no-send-logs \
  --reporter=github-actions \
  github-actions.eval.ts

Validation

  • mise exec -- pre-commit run --all-files
  • cargo test --test eval_golden
  • local GitHub Actions reporter example

Stacking

This PR is stacked on #264 and should be reviewed after that reporter API PR.

@AbhiPrasad Abhijeet Prasad (AbhiPrasad) deleted the abhi-feat-test-github-actions-reporter branch July 10, 2026 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant