feat: update project to support new experiment and benchmark structure by joshblack · Pull Request #176 · primer/agent-eval

joshblack · 2026-08-27T05:06:52Z

This PR adds benchmark support to agent-eval, including benchmark configuration, execution, output, workflows, and website result handling.

This is the first entry in stack 214. #212 splits trial output into individual files, #201 adds reusable local sandbox images, #203 aligns the benchmark and experiment runner workflows, #204 adds durable plan sharding, and #199 adds the design system benchmark scenarios last.

Changelog

New

Add benchmark configuration and execution APIs.
Add benchmark output, tests, workflows, and example benchmark definitions.
Add website support for benchmark results and trends.
Add the @primer/agent-eval/sandbox entry point and schema-backed benchmark, experiment, scenario, treatment, and trial APIs.

Changed

Update experiment, trial, scenario, sandbox, and output models to support the benchmark flow.
Sort benchmark, experiment, and scenario discovery results for deterministic consumer output.
Allow script/run-benchmark.sh to select a benchmark with BENCHMARK_NAME, defaulting to design-system.
Preserve experiment scenario entries that use {path, name?}.
Preserve optional browser tests in the new trial model, including the legacy scenario.browser.test.ts filename.
Run browser tests with Playwright and combine their results with standard scenario test results.
Update CI and result handling for the new benchmark output format.
Derive the artifact directory from --output-dir or the directory containing --output.
Run optional capability setup after top-level benchmark setup for benchmark treatment trials.
Load filesystem module paths through file:// URLs while preserving URL-based module specifiers.
Accept Copilot sub-agent user.message events that omit agentMode and preserve their routing fields.
Remove the temporary agent-browser walkthrough skill before downloading trial skill artifacts.
Log commands before running them in evaluation sandboxes.
Keep the lockfile scoped to packages introduced by this stack layer.
Collect output token counts from Copilot model.message events while retaining compatibility with older output.
Compare benchmark test success rates across treatments and report equal metrics as a 0% change.
Restore interactive benchmark trend tooltips and model/treatment highlighting without hydration errors.
Add capability and dependent scenario filters to benchmark trends.
Format test pass axes from 0% to 100% and time axes in seconds.

Removed

Remove the --artifacts CLI option now that artifact placement follows the output path.
Stop passing the removed --artifacts option from the experiment workflow.
Remove superseded experiment helpers and baseline result handling.
Remove the legacy @primer/agent-eval/cli, @primer/agent-eval/experiments, @primer/agent-eval/scenarios, and @primer/agent-eval/output package exports. Use the executable for CLI usage and the package root or singular API entry points for programmatic usage.

Before / After

Before	After

Copilot

Copilot review overview

🔵 Needs a closer look

It introduces broad API/workflow/website refactors and new benchmark infrastructure, so it needs careful human validation of end-to-end behavior and compatibility.

Review tier: Lite
Findings: 1 · 3

New issues introduced by this change (4)

Severity	Finding
	`script/run-benchmark.sh` — `benchmark_name` is hard-coded to `design-system`, but the validation error message references…
	`packages/agent-eval/src/benchmark.ts` — `listBenchmarks` iterates over `readdir` results without sorting, which can lead to unstable…
	`packages/agent-eval/src/experiment.ts` — `listExperiments` iterates over `readdir` results without sorting, so experiment ordering can vary…
	`packages/agent-eval/src/scenario.ts` — `listScenarios` returns scenarios in whatever order the filesystem provides, which can be…

What changed in this PR

Adds first-class benchmark support to agent-eval end-to-end (config discovery, execution/output, CI workflows, and website pages), while consolidating/modernizing the public API surface (scenario/experiment/benchmark entry points, sandbox abstractions, trial/treatment modeling) and removing legacy baseline-oriented routes and helpers.

Changes:

Introduce benchmark configs/APIs, output handling, and a scheduled benchmark GitHub Actions workflow plus a local runner script.
Update website routing/UI to surface benchmarks (overview, benchmark list/detail, and run detail pages) and retire baseline pages/components.
Refactor core library structure: new scenario/experiment discovery APIs, new sandbox host abstractions, updated Copilot event parsing, and updated public exports/changesets.

File	Description
website/src/scenarios.ts	Switch website scenario loading to the new `@primer/agent-eval/scenario` entry point.
website/src/experiments.ts	Switch website experiment loading to the new `@primer/agent-eval/experiment` entry point with explicit directories.
website/src/benchmarks.ts	Add website-side benchmark discovery/loading via `@primer/agent-eval/benchmark`.
website/src/benchmark-page-data.ts	Compose benchmark + run data for benchmark pages/overview.
website/src/app/page.tsx	Home page now shows latest design-system benchmark overview.
website/src/app/layout.tsx	Update site metadata description to reflect benchmarks.
website/src/app/experiments/[id]/runs/[date]/page.tsx	Migrate experiment run details page to shared RunDetailsPage + new run-details helpers.
website/src/app/components/RunDetailsPage.tsx	Generalize run details UI to accept a generic “resource” (benchmarks or experiments).
website/src/app/components/ResourceTables.tsx	Add BenchmarksTable for the `/benchmarks` index route.
website/src/app/components/PageHeader.tsx	Replace “Baseline” nav item with “Benchmarks”.
website/src/app/components/Index.tsx	Remove baseline index UI implementation.
website/src/app/components/Index.module.css	Remove baseline index CSS.
website/src/app/components/BenchmarkTrends.module.css	Update benchmark trends styling and add new layout/hit-target styles.
website/src/app/components/BenchmarkOverview.tsx	Add benchmark overview component (latest metrics + trends).
website/src/app/components/BenchmarkOverview.module.css	Add styling for benchmark metric formatting.
website/src/app/components/BaselineOverview.tsx	Remove baseline overview component.
website/src/app/benchmarks/page.tsx	Add `/benchmarks` route to list benchmarks.
website/src/app/benchmarks/[id]/runs/[date]/page.tsx	Add benchmark run detail route reusing RunDetailsPage.
website/src/app/benchmarks/[id]/page.tsx	Add benchmark detail route (results + runs list).
website/src/app/benchmarks/[id]/components/Page.tsx	Add client UI for benchmark detail page tables and run links.
website/src/app/baseline/page.tsx	Remove baseline landing route.
website/src/app/baseline/[scenarioId]/page.tsx	Remove baseline per-scenario route.
website/README.md	Update website route documentation and document results bundle layout.
script/setup	Update setup to download recent retained benchmark artifacts instead of baseline.
script/run-benchmark.sh	Add local benchmark runner script (currently design-system-focused).
script/run-baseline.sh	Remove baseline runner script.
scenarios/005-agent-enables-theme-switching/scenario.config.ts	Update to `defineConfig` API.
scenarios/004-agent-setup-nextjs/scenario.config.ts	Update to `defineConfig` API.
scenarios/003-agent-uses-form-from-primer/scenario.config.ts	Update to `defineConfig` API.
scenarios/002-agent-uses-octicon-from-primer/scenario.config.ts	Update to `defineConfig` API.
scenarios/001-agent-uses-button-from-primer/scenario.config.ts	Update to `defineConfig` API.
scenarios/000-vite-template/scenario.config.ts	Update to `defineConfig` API.
scenarios/000-nextjs-template/scenario.config.ts	Update to `defineConfig` API.
README.md	Document benchmarks alongside experiments and update CLI usage/docs accordingly.
pnpm-workspace.yaml	Add `benchmarks` workspace package and reorder workspace paths.
packages/agent-eval/src/treatment.ts	Replace treatment types with zod schemas + ControlTreatment export.
packages/agent-eval/src/treatment.test.ts	Add schema-focused treatment tests.
packages/agent-eval/src/scenarios.ts	Remove legacy scenarios entry point.
packages/agent-eval/src/scenarios.test.ts	Remove legacy scenarios tests.
packages/agent-eval/src/scenario.ts	Add new scenario discovery/loading APIs + schemas and browser test filename support.
packages/agent-eval/src/scenario.test.ts	Replace scenario tests to validate new scenario APIs/behavior.
packages/agent-eval/src/scenario-config.ts	Remove legacy `defineScenario` helper.
packages/agent-eval/src/sandbox/virtual.ts	Add virtual sandbox implementation for tests/unit flows.
packages/agent-eval/src/sandbox/virtual.test.ts	Add virtual sandbox tests.
packages/agent-eval/src/sandbox/types.ts	Define the sandbox interface/types and plugin/MCP config shapes.
packages/agent-eval/src/sandbox/system.test.ts	Add/expand tests for SystemSandbox helpers and SandboxSchema acceptance.
packages/agent-eval/src/sandbox/path.ts	Add container path resolution helper.
packages/agent-eval/src/sandbox/path.test.ts	Add tests for container path resolution.
packages/agent-eval/src/sandbox/index.ts	Add sandbox barrel export entry point.
packages/agent-eval/src/sandbox/index.test.ts	Validate sandbox entry point exports.
packages/agent-eval/src/sandbox/constants.ts	Centralize sandbox paths/user constants.
packages/agent-eval/src/sandbox/constants.test.ts	Add consistency tests for sandbox constants.
packages/agent-eval/src/sandbox/captured-stream.ts	Add captured stream utility for line-aware output capture.
packages/agent-eval/src/sandbox/captured-stream.test.ts	Add tests for captured stream behavior.
packages/agent-eval/src/sandbox.test.ts	Remove legacy sandbox tests.
packages/agent-eval/src/run.test.ts	Remove legacy run tests.
packages/agent-eval/src/resolve-experiment-scenario.ts	Remove legacy experiment scenario resolver.
packages/agent-eval/src/plan.ts	Add plan creation + queued execution with retry logic.
packages/agent-eval/src/plan.test.ts	Add tests for plan shuffling, execution ordering, and retry behavior.
packages/agent-eval/src/output.ts	Remove legacy monolithic output model.
packages/agent-eval/src/output.test.ts	Remove legacy output tests.
packages/agent-eval/src/model.ts	Replace model config expansion with model-variant schemas/helpers.
packages/agent-eval/src/model.test.ts	Update tests for new model variant expansion behavior.
packages/agent-eval/src/mcp-config.test.ts	Add tests for MCP config schema parsing.
packages/agent-eval/src/logger.ts	Add shared logger setup (pretty locally, quiet in tests).
packages/agent-eval/src/logger.test.ts	Add logger tests.
packages/agent-eval/src/index.ts	Update public exports to new benchmark/experiment/scenario/sandbox/trial APIs.
packages/agent-eval/src/index.test.ts	Add public-export surface tests.
packages/agent-eval/src/host.ts	Add SystemHost/VirtualHost abstraction for filesystem/module loading and sandbox creation.
packages/agent-eval/src/host.test.ts	Add tests for host behaviors.
packages/agent-eval/src/experiments.ts	Remove legacy experiments entry point.
packages/agent-eval/src/experiments.test.ts	Remove legacy experiments tests.
packages/agent-eval/src/experiment.test.ts	Add tests for new experiment discovery/loading and inline scenario resolution.
packages/agent-eval/src/experiment-output.test.ts	Add tests for experiment output serialization/portability behavior.
packages/agent-eval/src/experiment-config.ts	Remove legacy experiment config types/helpers.
packages/agent-eval/src/environment.ts	Add unified environment config resolution (including `--output-dir` semantics).
packages/agent-eval/src/environment.test.ts	Add tests for environment config resolution/validation.
packages/agent-eval/src/copilot-cli.ts	Extend Copilot event parsing to accept new/optional fields and `model.message` token counts.
packages/agent-eval/src/copilot-cli.test.ts	Add/update tests for new Copilot event variants.
packages/agent-eval/src/cli.test.ts	Add CLI smoke tests for help + required token + option validation.
packages/agent-eval/src/benchmark.test.ts	Add benchmark discovery/execution/output tests.
packages/agent-eval/rolldown.config.ts	Add benchmark entry point and switch scenario entry to `src/scenario.ts`.
packages/agent-eval/README.md	Update package docs for benchmarks, new APIs, and result bundle guidance.
packages/agent-eval/package.json	Update exports (add benchmark/sandbox, remove legacy entry points) and add new deps.
experiments/noop.ts	Add experiment global + treatment setup examples under new config model.
experiments/baseline.ts	Remove baseline experiment definition.
benchmarks/tsconfig.json	Add TS config for benchmarks workspace package.
benchmarks/package.json	Add benchmarks workspace package definition.
benchmarks/noop.ts	Add noop benchmark definition (test/e2e scaffold).
benchmarks/design-system.ts	Add design system benchmark definition and scenario grouping.
.gitignore	Ignore root-level output.json.
.github/workflows/report-workflow-failures.yml	Update failure reporting to track `benchmark` workflow instead of `baseline`.
.github/workflows/experiment.yml	Update experiment workflow artifact layout + merging for new output model.
.github/workflows/deploy.yml	Update deploy trigger and artifact download filter to `benchmark-*`.
.github/workflows/benchmark.yml	Add scheduled/dispatch benchmark workflow to produce retained benchmark artifacts.
.github/workflows/baseline.yml	Remove baseline workflow.
.changeset/tidy-cli-entrypoint.md	Changeset: remove `@primer/agent-eval/cli` entry point.
.changeset/tender-benchmarks-compare.md	Changeset: benchmark comparison metric adjustment.
.changeset/steady-trial-results.md	Changeset: exclude temporary walkthrough skill from artifacts.
.changeset/quick-copilot-metrics.md	Changeset: accept sub-agent user messages missing agentMode + collect model.message token counts.
.changeset/portable-results-bundle.md	Changeset: add `--output-dir` portable bundle mode.
.changeset/plain-scenarios-discover.md	Changeset: scenario discovery host support + browser test filename support.
.changeset/neat-models-vary.md	Changeset: new model variant schema/helpers replacing older model exports.
.changeset/lucky-sandboxes-open.md	Changeset: new `@primer/agent-eval/sandbox` entry point and sandbox interface.
.changeset/kind-trials-compare.md	Changeset: new trial/treatment schemas and trial comparison helpers.
.changeset/happy-output-maps.md	Changeset: new map-based experiment output model and (de)serialization helpers.
.changeset/gentle-scenarios-unify.md	Changeset: migrate scenario APIs to `@primer/agent-eval/scenario`.
.changeset/fresh-experiment-configs.md	Changeset: add ExperimentConfigSchema + new experiment config model.
.changeset/calm-experiments-unify.md	Changeset: migrate experiment APIs to `@primer/agent-eval/experiment`.
.changeset/bright-benchmarks-run.md	Changeset: add benchmark APIs + CLI selection support.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Copilot review overview

🟡 Changes recommended

The experiment GitHub Actions workflow still passes the removed --artifacts flag and SystemHost.loadModule() needs more robust path handling to avoid runtime failures on non-POSIX paths.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 · 2

New issues introduced by this change (3)

Severity	Finding
	`.github/workflows/experiment.yml` — The workflow still passes `--artifacts`, but the updated CLI options no longer include an…
	`benchmarks/noop.ts` — The benchmark config schema only supports a top-level `setup` (see `BenchmarkConfigSchema` in…
	`packages/agent-eval/src/host.ts` — `SystemHost.loadModule()` calls `import(filepath)` directly. Several call sites pass filesystem…

Issues resolved since last review (4)

Severity	Finding
	`packages/agent-eval/src/scenario.ts` — `listScenarios` returns scenarios in whatever order the filesystem provides, which can be… View resolved comment
	`packages/agent-eval/src/experiment.ts` — `listExperiments` iterates over `readdir` results without sorting, so experiment ordering can vary… View resolved comment
	`packages/agent-eval/src/benchmark.ts` — `listBenchmarks` iterates over `readdir` results without sorting, which can lead to unstable… View resolved comment
	`script/run-benchmark.sh` — `benchmark_name` is hard-coded to `design-system`, but the validation error message references… View resolved comment

Files not reviewed (1)

pnpm-lock.yaml: Generated file

Copilot

Copilot review overview

🔵 Needs a closer look

website/src/runs.ts has a null-safety bug in isLegacyRunOutput that can throw while parsing unsupported/invalid JSON, which would break run listing/loading at runtime.

Review tier: Lite
Findings: None

Issues resolved since last review (3)

Severity	Finding
	`packages/agent-eval/src/host.ts` — `SystemHost.loadModule()` calls `import(filepath)` directly. Several call sites pass filesystem… View resolved comment
	`benchmarks/noop.ts` — The benchmark config schema only supports a top-level `setup` (see `BenchmarkConfigSchema` in… View resolved comment
	`.github/workflows/experiment.yml` — The workflow still passes `--artifacts`, but the updated CLI options no longer include an… View resolved comment

Files not reviewed (1)

pnpm-lock.yaml: Generated file

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

website/src/runs.ts:87

isLegacyRunOutput can throw at runtime when the parsed JSON is not an object (or is missing fields), because it dereferences output.treatments / output.results even when asRecord() returns null. Since parseOutput() calls this on arbitrary JSON, the type guard should be null-safe and only access properties after confirming output is a record.
packages/agent-eval/src/plan.ts:10
Grammar: “to be ran” should be “to be run”.

…hards

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d12a97a-b8d3-45ad-a09a-3c76d1411705

Remove scenario package entries that are introduced by the final stack layer instead of the benchmark infrastructure branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d12a97a-b8d3-45ad-a09a-3c76d1411705

Allow user.message events without agentMode and preserve the routing fields emitted for sub-agent prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d12a97a-b8d3-45ad-a09a-3c76d1411705

Remove package export declarations for legacy entry points that are no longer built, allowing publint to validate a clean package build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d12a97a-b8d3-45ad-a09a-3c76d1411705

Remove the temporary agent-browser skill after walkthrough capture so downloaded skill artifacts contain only the evaluation configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d12a97a-b8d3-45ad-a09a-3c76d1411705

Read output token counts from model message events while retaining compatibility with older assistant message output. Compare test success rates across benchmark treatments and report equal values as 0%. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 87b13945-ea60-40ed-b1b1-1c4e9ee24723

Fix chart hydration errors, restore interactive trend tooltips and highlighting, and add capability and scenario filters with metric-aware axes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d12a97a-b8d3-45ad-a09a-3c76d1411705

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7eb8999b-2277-4d41-915f-fb35cac43373

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 33a4d86e-9f6e-4031-b77c-499d3530b6fc

Copilot

Copilot review overview

🟡 Changes recommended

The new experiment output builder can silently corrupt results when treatment names collide (including a user-defined “Control”), so it should fail fast on duplicate treatment names before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1

New issues introduced by this change (1)

Severity	Finding
	`packages/agent-eval/src/experiment.ts` — `output()` keys `treatments` by `trial.treatment.name` and also stores `treatmentId` as the…

Files not reviewed (1)

pnpm-lock.yaml: Generated file

+    if (!result.treatments.has(trial.treatment.name)) {
+      result.treatments.set(trial.treatment.name, trial.treatment)
+    }


This was referenced Sep 3, 2026

test: add benchmark scenarios #199

Draft

perf: prebuild local sandbox images #201

Merged

feat: split trial output files #212

Merged

joshblack marked this pull request as ready for review September 4, 2026 03:33

Copilot AI lite review requested due to automatic review settings September 4, 2026 03:33

Copilot started reviewing on behalf of joshblack September 4, 2026 03:33 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread script/run-benchmark.sh Outdated

Comment thread packages/agent-eval/src/benchmark.ts Outdated

Comment thread packages/agent-eval/src/experiment.ts Outdated

Comment thread packages/agent-eval/src/scenario.ts Outdated

Copilot AI review requested due to automatic review settings September 4, 2026 03:48

joshblack force-pushed the feat/add-benchmark-support branch from 31f9d0e to 53f5073 Compare September 4, 2026 03:48

Copilot started reviewing on behalf of joshblack September 4, 2026 03:49 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread .github/workflows/experiment.yml

Comment thread benchmarks/noop.ts

Comment thread packages/agent-eval/src/host.ts

joshblack changed the title ~~feat: add support for benchmarks~~ refactor: update project to support new experiment and benchmark structure Sep 4, 2026

joshblack changed the title ~~refactor: update project to support new experiment and benchmark structure~~ feat: update project to support new experiment and benchmark structure Sep 4, 2026

Copilot AI review requested due to automatic review settings September 4, 2026 04:11

Copilot started reviewing on behalf of joshblack September 4, 2026 04:11 View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

joshblack added 14 commits September 3, 2026 23:28

feat: add benchmark folder

c4c1036

feat: add benchmark scenarios

5de02fb

feat: add run to benchmark

ebc72b2

test: add host for local testing

742ebe4

refactor: update scenario implementation

73c5a55

refactor: update model implementation

de0d665

test: add test file for benchmarks

797a9c4

feat: add benchmark functionality

ab2f28e

feat: add trial concept for individual treatment runs

0a0c7fa

feat: add plan concept for ordering trials and storing order across s…

f1a7eda

…hards

refactor: update benchmarks, scenario, treatment, and run

249e755

refactor: add virtual and system sandboxes

ff1dc6b

feat: add sandbox interface for testing

dc9807c

refactor: remove host arg from runTrial

4a6ea53

joshblack and others added 24 commits September 3, 2026 23:28

refactor: update tests and benchmark summaries

d09dff1

ci: add workflows for benchmark

02f360f

feat: switch up output format, change website

d3b45e1

refactor: remove baseline results

2f0e34a

feat: move towards benchmark trends

db929c3

refactor: clean up benchmark models and sandbox cleanup

cb59e18

refactor: move benchmark scenarios to stacked change

aa64c16

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

chore: log sandbox commands

2e98455

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d12a97a-b8d3-45ad-a09a-3c76d1411705

fix: accept sub-agent user messages

02edbcd

Allow user.message events without agentMode and preserve the routing fields emitted for sub-agent prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d12a97a-b8d3-45ad-a09a-3c76d1411705

fix: preserve scenario path and browser tests

3897fc6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

refactor: remove cli package entrypoint

5ca8506

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

docs: shorten benchmark changesets

031cf72

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

docs: update benchmark API documentation

1c3bc63

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

refactor: remove baseline experiment

3f36743

test: consolidate experiment coverage

f57a0f5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7eb8999b-2277-4d41-915f-fb35cac43373

fix: resolve benchmark support integration

ac56793

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7eb8999b-2277-4d41-915f-fb35cac43373

refactor: derive artifact directory from output

47d2719

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7eb8999b-2277-4d41-915f-fb35cac43373

fix: address benchmark review feedback

e58491f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: address remaining benchmark review feedback

76ed0da

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 33a4d86e-9f6e-4031-b77c-499d3530b6fc

Copilot AI review requested due to automatic review settings September 4, 2026 04:29

joshblack force-pushed the feat/add-benchmark-support branch from 2488da5 to 76ed0da Compare September 4, 2026 04:29

Copilot started reviewing on behalf of joshblack September 4, 2026 04:30 View session

joshblack merged commit c9d33ed into main Sep 4, 2026
10 checks passed

joshblack deleted the feat/add-benchmark-support branch September 4, 2026 04:32

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Comment thread packages/agent-eval/src/experiment.ts

Comment on lines +293 to +295

if (!result.treatments.has(trial.treatment.name)) {

result.treatments.set(trial.treatment.name, trial.treatment)

}

Navigation Menu

Uh oh!

feat: update project to support new experiment and benchmark structure - #176

feat: update project to support new experiment and benchmark structure#176
joshblack merged 52 commits into
mainfrom
feat/add-benchmark-support

joshblack commented Aug 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

joshblack commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog

New

Changed

Removed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

joshblack commented Aug 27, 2026 •

edited

Loading