Skip to content

feat(evaluation): add trajectory improvement metrics - #9

Open
linhongyu510 wants to merge 2 commits into
ByteDance-Seed:mainfrom
linhongyu510:feat/trajectory-improvement-metrics
Open

feat(evaluation): add trajectory improvement metrics#9
linhongyu510 wants to merge 2 commits into
ByteDance-Seed:mainfrom
linhongyu510:feat/trajectory-improvement-metrics

Conversation

@linhongyu510

@linhongyu510 linhongyu510 commented Aug 14, 2026

Copy link
Copy Markdown

Summary

Add trajectory-level improvement metrics to EdgeBench evaluation history so iterative agent runs can be analyzed beyond the final score.

Metrics include initial, final, and best values; total improvement; improving/non-improving counts; and time to first improvement and best result. Both maximize and minimize directions are supported.

Agent-visible history is recomputed from agent-visible entries, so automatic submissions are not leaked.

Robustness

  • orders fully timestamped submissions by submitted_at
  • excludes missing, NaN, and infinite rewards
  • preserves strict JSON serialization on numeric overflow
  • retains insertion order when timestamps are incomplete
  • returns a stable empty-trajectory shape

Validation

  • pytest tests/test_trajectory_metrics.py -q — 8 passed
  • API coverage for filtered agent history
  • git diff --check

AI assistance

Developed with AI assistance; metric semantics and visibility boundaries were manually reviewed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant