feat(eval): introduce new reporter API

[Abhijeet Prasad (AbhiPrasad)]

AI Summary

Summary

This change introduces a unified reporter system for bt eval. Eval execution now produces a canonical lifecycle event stream that can be consumed by terminal, machine-readable, artifact, and devserver reporters without coupling rendering behavior to execution logic.

User-facing impact

Selectable reporters

bt eval now supports repeatable reporter selection:

bt eval . --reporter=fancy
bt eval . --reporter=verbose
bt eval . --reporter=jsonl
bt eval . --reporter=events
bt eval . --reporter=dot
bt eval . --reporter=junit --output-file=results.xml
bt eval . --reporter=github-actions
bt eval . --reporter=silent

Equivalent environment configuration is available through:

BRAINTRUST_EVAL_REPORTER=fancy,github-actions
BRAINTRUST_EVAL_OUTPUT_FILE=results.xml

Explicit --reporter flags replace the default reporter set. Reporters can be composed as long as no more than one claims stdout.

Existing defaults remain familiar

Existing commands retain their intended behavior:

• bt eval uses the fancy reporter.
• bt eval --verbose uses the verbose terminal reporter.
• bt eval --jsonl produces JSONL summaries on stdout while retaining human-facing progress and errors on stderr.
• --list remains an execution mode rather than a reporter.

Reliable machine-readable output

User code printing to stdout no longer corrupts --jsonl output. Under --jsonl, captured user console output is routed to stderr, leaving stdout as parseable JSONL.

The new events reporter exposes the full canonical lifecycle as NDJSON for integrations and custom tooling.

New CI reporters

The reporter API enables CI-focused output:

• dot renders compact per-case status.
• junit writes JUnit XML and requires --output-file.
• github-actions emits escaped workflow annotations.
• Artifact write failures may veto an otherwise successful run, ensuring CI does not silently continue without required output.

Case-oriented reporters require real per-case results. When used with an older Braintrust SDK that only exposes aggregate progress, they fail with an actionable upgrade message rather than inventing incomplete case data.

Deprecations and compatibility

There are no CLI deprecations.

--jsonl and --verbose remain supported as permanent convenience aliases. Existing eval files and runner configuration continue to work.

The runner decoder supports both:

• The new canonical event protocol.
• The previous processing, start, progress, summary, error, and done vocabulary through a legacy adapter.

Older SDKs therefore continue to work with fancy, verbose, and summary reporters. Only reporters requiring full case fidelity need an SDK upgrade.

Migration story

Existing CLI users

No migration is required. Existing commands retain their defaults, with one intentional improvement: --jsonl now guarantees clean stdout.

Users can adopt explicit reporters incrementally:

# Existing
bt eval . --jsonl

# Explicit equivalent machine reporter
bt eval . --reporter=jsonl

# CI artifact reporting
bt eval . --reporter=junit --output-file=results.xml

SDK compatibility

The bundled JavaScript and Python runners now emit the canonical protocol. They feature-detect available case information:

• Newer SDKs can provide real case IDs, statuses, scores, durations, and streaming deltas.
• Older SDKs fall back to synthetic completion events for terminal progress.
• Case-dependent reporters refuse degraded streams and explain that the installed SDK must be upgraded.

Devserver clients

Legacy devserver SSE remains the default and is byte-compatible for existing browser clients.

Clients can opt into canonical events by sending:

x-bt-stream-fmt: canonical

This allows the playground and other clients to migrate independently without breaking older versions.

Why the new API is better

One lifecycle across every output format

All reporters consume the same structured lifecycle:

run:start
eval:start
case:start
case:end
eval:end
run:end

Errors, console messages, progress totals, and case deltas are modeled as structured side-channel events. Terminal output, JSONL, JUnit, GitHub annotations, and devserver SSE no longer maintain separate interpretations of an eval run.

Stable identity instead of name-based correlation

Events carry runId, evalId, and caseId. Reporters no longer need to correlate progress and summaries using display names, which were not guaranteed to match.

Case IDs use root span IDs when available, creating a direct link between reporter output and trace data.

Better separation of concerns

Execution policy, output format, and verbosity are now independent:

• Run modes decide what executes.
• Reporters decide how results are rendered.
• Verbosity is represented by selecting fancy or verbose.

Adding a new output format no longer requires modifying eval execution or the devserver.

Safer composition

A shared terminal facade coordinates persistent lines and live progress regions. This prevents multiple reporters from tearing terminal output.

The manager also:

• Serializes event delivery.
• Isolates reporter failures.
• Logs a failing reporter only once.
• Guarantees exactly one run:end.
• Prevents multiple machine reporters from interleaving on stdout.
• Aggregates explicit end-of-run vetoes.

Efficient streaming

Large case inputs and outputs remain in the SDK process. Live case:delta events are emitted only when an installed reporter advertises interest, avoiding unnecessary serialization and process-boundary traffic.

[Abhijeet Prasad (AbhiPrasad)]

1. Overall architecture

flowchart LR
    User["User / CI"] --> CLI["bt eval"]

    CLI -->|starts subprocess| Runner["Bundled JS or Python runner"]
    Runner --> SDK["Installed Braintrust SDK"]
    SDK --> Eval["Eval execution"]

    Eval --> SDKManager["SDK ReporterManager"]
    SDKManager --> Bridge["bt bridge reporter"]
    Bridge -->|Canonical events over SSE| CLIManager["bt ReporterManager"]

    CLIManager --> Fancy["fancy / verbose / dot"]
    CLIManager --> Machine["jsonl / events"]
    CLIManager --> Artifact["junit"]
    CLIManager --> CI["github-actions"]

    Fancy --> STDERR["stderr"]
    Machine --> STDOUT["stdout"]
    Artifact --> FILE["results.xml"]
    CI --> STDERR

Loading

The key change is that eval execution produces one structured event stream. Every output format consumes that stream instead of implementing its own eval logic.

---

2. A normal bt eval run

sequenceDiagram
    actor User
    participant BT as bt CLI
    participant Runner as Bundled runner
    participant SDK as Braintrust SDK
    participant RM as bt ReporterManager
    participant Reporter as Selected reporters

    User->>BT: bt eval tests/ --reporter=fancy
    BT->>BT: Validate reporter selection
    BT->>Runner: Spawn with SSE endpoint
    Runner->>SDK: Discover and execute evaluators

    Runner-->>BT: run:start
    BT->>RM: dispatch(run:start)
    RM->>Reporter: onRunStart()

    loop Each evaluator
        Runner-->>BT: eval:start
        BT->>RM: dispatch(eval:start)
        RM->>Reporter: onEvalStart()

        Runner-->>BT: eval:progress(totalCases)
        RM->>Reporter: onProgress()

        loop Concurrent cases
            Runner-->>BT: case:start
            RM->>Reporter: onCaseStart()

            opt Reporter requested streaming output
                Runner-->>BT: case:delta
                RM->>Reporter: onCaseDelta()
            end

            Runner-->>BT: case:end
            RM->>Reporter: onCaseEnd()
        end

        Runner-->>BT: eval:end
        RM->>Reporter: onEvalEnd()
    end

    Runner-->>BT: run:end
    BT->>RM: dispatch(run:end)
    RM->>Reporter: onRunEnd()

    BT->>RM: finish()
    RM->>Reporter: finish terminal and artifacts
    RM-->>BT: Aggregate reporter vetoes
    BT-->>User: Final exit status

Loading

Reporter callbacks are delivered serially, even when evaluators and cases execute concurrently.

---

3. Canonical lifecycle

flowchart TD
    RS["run:start<br/>runId, evaluatorCount, protocolVersion"]

    RS --> ES1["eval:start<br/>evalId, name, experiment"]
    RS --> ES2["eval:start<br/>evalId, name, experiment"]

    ES1 --> CS1["case:start<br/>caseId, index, name"]
    ES1 --> CS2["case:start<br/>caseId, index, name"]
    ES2 --> CS3["case:start<br/>caseId, index, name"]

    CS1 --> CE1["case:end<br/>status, duration, scores"]
    CS2 --> CE2["case:end<br/>status, duration, scores"]
    CS3 --> CE3["case:end<br/>status, duration, scores"]

    CE1 --> EE1["eval:end<br/>counts, summary, errors"]
    CE2 --> EE1
    CE3 --> EE2["eval:end<br/>counts, summary, errors"]

    EE1 --> RE["run:end<br/>status, duration, errors"]
    EE2 --> RE

    ERR["error<br/>structured scope"] -.-> RS
    CON["console<br/>stdout or stderr"] -.-> ES1
    PROG["eval:progress<br/>expected total"] -.-> ES2
    DELTA["case:delta<br/>opt-in streaming"] -.-> CS1

Loading

Lifecycle events establish ordering and identity. Side-channel events can occur anywhere inside the run.

---

4. What SDKs are expected to do

An SDK is an event producer and may also host its own reporters.

flowchart TD
    Engine["SDK eval engine"] --> Emit["Canonical event emitter"]

    Emit --> Manager["SDK ReporterManager"]

    Manager --> Builtin["SDK built-in reporters"]
    Manager --> Custom["User-defined reporters"]
    Manager --> HostBridge["Host bridge reporter"]

    HostBridge -->|bt eval| BtSSE["SSE to bt CLI"]
    HostBridge -->|SDK devserver| PlaygroundSSE["SSE to playground"]
    HostBridge -->|standalone SDK| LocalTerminal["Local terminal"]

    Manager --> Interest{"Any reporter wants<br/>case deltas?"}
    Interest -->|Yes| Stream["Generate case:delta"]
    Interest -->|No| Skip["Skip delta generation"]

Loading

SDK responsibilities

An SDK should:

1. Create one runId for the invocation.
2. Assign a stable evalId to each evaluator.
3. Use the case root span ID as caseId.
4. Emit lifecycle events in scope order.
5. Serialize dispatch through its reporter manager.
6. Emit structured errors immediately.
7. Aggregate the same errors into eval:end or run:end.
8. Emit run:end exactly once, including crashes and interruption paths.
9. Only generate case:delta when a reporter requests it.
10. Keep full input, output, and expected values in-process.

The SDK should not let reporters decide execution order, retries, concurrency, or normal run success.

---

5. Per-case SDK workflow

sequenceDiagram
    participant Engine as SDK eval engine
    participant Span as Root case span
    participant Manager as SDK ReporterManager
    participant Bridge as Host bridge

    Engine->>Span: Create case root span
    Span-->>Engine: rootSpanId

    Engine->>Manager: case:start(caseId=rootSpanId)
    Manager->>Bridge: onCaseStart()

    Engine->>Engine: Run task

    opt Delta subscription active
        Engine->>Manager: case:delta(caseId, text/json/reasoning)
        Manager->>Bridge: onCaseDelta()
    end

    Engine->>Engine: Run scorers
    Engine->>Span: Record output, scores, and errors

    Engine->>Manager: case:end(caseId, status, duration, scores)
    Manager->>Bridge: onCaseEnd()

Loading

case:end is emitted after task and scorer processing, so it carries the final status and scores.

The case payload deliberately does not include complete input or output data. Custom reporters needing those values run inside the SDK process.

---

6. Reporter placement

Standalone SDK execution

flowchart LR
    App["Application"] --> SDK["SDK Eval()"]
    SDK --> Manager["SDK ReporterManager"]
    Manager --> Terminal["SDK terminal reporter"]
    Manager --> Custom["Custom in-process reporter"]

Loading

Custom reporters have in-process access to full case data.

bt eval

flowchart LR
    BT["bt CLI"] --> Runner["Bundled runner"]
    Runner --> SDK["Installed SDK"]
    SDK --> SDKManager["SDK ReporterManager"]
    SDKManager --> Bridge["bt bridge reporter"]
    Bridge -->|Canonical SSE| BTManager["bt ReporterManager"]
    BTManager --> Selected["CLI-selected reporters"]

Loading

The SDK does not need to know which CLI reporter was selected. It only emits the canonical stream.

The exception is case:delta interest: bt tells the runner whether any selected reporter needs deltas.

Playground/devserver

flowchart LR
    Browser -->|POST /eval| Devserver["bt devserver"]
    Devserver --> Runner
    Runner --> SDK
    SDK --> Canonical["Canonical event stream"]
    Canonical --> Bridge["HTTP bridge reporter"]

    Browser -->|No format header| Legacy["Legacy playground SSE"]
    Browser -->|x-bt-stream-fmt: canonical| Modern["Canonical SSE"]

Loading

---

7. Old SDK versus new SDK

flowchart TD
    Runner["Bundled bt runner"] --> Detect{"Does installed SDK expose<br/>real case lifecycle data?"}

    Detect -->|Yes| Real["Emit real case:start / case:end<br/>caseId = root span ID<br/>scores + duration + status"]
    Detect -->|No| Synthetic["Derive synthetic case:end<br/>from legacy completion progress"]

    Real --> Full["All reporters available"]
    Full --> Dot["dot"]
    Full --> JUnit["junit"]
    Full --> GHA["github-actions"]
    Full --> Fancy["fancy / verbose"]

    Synthetic --> Basic["Summary and progress reporters work"]
    Basic --> Fancy
    Basic --> JSONL["jsonl / events"]

    Synthetic --> Refuse["Case-dependent reporters refuse"]
    Refuse --> Upgrade["Actionable SDK upgrade message"]

Loading

Synthetic events are sufficient to count completed cases, but they do not invent names, scores, durations, or root span IDs.

---

8. Devserver compatibility and migration

sequenceDiagram
    participant Client
    participant Devserver
    participant Runner
    participant SDK

    Client->>Devserver: POST /eval
    Devserver->>Runner: Start eval
    Runner->>SDK: Execute
    SDK-->>Runner: Canonical lifecycle
    Runner-->>Devserver: Canonical SSE

    alt No x-bt-stream-fmt header
        Devserver-->>Client: Legacy start/progress/summary/error/done
    else x-bt-stream-fmt: canonical
        Devserver-->>Client: run:start/eval:start/case:*/eval:end/run:end
    end

Loading

Existing playground clients require no changes. New clients can opt into the canonical protocol when ready.

---

9. Why SDK integration is simpler

Previously, hosts depended on unrelated callback shapes:

flowchart LR
    Eval["Eval execution"] --> Progress["Progress callback"]
    Eval --> Stream["Stream callback"]
    Eval --> Start["onStart callback"]
    Eval --> Summary["Summary reporter"]
    Eval --> Error["Ad-hoc errors"]

Loading

Now those callbacks converge into one contract:

flowchart LR
    Eval["Eval execution"] --> Events["Canonical event emitter"]
    Events --> Manager["ReporterManager"]
    Manager --> Any["Any reporter or bridge"]

Loading

This gives SDK implementations:

• One lifecycle to test.
• Stable IDs instead of name-based joins.
• One failure-isolation boundary.
• One termination guarantee.
• A standard way to advertise expensive streaming interest.
• The same bridge implementation for CLI and devserver hosts.
• A clean path for custom in-process reporters without expanding the wire payload.

[github-actions]

Latest downloadable build artifacts for this PR commit 91930ec9b058:

• Workflow run: https://github.com/braintrustdata/bt/actions/runs/29053876986
• Download all artifacts (GitHub CLI): gh run download 29053876986 --repo braintrustdata/bt
• Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

• artifacts-build-global
• artifacts-build-local-aarch64-pc-windows-msvc
• artifacts-build-local-x86_64-pc-windows-msvc
• artifacts-build-local-x86_64-apple-darwin
• artifacts-build-local-x86_64-unknown-linux-musl
• artifacts-build-local-aarch64-apple-darwin
• artifacts-build-local-x86_64-unknown-linux-gnu
• artifacts-build-local-aarch64-unknown-linux-gnu
• artifacts-plan-dist-manifest
• cargo-dist-cache