Skip to content

fix(Evals): binary_tests no longer depends on GNU timeout and runs its command through a shell (#2188) - #2195

Open
Bowlerjim wants to merge 1 commit into
danielmiessler:mainfrom
Bowlerjim:fix/2188-binary-tests-macos
Open

Bowlerjim wants to merge 1 commit into
danielmiessler:mainfrom
Bowlerjim:fix/2188-binary-tests-macos

Conversation

@Bowlerjim

Copy link
Copy Markdown

Fixes #2188.

Two defects in one line. The grader shelled out to timeout, which is GNU coreutils and absent on a stock macOS install, so every test file came back bun: command not found: timeout and was graded failed (0/N on a suite that passes). The same line interpolated ${command} into a tagged template, which escapes a multi-word test command into a single argv token.

This spawns /bin/sh -c (cmd /c on win32) with Bun's own timeout/killSignal instead of the external binary.

One non-obvious detail: the pipe drain is capped rather than awaited outright. A killed shell can leave an orphan child holding stdout open, and awaiting new Response(proc.stdout).text() in that state made the timeout worthless, measured at 30s on a 1.5s budget.

Verified on macOS:

case before after
passing test file 0/1 1/1
failing test file 0/1 0/1
sleep 30, timeout_ms 1500 0/1 after 30s 0/1 after 3.5s

🤖 Generated with Claude Code

https://claude.ai/code/session_01PuS77R8a6Lhp7jK6dnwdTh

…its command through a shell (danielmiessler#2188)

The grader shelled out to `timeout`, which is GNU coreutils and absent on a
stock macOS install, so every test file came back `bun: command not found:
timeout` and was graded failed — 0/N on a suite that passes. The same line
interpolated `${command}` into a tagged template, which escapes a multi-word
test command into a single argv token.

Spawn `/bin/sh -c` (`cmd /c` on win32) with Bun's own `timeout`/`killSignal`
instead. The pipe drain is capped, because a killed shell can leave an orphan
child holding stdout open: awaiting it outright made the timeout worthless
(measured 30s on a 1.5s budget).

Verified on macOS: passing file -> 1/1 (was 0/1), failing file -> 0/1,
`sleep 30` with timeout_ms 1500 -> 0/1 in 3.5s (was 30s).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PuS77R8a6Lhp7jK6dnwdTh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evals binary_tests grader always reports 0/N on macOS: it shells out to timeout (GNU coreutils) and interpolates the test command as one argv token

1 participant