Skip to content

Fix submission intake reliability#56

Merged
isPANN merged 2 commits into
mainfrom
codex/fix-submission-intake-reliability
Jul 18, 2026
Merged

Fix submission intake reliability#56
isPANN merged 2 commits into
mainfrom
codex/fix-submission-intake-reliability

Conversation

@isPANN

@isPANN isPANN commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • snapshot R2 incoming/ keys and transition only objects from that batch
  • quarantine permanent input failures while retaining retryable failures
  • serialize scoring workflow runs and scope R2 credentials to storage steps
  • enforce the official schema, pinned library commit, and clean-run requirement
  • preserve submitter and run metadata through private scoring
  • validate request byte size in the Worker and stop exposing internal R2 keys

Why

The previous recursive archive could move submissions that arrived during scoring without
processing them, while one malformed object could jam the entire queue. Scored detail also
dropped metadata needed by downstream leaderboard entries.

User impact

The CLI contract is unchanged. Internal submitters still use
python -m benchmark.submit; test submissions remain private and excluded from the public
leaderboard.

Validation

  • uv run pytest -q — 218 passed, 4 skipped
  • wrangler deploy --dry-run
  • Python compile check
  • JSON/YAML parse checks
  • git diff --check

@isPANN
isPANN marked this pull request as ready for review July 18, 2026 15:18
@isPANN
isPANN merged commit e986921 into main Jul 18, 2026
1 check passed
@isPANN
isPANN deleted the codex/fix-submission-intake-reliability branch July 18, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant