Issues

• evaluate_from_api.py: a failed API call drops the question from the result file and the summary denominator, so reported accuracy changes with which calls fail

  TIGER-AI-Lab/MMLU-Pro#82 · Filter by author shaurya416 opened on Sep 23, 2026

• Would an EvalPort-format export of MMLU-Pro be welcome in EvalPort's benchmarks/?

  TIGER-AI-Lab/MMLU-Pro#81 · Filter by author adhabnr-ux opened on Aug 22, 2026

• EvalScope integration: one-command evaluation support for MMLU-Pro

  TIGER-AI-Lab/MMLU-Pro#80 · Filter by author Yunnglin opened on Jul 24, 2026

• Benchmark Result Mismatch

  TIGER-AI-Lab/MMLU-Pro#79 · Filter by author Linux-Server opened on May 13, 2026

• Typo report

  TIGER-AI-Lab/MMLU-Pro#76 · Filter by author Qifeng-Wu99 opened on Oct 17, 2025

• questions about dataset construction details

  TIGER-AI-Lab/MMLU-Pro#75 · Filter by author Anna-lsy opened on Oct 14, 2025

• Validate small high-scoring models

  TIGER-AI-Lab/MMLU-Pro#69 · Filter by author EwoutH opened on Mar 17, 2025

• Add Grok-1212

  TIGER-AI-Lab/MMLU-Pro#61 · Filter by author johns2s opened on Dec 19, 2024

• Add DeepSeek-V2.5-1210

  TIGER-AI-Lab/MMLU-Pro#58 · Filter by author EwoutH opened on Dec 13, 2024

• Add Amazon Nova models

  TIGER-AI-Lab/MMLU-Pro#51 · Filter by author EwoutH opened on Dec 4, 2024

• Add Gemini-Exp-1114 and Gemini-Exp-1121

  TIGER-AI-Lab/MMLU-Pro#47 · Filter by author EwoutH opened on Nov 18, 2024

• OpenAI o1-preview and o1-mini

  TIGER-AI-Lab/MMLU-Pro#21 · Filter by author EwoutH opened on Sep 17, 2024
  11