Evaluator Bench
Epoch AI, 2025-01-23; retrieved 2026-09-14 tier 3 self (epoch)confirmed

Clarifying the creation and use of the FrontierMath benchmark

https://epoch.ai/latest/openai-and-frontiermath

OpenAI commissioned 300 problems, owns them, has statements and solutions except 50-problem holdout; contractor communication gap. Cited from a search snippet; not re-fetched in full. [verify 2026-09-15] fetched; page reachable, citation matches; 1 quoted span(s) found verbatim

Signals citing this source

  • Epoch AI, F (against): OpenAI commissioned and owns most of FrontierMath and had the problems and solutions; a contract barred disclosing this until the o3 launch.“we cannot share the questions and answers with other parties without written permission from OpenAI”
  • Epoch AI, R (against): Contributors were not told the funder; disclosure came after the fact.“Per our agreement, we needed OpenAI’s permission before publicly disclosing their involvement”
  • Epoch AI, R (for): After the disclosure failure, created a 50-problem holdout OpenAI cannot see.“We are finalizing a 50-problem set for which OpenAI will only receive the problem statements and not the solutions”
  • Epoch AI, A (against): Cannot share the commissioned set with other labs without OpenAI's permission.“we cannot share the questions and answers with other parties without written permission from OpenAI”
  • Epoch AI, S (for): Evaluates any model on FrontierMath at its discretion.“Epoch AI is free to conduct and publish evaluations of any models using the FrontierMath problem set”
  • Epoch AI, S (against): FrontierMath, the asset on which Epoch evaluates OpenAI and other labs, was commissioned and is owned by OpenAI, which holds the problems and solutions outside the 50-problem holdout.“OpenAI commissioned Epoch AI to produce 300 advanced math problems for AI evaluation”

Ledger rows citing this URL

  • T34 OpenAI commissioned FrontierMath problems and owns them; paid consultation client per transparency page confirmed