evaluator; distance from a lab: 1
Epoch AI
Scorecard
Capability-tracking and benchmark nonprofit; the FrontierMath episode is the canonical undisclosed-funding case.
Nonprofit, Remote. Independent referee; listed with independent referees. Confidence high. Domains: benchmarks.
not ranked 8/8 evidenced · Retrieved & confirmed Scores by preset: Lab procurement –; Regulator or auditor –; Public trust –; Equal weights –. Weakest evidenced dimension: access depth (lab-granted) 1/4.
The weighted numbers are hidden until you ask for them: two readers with different priorities see different numbers, and neither is the site's.
What would move the score. A standing rule against confidential lab funding of any evaluation asset.
- Funding 2 to 3: score –
- Governance 2 to 3: score –
- Personnel 2 to 3: score –
Dissent, lower. Funding should be 1. OpenAI commissioned and owns FrontierMath, the asset Epoch's reputation rests on, and held approval over disclosing that fact for a year. OpenAI, Google DeepMind, xAI and Anthropic are all paying clients, the revenue split is undisclosed, and Epoch invests in AI stocks. 'Every major lab' as clients with an undisclosed share reads as F.5's material lab revenue, not F.3's occasional fee.
Dissent, higher. Funding should be 3. Coefficient's $25.1M over eight grants dwarfs client fees, and Epoch discloses every donation above $70,000 and every client by name and year. It charges at least consultant rates so labs do not get subsidised work, keeps a 50-problem holdout OpenAI cannot see, and publishes evaluations of every lab's models on the same benchmark, including results OpenAI did not commission.
Each value is the tightest admissible cap or, with no cap, the highest admissible floor, under RULES.md. The binding signal is highlighted. A dash means no admissible signal sets a bound under the Retrieved & confirmed policy.
Funding 2/4
Anchor 2: Labs pay per engagement; otherwise diversified.
evidence: tier 1 (filing/index)
floor and cap disagree
Conflict: floor 3 from epoch.05 (F.6) against cap 2 from epoch.01, epoch.11, epoch.13 (F.3, F.3, F.3); resolved at 2 by F.3: F.3 fee cap decides; the philanthropic floor is itself limited to 3 by F.6.
Curator: Anchor 2 on funding given the cited signals. Curator disclosure (2026-09-15): the curator holds small public-market shareholdings in Google and Meta. This evaluator has a confirmed ledger tie to Google (contract row T35). Per the disclosure Rule, this assessment is flagged for public review: the rationale names the holding so readers can weigh it, and any reader can file a correction through the contribution path.
Resolution: F.3 decides 2. F.3 fee cap decides; the philanthropic floor is itself limited to 3 by F.6.
- OpenAI commissioned and owns most of FrontierMath and had the problems and solutions; a contract barred disclosing this until the o3 launch. caps at 2 by F.3binding“we cannot share the questions and answers with other parties without written permission from OpenAI”
Clarifying the creation and use of the FrontierMath benchmark (Epoch AI, 2025-01-23) tier 3 self (epoch)confirmed OpenAI quietly funded independent math benchmark (The Decoder, 2025-01-19) tier 4 press (primary)confirmed2026-09-14 - Primarily philanthropically funded (Open Philanthropy). floors at 3 by F.6“Epoch AI, a nonprofit primarily funded by Open Philanthropy”
AI benchmarking organization criticized for waiting to disclose funding from OpenAI (TechCrunch, 2025-01-19) tier 4 press (primary)confirmed2026-09-14 - Transparency page lists Google DeepMind and OpenAI as paying consultation clients and a $600K Tallinn DAF gift; Coefficient grants of about $24.5M per index. caps at 2 by F.3binding“Google DeepMind 2026 Evaluating the AI co-mathematician on FrontierMath 2025 Model Evaluations”
Transparency (Epoch AI, 2026) tier 3 self (epoch)confirmed Coefficient Giving grants index (Coefficient Giving, 2026-09-11) tier 1 indeximported2026-09-11 - Transparency page lists eight Coefficient grants (largest $8,500,000 in April 2025; about $25M cumulative), a $600,000 Tallinn DAF gift, $195,000 from SFF, and paying clients including OpenAI, Google DeepMind, xAI, an Anthropic pilot, the EU AI Office, Sequoia Capital Global Equities and Bridgewater. caps at 2 by F.3binding“xAI 2025 Model Evaluations Anthropic 2024 Small pilot for new benchmark”
Transparency (Epoch AI, 2026) tier 3 self (epoch)confirmed2026 - Epoch says it charges lab and industry clients at least industry-consultant rates so it is not subsidising their work, and commits to disclosing sponsorship and data-access agreements; its eight Coefficient grants sum to $25.1M by its own list. floors at 2 by F.3“We aim to charge prices at least on par with industry consultants”
Transparency (Epoch AI, 2026) tier 3 self (epoch)confirmed2026
Governance 2/4
Anchor 2: For-profit or PBC with a published COI policy.
evidence: tier 4 (press)
Floored at 2 by epoch.06 (G.1): Nonprofit.. No admissible signal caps it.
- Nonprofit. floors at 2 by G.1binding“Epoch AI, a nonprofit primarily funded by Open Philanthropy”
AI benchmarking organization criticized for waiting to disclose funding from OpenAI (TechCrunch, 2025-01-19) tier 4 press (primary)confirmed2026-09-14
Personnel 2/4
Anchor 2: Frequent two-way hiring; recusal on request.
evidence: tier 4 (press)
Floored at 2 by epoch.09 (P.6): No lab roles found.. No admissible signal caps it.
- No lab roles found. floors at 2 by P.6binding
AI benchmarking organization criticized for waiting to disclose funding from OpenAI (TechCrunch, 2025-01-19) tier 4 press (primary)confirmed2026-09-14
Access depth (lab-granted) 1/4
Anchor 1: Pre-release API with safeguards on.
evidence: tier 3 (self)
mechanism: lab-controlled
Capped at 1 by epoch.07 (A.1): Cannot share the commissioned set with other labs without OpenAI's permission.. Floors up to 1 from epoch.18 do not exceed the cap.
- Cannot share the commissioned set with other labs without OpenAI's permission. caps at 1 by A.1binding“we cannot share the questions and answers with other parties without written permission from OpenAI”
Clarifying the creation and use of the FrontierMath benchmark (Epoch AI, 2025-01-23) tier 3 self (epoch)confirmed2026-09-14 - Epoch reports pre-release access to evaluate GPT-5.4 (March 2026) and Meta's Muse Spark on FrontierMath. floors at 1 by A.1“We had pre-release access to evaluate the model”
GPT-5.4 set a new record on FrontierMath (Epoch AI (Substack), 2026-03-05) tier 3 self (epoch)confirmed Epoch AI on X: pre-release access to Meta's Muse Spark (Epoch AI (X), 2026) tier 3 self (epoch)confirmed2026-09-15
Scope control 3/4
Anchor 3: Evaluator sets scope and can add questions.
evidence: tier 3 (self)
Capped at 3 by epoch.17 (S.6): FrontierMath, the asset on which Epoch evaluates OpenAI and other labs, was commissioned and is owned by OpenAI, which holds the problems and solutions outside the 50-problem holdout.. Floors up to 3 from epoch.08 do not exceed the cap.
- Evaluates any model on FrontierMath at its discretion. floors at 3 by S.5“Epoch AI is free to conduct and publish evaluations of any models using the FrontierMath problem set”
Clarifying the creation and use of the FrontierMath benchmark (Epoch AI, 2025-01-23) tier 3 self (epoch)confirmed2026-09-14 - FrontierMath, the asset on which Epoch evaluates OpenAI and other labs, was commissioned and is owned by OpenAI, which holds the problems and solutions outside the 50-problem holdout. caps at 3 by S.6binding“OpenAI commissioned Epoch AI to produce 300 advanced math problems for AI evaluation”
Clarifying the creation and use of the FrontierMath benchmark (Epoch AI, 2025-01-23) tier 3 self (epoch)confirmed2026-09-15
Publication rights 2/4
Anchor 2: Publishes; lab reviews with broad redaction.
evidence: tier 3 (self)
mechanism: lab-controlled
Capped at 2 by epoch.02 (R.2): Contributors were not told the funder; disclosure came after the fact.. Floors up to 2 from epoch.03 do not exceed the cap.
- Contributors were not told the funder; disclosure came after the fact. caps at 2 by R.2binding“Per our agreement, we needed OpenAI’s permission before publicly disclosing their involvement”
AI benchmarking organization criticized for waiting to disclose funding from OpenAI (TechCrunch, 2025-01-19) tier 4 press (primary)confirmed Clarifying the creation and use of the FrontierMath benchmark (Epoch AI, 2025-01-23) tier 3 self (epoch)confirmed2026-09-14 - After the disclosure failure, created a 50-problem holdout OpenAI cannot see. floors at 2 by R.1“We are finalizing a 50-problem set for which OpenAI will only receive the problem statements and not the solutions”
Clarifying the creation and use of the FrontierMath benchmark (Epoch AI, 2025-01-23) tier 3 self (epoch)confirmed2026-09-14
Method transparency 3/4
Anchor 3: Tasks or code partly open.
evidence: tier 4 (press)
Floored at 3 by epoch.04 (M.1): Rigorous public data on compute, training runs and capability trends.. No admissible signal caps it.
- Rigorous public data on compute, training runs and capability trends. floors at 3 by M.1binding
AI benchmarking organization criticized for waiting to disclose funding from OpenAI (TechCrunch, 2025-01-19) tier 4 press (primary)confirmed2026-09-14 - Publishes a transparency page listing donations above $70K and paying clients. informational (Informational: a funding-transparency page is not evidence about method transparency; no M anchor or rule maps it. Recommend re-dimensioning to F (for) where the disclosure counts toward anchor 4's 'disclosed'.)“Here, we list only donations of $70,000 USD or more”
Transparency (Epoch AI, 2026) tier 3 self (epoch)confirmed2026-09-14
Role incompatibility 2/4
Anchor 2: Consults for labs.
evidence: tier 3 (self)
held at a supportable anchor
floor and cap disagree
Conflict: floor 3 from epoch.10 (X.7) against cap 2 from epoch.16 (X.3); resolved at 2 by X.3: epoch.10's raw floor of 4 is held to 3 by C15/C16 and still exceeds the X.3 cap of 2; X.3 decides because the client list is on Epoch's own page.
Resolution: X.3 decides 2. epoch.10's raw floor of 4 is held to 3 by C15/C16 and still exceeds the X.3 cap of 2; X.3 decides because the client list is on Epoch's own page.
- No commercial products. floors at 4 by X.7
AI benchmarking organization criticized for waiting to disclose funding from OpenAI (TechCrunch, 2025-01-19) tier 4 press (primary)confirmed2026-09-14 - Epoch states it invests part of its funds in semiconductor and AI stocks as part of a diversified portfolio. caps at 3 by X.5“We invest part of our funds in semiconductor and AI stocks as part of a diversified portfolio”
Transparency (Epoch AI, 2026) tier 3 self (epoch)confirmed2026 - Epoch's transparency page lists paid consultation and evaluation work for OpenAI (2024 to 2026), Google DeepMind (2024 to 2026), xAI (2025) and an Anthropic pilot (2024). caps at 2 by X.3binding“OpenAI 2026 FrontierMath Data Audit”
Transparency (Epoch AI, 2026) tier 3 self (epoch)confirmed2026-09-15 - Document request: Which AI stocks Epoch holds and whether they include developers it evaluates.
Values under each evidence policy
| Dimension | Leads included | Retrieved & confirmed | Against interest | Verified spans | Primary only |
|---|---|---|---|---|---|
| Funding | 2 | 2 | 2 | 2 | – |
| Governance | 2 | 2 | 2 | 2 | – |
| Personnel | 2 | 2 | 2 | – | – |
| Access depth (lab-granted) | 1 | 1 | 1 | 1 | – |
| Scope control | 3 | 3 | 3 | 3 | – |
| Publication rights | 2 | 2 | 2 | 2 | – |
| Method transparency | 3 | 3 | 3 | – | – |
| Role incompatibility | 2 | 2 | 2 | 2 | – |
| Score, Lab procurement | – not ranked, 8/8 | – not ranked, 8/8 | – not ranked, 8/8 | – not ranked, 6/8 | – not ranked, 0/8 |
| Score, Regulator or auditor | – not ranked, 8/8 | – not ranked, 8/8 | – not ranked, 8/8 | – not ranked, 6/8 | – not ranked, 0/8 |
| Score, Public trust | – not ranked, 8/8 | – not ranked, 8/8 | – not ranked, 8/8 | – not ranked, 6/8 | – not ranked, 0/8 |
| Score, Equal weights | – not ranked, 8/8 | – not ranked, 8/8 | – not ranked, 8/8 | – not ranked, 6/8 | – not ranked, 0/8 |
Traced money and ties
hop 0: 4 rows; hop 1: 1 row (daf_grant $0.6M); hop 2: 3 rows (grant $24.5M, recommendation $0.2M). Confirmed rows: 4 of 8 Dollar sums: confirmed + unaudited USD rows only; imported figures are not re-derived and never summed. Second hop traced for 7 of 7 sources. Ties: none within two steps recorded. Checked and not found: 0.
Ledger
Money in
- T07 from Coefficient Giving (formerly Open Philanthropy): grant $24.51M, 2021-2025. cumulative confirmedtier 1 indexhttps://epoch.ai/about/transparency
- T34 from OpenAI: contract undisclosed, 2024. OpenAI commissioned FrontierMath problems and owns them; paid consultation client per transparency page confirmedtier 3 selfhttps://epoch.ai/latest/openai-and-frontiermath
- T35 from Google / Google DeepMind: contract undisclosed, 2025-2026. Google DeepMind listed as a paying consultation client confirmedtier 3 selfhttps://epoch.ai/about/transparency
- T36 from Jaan Tallinn: daf_grant $0.60M, 2025-01. via DAF per Epoch transparency page confirmedtier 3 selfhttps://epoch.ai/our-funding
- T74 from Coefficient Giving (formerly Open Philanthropy): grant $8.50M; detail of T06 — not additive, 2025-04. largest single Epoch grant per transparency page unauditedtier 3 selfhttps://epoch.ai/about/transparency
- T77 from Survival and Flourishing Fund: recommendation $0.20M, 2026-05. per Epoch transparency page unauditedtier 3 selfhttps://epoch.ai/about/transparency
- T82 from xAI: contract undisclosed, 2025. model evaluations for xAI, paying client per transparency page unauditedtier 3 selfhttps://epoch.ai/about/transparency
- T83 from Anthropic: contract undisclosed, 2024. pilot engagement per transparency page unauditedtier 3 selfhttps://epoch.ai/about/transparency
Funding graph, focused
Every one-way chain to a lab and to an evaluator at full strength, everything else faded. Hover any name to trace its connections; click a name to open its page.