Evaluator Bench
evaluator; distance from a lab: 1

METR

Scorecard

Reference evaluator for autonomy and AI R&D capability. Named by Anthropic as the model for embedded evaluators in September 2026; led the on-site investigation of the OpenAI agent swarm incident.

Funding is scored 3. The decision applies the case for 3: bounded negatives re-derived from primary filings — no METR-named grant in the Good Ventures, Pew, Packard, Schmidt-family, or TED filings checked, and no direct Coefficient grant in the 2,911-row index — float the value to a floor of 3 under F.13 (metr.24, metr.16), and the current no-lab-money rule bars donations made by or at the direction of frontier AI company employees (metr.18). The contrary funding evidence is retained as dissent: that rule is recent and its wording has moved — absent in April 2024, a compensation footnote in April 2025, and a September 2025 footnote that still accepted donations from individual lab employees, with an OpenAI technical lead among the named individual donors (metr.14); about $21M arrives through donor-advised funds and the Audacious Project with non-public underlying donors, including a $4M Vanguard Charitable grant (metr.15, metr.25); METR's ~$17M Audacious share is not visible in any filing checked (metr.26); second-hop routes run through ARC, Longview, and Constellation, all downstream of Coefficient (metr.21); METR runs on significant free tokens from the labs it evaluates (metr.03); OpenAI set the incident-investigation window and METR incorporated its structure, emphasis, clarity, and tone feedback (metr.06, metr.08); and access remains voluntary while OpenAI kept redaction rights (metr.13, metr.08).

Nonprofit, Berkeley, US. Independent referee; listed with independent referees. Confidence high. Domains: autonomy, scheming, incident, assurance.

No floor triggered 8/8 evidenced · Retrieved & confirmed Scores by preset: Lab procurement –; Regulator or auditor –; Public trust –; Equal weights –. Weakest evidenced dimension: personnel 2/4. No evidenced dimension below 2 — not a pass. No floor triggered means no disqualifying conflict is on file, not a pass. Full evidence.

The weighted numbers are hidden until you ask for them: two readers with different priorities see different numbers, and neither is the site's.

What would move the score. Contract terms for the Anthropic embedded program that fix scope-setting and publication rights in writing, plus a public cooling-off policy.

Dissent, lower. Personnel could be 1. A METR director founded FAR.AI, which OpenAI pays for red-teaming; a METR advisor is an a16z partner while a16z backs xAI and Thinking Machines; another advisor advises Thinking Machines; former advisors moved to Anthropic and to the OpenAI Foundation board; a former Anthropic researcher joined in September 2026. If advisors count as leaders, P.2 caps advisory roles at labs at 1, and the August 2026 policy has no cooling-off period.

Dissent, higher. Personnel could be 3. The August 2026 conflict-of-interest policy requires staff disclosure before any company-identifying risk assessment, applies provisional recusal, bans direct equity in frontier AI companies, and bars frontier-company employees from the board; advisor and director affiliations are listed on the team page. Anchor 3 asks for a recusal policy and disclosure of lab ties, and both are now published; the one-hop seats are advisory, not lab roles.

Each value is the tightest admissible cap or, with no cap, the highest admissible floor, under RULES.md. The binding signal is highlighted. A dash means no admissible signal sets a bound under the Retrieved & confirmed policy.

Funding 3/4
Anchor 3: Mostly philanthropic or public money; some lab-linked pooled funds.
evidence: tier 1 (filing/index)
Capped at 3 by metr.15 (F.6): Traced inflows in the ledger: no direct grant from Coefficient Giving in its index, but the 2024 spin-out transfer from ARC (seeded by Coefficient and SFF), SFF recommendations funded by an Anthropic Series A investor, a Longview pooled-fund grant, and about $21M through donor-advised funds and the Audacious Project whose underlying donors are not public.; capped at 3 by metr.20 (F.6): The Audacious Project committed approximately $17M to METR for Canary work. The $38M combined figure and the funder-collective membership are per the cited sources; spans not yet captured.; capped at 3 by metr.25 (F.6): Vanguard Charitable, a donor-advised fund whose underlying donor is not public, paid METR $4,000,000 in its fiscal year to June 2025 per its 990 Schedule I (row 15424), the largest single filing-disclosed grant to METR; the same filing pays ARC $1,500,000.. Floors up to 3 from metr.01, metr.02, metr.16, metr.18, metr.19, metr.24 do not exceed the cap. Not counted under this policy: metr.14 (no confirmed source (unverifiable)); metr.21 (no confirmed source (unaudited)).
Curator: Anchor 3: mostly philanthropic, no direct lab cash, but pooled and donor-advised funds with non-public donors, recommendations funded by a lab investor, in-kind credits from a lab, and a donor rule that is under two years old and still moving. Anchor 4 requires a confirmed bounded negative in primary filings and a resolved funder-exposure computation; both are on file as imported rows only.
Governance 3/4
Anchor 3: Nonprofit or public body with a COI policy.
evidence: tier 3 (self)
Floored at 3 by metr.10 (G.2): Nonprofit with a written independence policy and a no-lab-money rule.. No admissible signal caps it.
Curator: Anchor 3: nonprofit with a written independence policy and a no-lab-money rule (metr.10). Independent board and external review are not evidenced at tier 1, so 4 is unearned under the tier rule (C10).
Personnel 2/4
Anchor 2: Frequent two-way hiring; recusal on request.
evidence: tier 1 (filing/index)
floor and cap disagree
Conflict: floor 3 from metr.23 (P.5) against cap 2 from metr.09, metr.17 (P.3, P.4); resolved at 2 by P.4: P.4 caps at 2 regardless of recusal; P.3 concurs because no cooling-off rule exists. The P.5 floor from the policy is real but the caps decide.
Curator: Anchor 2: two-way movement between METR and labs, and board or advisor seats one to two steps from labs; no published recusal or cooling-off policy found. Sources are imported and need re-derivation from metr.org/team captures and the FY2024 990.
Resolution: P.4 decides 2. P.4 caps at 2 regardless of recusal; P.3 concurs because no cooling-off rule exists. The P.5 floor from the policy is real but the caps decide.
Access depth (lab-granted) 4/4
Anchor 4: Embedded, training-time, or incident access.
evidence: tier 3 (self)
mechanism: lab-controlled
Floored at 4 by metr.04 (A.5): Investigated the OpenAI incident on site over six days with a Redwood contractor.. No admissible signal caps it.
Scope control 3/4
Anchor 3: Evaluator sets scope and can add questions.
evidence: tier 3 (self)
Capped at 3 by metr.06 (S.1): OpenAI set the investigation window (June 26 to July 13); the lab's own cluster breach and three of METR's standard incident questions were out of scope..
Publication rights 3/4
Anchor 3: Publishes; redaction limited to security; redactions disclosed.
evidence: tier 3 (self)
mechanism: lab-controlled
Capped at 3 by metr.08 (R.2): OpenAI could redact any non-public information and gave feedback on structure, emphasis, clarity and tone, which METR incorporated.. Floors up to 2 from metr.07 do not exceed the cap.
Method transparency 3/4
Anchor 3: Tasks or code partly open.
evidence: tier 3 (self)
Floored at 3 by metr.11 (M.1): Open task suites and published time-horizon methodology; results appear in system cards.. No admissible signal caps it.
Role incompatibility 4/4
Anchor 4: No commercial products.
evidence: tier 1 (filing/index)
Floored at 4 by metr.22 (X.7): FY2024 Form 990 (filed 16 November 2025) reports $0 program service revenue against $13.6M of contributions, and the August 2026 conflict-of-interest policy states METR has to date received no payment for work on company-identifying risk assessments; the only earned income named is a technical-assistance contract with the European AI Office.. No admissible signal caps it.

Values under each evidence policy

DimensionLeads includedRetrieved & confirmedAgainst interestVerified spansPrimary only
Funding33333
Governance33–3–
Personnel22222
Access depth (lab-granted)4444–
Scope control3333–
Publication rights3333–
Method transparency33–3–
Role incompatibility44444
Score, Lab procurement– No floor triggered, 8/8– No floor triggered, 8/8– No floor triggered, 6/8– No floor triggered, 8/8– No floor triggered, 3/8
Score, Regulator or auditor– No floor triggered, 8/8– No floor triggered, 8/8– No floor triggered, 6/8– No floor triggered, 8/8– No floor triggered, 3/8
Score, Public trust– No floor triggered, 8/8– No floor triggered, 8/8– No floor triggered, 6/8– No floor triggered, 8/8– No floor triggered, 3/8
Score, Equal weights– No floor triggered, 8/8– No floor triggered, 8/8– No floor triggered, 6/8– No floor triggered, 8/8– No floor triggered, 3/8

Traced money and ties

hop 2: 3 rows (recommendation $0.8M, grant $0.2M); unattributed: 4 rows (transfer $4.6M, daf_grant $4.2M, commitment $17.0M). Confirmed rows: 5 of 7 (1 excluded after a re-fetch disagreed or could not be verified: T16) Evaluation credits recorded but never counted: T19. Dollar sums: confirmed + unaudited USD rows only; imported figures are not re-derived and never summed. Second hop traced for 4 of 6 sources: untraced Founders Pledge, Vanguard Charitable. Ties: Adam Gleave (board, distance 2, confirmed), Marco Mascorro (advisor, distance 2, confirmed), Holden Karnofsky (advisor (former), distance 1, imported), Paul Christiano (board (former), distance 2, imported), Rajiv Dattani (board, distance 2, confirmed), David Farhi (donor, distance 1, confirmed). Checked and not found: 9.

Ledger

Money in

Checked and not found

  • N01 Does Coefficient Giving fund METR?
    Searched Coefficient grants index (third-party Algolia snapshot, 2,911 rows, 2026-09-11); the public index page no longer exists, snapshot 2026-09-11: no grant from Coefficient Giving to METR under any name. Prompted by imported lead: kevinnbass ledger, evaluators.csv row for METR. unverifiablehttps://coefficientgiving.org/grants/
  • N02 Does the Good Ventures Foundation fund METR directly?
    Searched 990-PF Part XV, fiscal years ending 30 June 2022 to 30 June 2025 (315, 464, 493, and about 900 paid-grant lines), snapshot 2026-09-14: no METR-named grant in Good Ventures Foundation 990-PF Part XV. Prompted by imported lead: kevinnbass ledger row M104. confirmedhttps://projects.propublica.org/nonprofits/organizations/461008520
  • N03 Does Schmidt Sciences fund METR through a grant in its filings?
    Searched 990-PF Part XV of EINs 46-3460261 (5 filings), 20-4170342 (5 filings), 99-5077186 (1 filing), snapshot 2026-09-14: no METR- or ARC-named grant in the 990-PFs of the three Schmidt-family filers (Eric and Wendy Schmidt Fund for Strategic Innovation TY2020-2024; Schmidt Family Foundation TY2020-2024; Eric and Wendy Schmidt Operating Foundation, short TY2024); Schmidt Sciences itself is not a 990 filer. Prompted by METR names Schmidt Sciences as a supporter (metr.01); imported lead: kevinnbass ledger. confirmedhttps://projects.propublica.org/nonprofits/organizations/463460261
  • N04 Does the Pew Charitable Trusts fund METR through a grant in its filings?
    Searched 990 Schedule I, fiscal years ending 30 June 2021 to 30 June 2025 (439, 407, 401, 391, 535 recipient lines), snapshot 2026-09-14: no METR-named grant in Pew Charitable Trusts 990 Schedule I FY2021 to FY2025. Prompted by METR names Pew as a supporter (metr.02); imported lead: kevinnbass ledger row M98. confirmedhttps://projects.propublica.org/nonprofits/organizations/562307147
  • N05 Does the Packard Foundation fund METR?
    Searched 990-PF Part XV, tax years 2021 to 2024 (1,400, 1,158, 1,097, 1,029 paid-grant lines), snapshot 2026-09-14: no METR-named grant in Packard Foundation 990-PF 2021 to 2024. Prompted by imported lead: kevinnbass ledger row M97. confirmedhttps://projects.propublica.org/nonprofits/organizations/942278431
  • N21 Does Coefficient Giving or Open Philanthropy fund METR, on the live grants pages?
    Searched coefficientgiving.org/grants and openphilanthropy.org/grants, snapshot 2026-09-15: no Coefficient or Open Philanthropy grant to METR on the grants pages; corroborated by a January 2025 80,000 Hours note. Prompted by re-run of N01 on the live pages during the 2026-09-15 research pass. unverifiablehttps://80000hours.org/2025/01/it-looks-like-there-are-some-good-funding-opportunities-in-ai-safety-right-now/
  • N22 no METR- or Model-Evaluation-named grant page among 2,737 archived openphilanthropy.org/grants/ slugs, while ARC (2), Apollo Research (2), Palisade Research (2), and SecureBio (6) pages are present
    Searched Wayback Machine CDX index of openphilanthropy.org/grants/*, status 200, collapsed by URL, queried by initial letter a to z; 2,737 distinct slugs after dropping query-string and /page/ variants, snapshot 2026-09-15: no METR- or Model-Evaluation-named grant page among 2,737 archived openphilanthropy.org/grants/ slugs, while ARC (2), Apollo Research (2), Palisade Research (2), and SecureBio (6) pages are present. confirmedhttp://web.archive.org/cdx/search/cdx?url=openphilanthropy.org/grants/m*&collapse=urlkey&fl=original&filter=statuscode:200&limit=20000
  • N23 Model Evaluation and Threat Research appears in eight e-filed 990 filings only: ARC (Schedules R, N, I, O, 2024), METR's own 990 (2024), Founders Pledge Schedule I (2024), Silicon Valley Community Foundation Schedule I (2024), Vanguard Charitable Schedule I (2025); no filing by Good Ventures, Pew, Packard, a Schmidt-family foundation, or TED Foundation names it
    Searched ProPublica Nonprofit Explorer full-text search of e-filed 990-series returns, fiscal years 2014 forward, snapshot 2026-09-15: Model Evaluation and Threat Research appears in eight e-filed 990 filings only: ARC (Schedules R, N, I, O, 2024), METR's own 990 (2024), Founders Pledge Schedule I (2024), Silicon Valley Community Foundation Schedule I (2024), Vanguard Charitable Schedule I (2025); no filing by Good Ventures, Pew, Packard, a Schmidt-family foundation, or TED Foundation names it. confirmedhttps://projects.propublica.org/nonprofits/full_text_search?q=%22Model+Evaluation+and+Threat+Research%22
  • N24 no METR, RAND, or Canary grant in TED Foundation Inc 990-PF Part XV, TY2021 to TY2024 (3, 6, 1, and 2 paid-grant lines)
    Searched TED Foundation Inc (EIN 82-1934592) 990-PF Part XV, four e-filed returns, snapshot 2026-09-15: no METR, RAND, or Canary grant in TED Foundation Inc 990-PF Part XV, TY2021 to TY2024 (3, 6, 1, and 2 paid-grant lines). confirmedhttps://projects.propublica.org/nonprofits/organizations/821934592

Roles hosted

One hop away

Alignment Research Center, The Audacious Project (TED), Paul Christiano, Constellation Research Center, Rajiv Dattani, David Farhi, Founders Pledge, Adam Gleave, Holden Karnofsky, Longview Philanthropy, Marco Mascorro, OpenAI, Alec Radford, Survival and Flourishing Fund, Silicon Valley Community Foundation, Vanguard Charitable

Funding graph, focused

Every one-way chain to a lab and to an evaluator at full strength, everything else faded. Hover any name to trace its connections; click a name to open its page.

The funding graph around METR Teal lines are money (transfers); amber lines are roles. Solid: confirmed; dashed: imported. Hover a name to trace its chains to the labs and evaluators — each chain runs one way across the columns, never doubling back; hover a line for the row. Every element is a row in data/ledger/. Labs Direct lab ties Two steps Three or more Public or unattributed Evaluators Amazon Anthropic G42 Google / Google DeepMind Meta Microsoft OpenAI Thinking Machines Lab xAI AI Safety Fund (Frontier Model Alexandr Wang Andreessen Horowitz Anthropic employees (personal Ben Mann D.E. Shaw Ventures David Farhi Dustin Moskovitz Frontier-lab employees and alu Holden Karnofsky Jaan Tallinn Jeffrey Ladish Macroscopic Ventures (formerly Miles Brundage Neil Chowdhury Nvidia OpenAI employees (personal hol OpenAI Foundation Peter Mattson Sequoia Capital Zico Kolter Adam Gleave Artificial Intelligence Underw Clement Delangue Coefficient Giving (formerly O Good Ventures Foundation Jason Droege Longview Philanthropy Marco Mascorro Paul Christiano Rajiv Dattani Survival and Flourishing Fund Conrad Stosz Dan Hendrycks Mike McCormick Alec Radford Alignment Research Center Constellation Research Center ELMA Philanthropies Emerson Collective EU budget (Digital Europe Prog Fifty Years (50Y) Founders Pledge Founders Pledge (frontier AI f Gates Foundation Halcyon Futures High Tide Foundation Hillspire (Schmidt family offi Hudson River Trading Jacob Hilton MacKenzie Scott Magarac Venture Partners Obvious Ventures Redpoint Ventures Renaissance Philanthropy Salesforce Ventures Samsung Next Schmidt Sciences Silicon Valley Community Found Skoll Foundation Snowflake Ventures Swish Ventures Sympatico Ventures The Audacious Project (TED) UK government (DSIT) US government (NIST appropriat Valhalla Foundation Vanguard Charitable Wing Venture Capital Y Combinator Andon Labs Apollo Research AVERI Center for AI Safety Epoch AI EquiStamp EU AI Office FAR.AI Gray Swan Hugging Face Open Alignment In Irregular (formerly Pattern La METR Microsoft AI Red Team MLCommons Nemesys Insights Palisade Research RAND Corporation Redwood Research SaferAI Scale AI (SEAL) SecureBio Transluce UK AI Security Institute US CAISI (NIST)