Population: who is in, who is out
An organization is in scope if, within the 24 months to the evidence clock, it was cited as an external evaluator or red team in a frontier system card or a government evaluation report; or it is named in a statute, code, or standard as an evaluator; or it operates a leaderboard a frontier developer cites for a frontier model (RULES 11). Candidates that met a criterion but are not yet scored are listed as in-scope candidates for the next batch. Nothing here is a judgment about quality. The directory approximates the organizations demonstrably participating in the institutional frontier-AI evaluation ecosystem — not all credible frontier-AI evaluators. The excluded and unclear lists are part of the finding, not housekeeping. A hypothetical composite is never scored.
45 candidates checked: 14 in-scope-candidate, 23 out, 8 unclear. Every entry names the URL that decided it.
| Candidate | Type | Result | Criteria met | Reason | Checked |
|---|---|---|---|---|---|
| Artificial Analysis | private | out | a, c | Out by scope (D-001): a capability or performance leaderboard, not a safety evaluator. Cited by Anthropic and xAI as an independent third-party evaluator in system/model cards, and its index is cited by xAI and Amazon; capital structure not verified. source | 2026-09-15 |
| LM Arena (Arena, formerly LMArena / LMSYS Chatbot Arena) | vc | out | c | Out by scope (D-001): a capability or performance leaderboard, not a safety evaluator. Operates a leaderboard that a frontier developer (xAI) cites for a frontier model; funding rounds reported in search snippets were not verified from an opened page. source | 2026-09-15 |
| Stanford HELM / CRFM | academic | unclear | none | Leaderboard exists and covers frontier models, but no frontier-developer citation of HELM for a frontier model was found within the window; not exhaustive. source | 2026-09-15 |
| EleutherAI (lm-evaluation-harness) | nonprofit | unclear | none | Evaluation framework rather than an operated leaderboard; no frontier developer found citing it for a frontier model in the window, but the search space (model cards on Hugging Face) was not exhausted. source | 2026-09-15 |
| Vals AI | private | out | a, c | Out by scope (D-001): a capability or performance leaderboard, not a safety evaluator. Named as an external evaluator in Anthropic and xAI cards and operates leaderboards both cite; capital structure not verified. source | 2026-09-15 |
| Patronus AI | vc | out | none | No system-card, government-report, statutory, or developer-cited-leaderboard evidence found. source | 2026-09-15 |
| Haize Labs | vc | unclear | none | Publicly credited by a frontier developer as an external red team of a safeguard prototype within the window, but not in a system or model card; curator must decide whether a research post satisfies criterion (a). source | 2026-09-15 |
| Lakera (Check Point) | private | out | none | No evidence of being cited as an external evaluator or red team, or of a developer-cited leaderboard. source | 2026-09-15 |
| Promptfoo | bigtech | out | none | Tool used internally by labs, never cited as an external evaluator; since March 2026 a unit of a frontier developer (OpenAI). source | 2026-09-15 |
| Giskard (Phare) | vc | unclear | none | Operates a leaderboard covering frontier models and has a research tie to Google DeepMind (self-reported), but no developer citation of Phare for a frontier model was found. source | 2026-09-15 |
| Trail of Bits | private | out | none | Documented roles are vulnerability triage, disclosure and OSS patching for labs, not evaluation or red-teaming of a frontier model; revisit if a card names it as an evaluator. source | 2026-09-15 |
| NCC Group | private | out | none | No citation as an external evaluator or red team of a frontier model found. source | 2026-09-15 |
| Apart Research | nonprofit | unclear | none | Self-reported EU AI Office collaboration on harmful manipulation needs a primary EU document (contract award or report) before criterion (a) or (b) can be applied. source | 2026-09-15 |
| Virtue AI | vc | out | none | No citation as an external evaluator or red team of a frontier model found; founders reportedly now inside a frontier developer (unverified from an opened page). source | 2026-09-15 |
| Japan AI Safety Institute (J-AISI, IPA) | gov | in-scope-candidate | a | National institute that co-authored government evaluation reports of frontier models (models anonymised) and holds an evaluation-methods MoC with Anthropic; no pre-deployment system-card citation found. source | 2026-09-15 |
| Singapore AI Safety Institute (DTC/IMDA; AI Verify Foundation) | gov | in-scope-candidate | a | National institute that led government joint-testing reports evaluating (unnamed) frontier models; AI Verify Foundation's separate role was not evidenced from an opened page. source | 2026-09-15 |
| Korea AI Safety Institute (KAISI) | gov | in-scope-candidate | a | National institute that co-authored government evaluation reports and published its own (anonymised) multi-model safety assessment; a primary KAISI page (aisi.re.kr) was not opened. source | 2026-09-15 |
| Canadian AI Safety Institute (CAISI Canada / NRC) | gov | in-scope-candidate | a | Participated as an evaluator in the Network's government joint-testing reports on frontier models; no model-specific or pre-deployment report found. source | 2026-09-15 |
| INESIA (France: ANSSI, Inria, LNE, PEReN) | gov | in-scope-candidate | a | National institute whose constituent bodies ran and published joint evaluations of frontier models (unnamed) in July 2025. source | 2026-09-15 |
| Germany: DE-AISI nucleus (BSI and BNetzA) | gov | out | none | Expected entrant: institute created June 2026 with no published evaluation of a frontier model and no citation as an evaluator. source | 2026-09-15 |
| Australia AI Safety Institute | gov | unclear | none | Australia contributed to a government joint-testing report, but the AISI itself post-dates it and no AISI-authored evaluation or primary statement of frontier testing was opened. source | 2026-09-15 |
| OpenAI Preparedness team | bigtech | out | none | First-party unit of a frontier developer; not cited as an external evaluator in any system card or government report found (Microsoft AI Red Team is in the directory only because OpenAI's card names it as external). source | 2026-09-15 |
| Anthropic Frontier Red Team | bigtech | out | none | First-party unit of a frontier developer; the one cross-lab evaluation was published as a blog post, not cited in a system card. source | 2026-09-15 |
| Google DeepMind Frontier Safety team | bigtech | out | none | First-party unit of a frontier developer with no external-evaluator citation found. source | 2026-09-15 |
| Arcadia Impact | nonprofit | unclear | none | Builds the evaluation suite UK AISI uses and self-reports collaboration with AISI, but no government evaluation report or system card was found citing it as an evaluator. source | 2026-09-15 |
| Leap Labs | vc | out | none | No evidence of any inclusion criterion. source | 2026-09-15 |
| Robust Intelligence (Cisco AI Defense) | bigtech | out | none | Product unit of a non-frontier Big Tech firm with no evaluator citation found. source | 2026-09-15 |
| Pillar Security | vc | out | none | No evidence of any inclusion criterion. source | 2026-09-15 |
| Mindgard | vc | out | none | Self-initiated vulnerability research only; no citation as an external evaluator or red team. source | 2026-09-15 |
| SPLX (SplxAI) | private | out | none | Self-published red-team reports are not citations by a developer or government; no criterion met. source | 2026-09-15 |
| Collective Intelligence Project (CIP) | nonprofit | out | none | Public-input and deliberation organisation; no evaluator citation inside the 24-month window. source | 2026-09-15 |
| Deloitte Consulting (formerly Gryphon Scientific) | private | in-scope-candidate | a | Named bio-risk evaluator in OpenAI, Anthropic and Amazon cards. Note the directory's big4 entry is a hypothetical composite of assurance firms (present when this pass started; a staged deletion now sits in the working tree, not made by this pass); a Deloitte/Gryphon entry would be a real, cited organisation. source | 2026-09-15 |
| Signature Science | private | in-scope-candidate | a | Named as a co-developer of Anthropic's CBRN evaluations in four system cards. source | 2026-09-15 |
| Faculty.ai | private | in-scope-candidate | a | Named external evaluation partner in an Anthropic system card. source | 2026-09-15 |
| Dyno Therapeutics | vc | in-scope-candidate | a | Named biology evaluation partner in four Anthropic system cards. source | 2026-09-15 |
| Trajectory Labs, PBC | pbc | in-scope-candidate | a | Contracted external red team named in two Anthropic system cards. source | 2026-09-15 |
| 10a Labs | unknown | in-scope-candidate | a | Contracted external red team named in an Anthropic system card; organisation not otherwise researched. source | 2026-09-15 |
| Eleos AI Research (Frontier Ethics Research Institute) | nonprofit | in-scope-candidate | a | Named external assessor in an Anthropic system card, though for model-welfare rather than safety evaluation. source | 2026-09-15 |
| ActiveFence | vc | in-scope-candidate | a | Named external red team in an Amazon model card/technical report. source | 2026-09-15 |
| Gomes Group, Carnegie Mellon University | academic | in-scope-candidate | a | Named external chemistry red team in an Amazon model card/technical report. source | 2026-09-15 |
| ARC Prize Foundation (ARC-AGI) | nonprofit | out | c | Out by scope (D-001): a capability or performance leaderboard, not a safety evaluator. Operates a verified leaderboard that Anthropic and Google cite for frontier models. source | 2026-09-15 |
| Terminal-Bench (Stanford / Laude Institute / Harbor) | academic | out | c | Out by scope (D-001): a capability or performance leaderboard, not a safety evaluator. Benchmark and leaderboard cited by Anthropic and xAI for frontier models. source | 2026-09-15 |
| Mercor (APEX benchmarks) | vc | out | a, c | Out by scope (D-001): a capability or performance leaderboard, not a safety evaluator. Named as an external evaluator by xAI and operates leaderboards xAI cites. source | 2026-09-15 |
| Harvey AI (Legal Agent Benchmark) | vc | out | c | Out by scope (D-001): a capability or performance leaderboard, not a safety evaluator. Benchmark owner cited by xAI and Anthropic; a legal-AI vendor and lab customer rather than an evaluator, so role classification would be vendor. source | 2026-09-15 |
| Vector Institute | nonprofit | unclear | none | Tooling contributor to UK AISI and Canada's institute; no evaluator citation found. source | 2026-09-15 |