MLCommons
Scorecard
Industry consortium producing the AILuminate safety benchmark.
Consortium, San Francisco, US. Benchmark or consortium; listed with commercial and first-party. Confidence med. Domains: benchmarks, assurance.
Conditional floor 8/8 evidenced · Retrieved & confirmed Scores by preset: Lab procurement –; Regulator or auditor –; Public trust –; Equal weights –. Weakest evidenced dimension: access depth (lab-granted) 0/4. At least one evidenced conflict dimension (funding, governance, personnel, role incompatibility, scope, publication) at 1: usable with conditions the card names.
The weighted numbers are hidden until you ask for them: two readers with different priorities see different numbers, and neither is the site's.
What would move the score. Published funding shares and a hazard suite the members do not see in advance.
- Funding 1 to 2: score –, band becomes no floor triggered
- Governance 2 to 3: score –
- Personnel 2 to 3: score –
Dissent, lower. Funding (F) at 1 could be 0. The consortium was convened by Google engineers, its president is a Google senior staff engineer, and Google is a founding participant; the benchmarked labs sit in the working group that writes the benchmark and pay the dues that fund it. Anchor 0 covers an organization controlled by a frontier developer; a member-governed body whose president draws a Google salary is arguably that.
Dissent, higher. Funding (F) at 1 could be 2. More than 125 members and affiliates pay dues, most of them chipmakers, universities and non-profits rather than frontier labs, and no member is documented above a trivial share. The AI Verify Foundation co-funded AILuminate. Lab dues beside a broad disclosed base is the anchor-2 picture of diversified revenue rather than lab revenue as the principal source.
Each value is the tightest admissible cap or, with no cap, the highest admissible floor, under RULES.md. The binding signal is highlighted. A dash means no admissible signal sets a bound under the Retrieved & confirmed policy.
- Funded by member companies, including the labs whose models are benchmarked. caps at 1 by F.5binding“MLCommons is an industry consortium backed by several dozen tech firms”
MLCommons AI Safety / AILuminate (MLCommons, 2026) tier 3 self (mlcommons)confirmed AILuminate (MLCommons, 2026-04-27) tier 3 self (mlcommons)confirmed MLCommons releases new AILuminate benchmark for measuring AI model safety (SiliconANGLE, 2024-12-04) tier 4 press (primary)confirmed2026-09-14
- Consortium governance by members. caps at 2 by G.4binding“MLCommons is supported by over 125 members and affiliates”
MLCommons AI Safety / AILuminate (MLCommons, 2026) tier 3 self (mlcommons)confirmed AILuminate (MLCommons, 2026-04-27) tier 3 self (mlcommons)confirmed Mlcommons Association - Nonprofit Explorer (ProPublica, 2026) tier 1 indexconfirmed2026-09-14
- Working groups staffed by member-company employees. caps at 2 by P.4binding“People from the following organizations have collaborated within the working group”
MLCommons AI Safety / AILuminate (MLCommons, 2026) tier 3 self (mlcommons)confirmed AILuminate (MLCommons, 2026-04-27) tier 3 self (mlcommons)confirmed2026-09-14 - President is a Google senior staff engineer; members include the frontier developers whose models are benchmarked. caps at 2 by P.4“Peter Mattson is a senior staff engineer at Google. He founded and is President of MLCommons”
evaluators.csv (24 evaluators: Coefficient and SFF totals, lab money, government contracts, leadership) (Kevin Bass (GitHub), 2026-09-14) tier 2 ledgerimported Leadership (MLCommons, 2026) tier 3 self (mlcommons)confirmed2026-09-11 - Document request: Request from MLCommons the board conflict-of-interest and recusal policy that applies to directors employed by member companies whose systems are benchmarked, including any recusal recorded for the President on AILuminate grading decisions.
- Used as neutral ground in the AVERI double-blind pilot. floors at 0 by A.1“AVERI, OpenMined, and MLCommons were not able to see the model’s weights”
AVERI Pilot Report: The World's First Double-Blind Evaluation of a Proprietary Language Model (AVERI, 2026-08) tier 3 self (averi)confirmed2026-09-14 - The only documented engagement evaluated a publicly released model inside an enclave; no pre-release access from a lab is documented. caps at 0 by A.1binding“AVERI, OpenMined, and MLCommons were not able to see the model’s weights”
AVERI Pilot Report: The World's First Double-Blind Evaluation of a Proprietary Language Model (AVERI, 2026-08) tier 3 self (averi)confirmed2026-09-15
- Benchmark design set by working group. floors at 3 by S.5binding“designed and developed by the MLCommons AI Risk and Reliability working group”
MLCommons AI Safety / AILuminate (MLCommons, 2026) tier 3 self (mlcommons)confirmed MLCommons Releases AILuminate Creative Commons DEMO Benchmark Prompt Dataset to Github (MLCommons, 2025-01-15) tier 3 self (mlcommons)confirmed2026-09-14
- Publishes results. floors at 2 by R.7binding“Public reports of 13 systems-under-test (SUTS)”
MLCommons AI Safety / AILuminate (MLCommons, 2026) tier 3 self (mlcommons)confirmed MLCommons Releases AILuminate Creative Commons DEMO Benchmark Prompt Dataset to Github (MLCommons, 2025-01-15) tier 3 self (mlcommons)confirmed2026-09-14
- Open benchmark specification and methodology. floors at 3 by M.1binding“The Official Test dataset is private to avoid SUTs “gaming” the benchmark”
MLCommons AI Safety / AILuminate (MLCommons, 2026) tier 3 self (mlcommons)confirmed AVERI Pilot Report: The World's First Double-Blind Evaluation of a Proprietary Language Model (AVERI, 2026-08) tier 3 self (averi)confirmed MLCommons Releases AILuminate Creative Commons DEMO Benchmark Prompt Dataset to Github (MLCommons, 2025-01-15) tier 3 self (mlcommons)confirmed2026-09-14
- Benchmarks free to use. floors at 3 by X.4binding“publicly released the AILuminate DEMO prompt dataset, a collection of 1,200 prompts”
MLCommons AI Safety / AILuminate (MLCommons, 2026) tier 3 self (mlcommons)confirmed MLCommons Releases AILuminate Creative Commons DEMO Benchmark Prompt Dataset to Github (MLCommons, 2025-01-15) tier 3 self (mlcommons)confirmed2026-09-14
Values under each evidence policy
| Dimension | Leads included | Retrieved & confirmed | Against interest | Verified spans | Primary only |
|---|---|---|---|---|---|
| Funding | 1 | 1 | 1 | 1 | – |
| Governance | 2 | 2 | 2 | 2 | 2 |
| Personnel | 2 | 2 | 2 | 2 | – |
| Access depth (lab-granted) | 0 | 0 | 0 | 0 | – |
| Scope control | 3 | 3 | – | 3 | – |
| Publication rights | 2 | 2 | – | 2 | – |
| Method transparency | 3 | 3 | 3 | 3 | – |
| Role incompatibility | 3 | 3 | – | 3 | – |
| Score, Lab procurement | – Conditional floor, 8/8 | – Conditional floor, 8/8 | – Conditional floor, 5/8 | – Conditional floor, 8/8 | – No floor triggered, 1/8 |
| Score, Regulator or auditor | – Conditional floor, 8/8 | – Conditional floor, 8/8 | – Conditional floor, 5/8 | – Conditional floor, 8/8 | – No floor triggered, 1/8 |
| Score, Public trust | – Conditional floor, 8/8 | – Conditional floor, 8/8 | – Conditional floor, 5/8 | – Conditional floor, 8/8 | – No floor triggered, 1/8 |
| Score, Equal weights | – Conditional floor, 8/8 | – Conditional floor, 8/8 | – Conditional floor, 5/8 | – Conditional floor, 8/8 | – No floor triggered, 1/8 |
Traced money and ties
hop 0: 1 row. Confirmed rows: 1 of 1 Dollar sums: confirmed + unaudited USD rows only; imported figures are not re-derived and never summed. Second hop traced for 1 of 1 sources. Ties: none within two steps recorded. Checked and not found: 1.
Ledger
Money in
- T63 from Google / Google DeepMind: partnership undisclosed, . founding participation; no contribution amount disclosed confirmedtier 3 selfhttps://mlcommons.org/about-us/
Checked and not found
- N13 Does Coefficient Giving fund MLCommons?
Searched Coefficient grants index, 2,911 rows, snapshot 2026-09-11: no grant from Coefficient Giving in its index. Prompted by imported lead: kevinnbass ledger row EV17. importedhttps://coefficientgiving.org/grants/
Roles hosted
- R52 Peter Mattson: principal. president; imported kevinnbass/metr-money-figure:EV17 importedhttps://mlcommons.org/about-us/leadership/
One hop away
Funding graph, focused
Every one-way chain to a lab and to an evaluator at full strength, everything else faded. Hover any name to trace its connections; click a name to open its page.