Nakajima, Y. (2026). Who Pays the Referee? Measuring Third-Party Evaluator Independence in Frontier AI Against the History of Assurance Regimes (v0.1 draft). https://github.com/yoheinakajima/evaluator-bench

Who Pays the Referee? Measuring Third-Party Evaluator Independence in Frontier AI Against the History of Assurance Regimes

Version 0.1 draft, 15 September 2026. All numbers regenerate from this repository at the pinned commit (python -m bench verify && python -m bench build && python -m bench scores lab --by-type && python -m bench industries && python -m bench paths). Every Dataset A value is derived from signals under RULES.md, and the numbers below are read under the retrieved-&-confirmed evidence policy (confirmed sources only) unless stated. Dataset B milestone years are coded from secondary sources and need primary-source verification before submission; Dataset B is exploratory historical context, not an empirical finding. Dataset A is single-coder; a second coder on every extreme is a gate for a citable tag.

Abstract

Frontier AI developers now cite outside evaluators in nearly every model release, and regulators in the European Union and Illinois require "independent" external evaluation — Illinois defines the test, borrowing financial audit's rule that neither party may hold a financial interest in the other. We ask two questions. How independent are the evaluators the field relies on, measured against criteria that other industries eventually wrote into law? And where does frontier AI sit on the path those industries took from voluntary assurance to regulated independence? We answer with two open datasets. Dataset A scores 24 organizations that evaluate frontier models for safety-relevant properties on eight independence dimensions (funding, governance, personnel, access depth, scope control, publication rights, method transparency, role incompatibility), each 0 to 4 with written anchors. Every value is derived, not typed: each signal names the anchor it supports and the published rule that says so, the value is the tightest cap or, failing any cap, the highest floor, and the reader chooses the evidence policy that decides which signals count; the default admits only sources that were re-fetched and confirmed. Independence has floors, so organizations are reported in bands (a disqualifying floor at any 0 on a conflict dimension, a conditional floor at any 1) before they are ranked by number. Everything is rebuilt from an append-only event log; one expected entrant is scored on the same rubric on a watchlist, excluded from rankings and every statistic. Dataset B codes the lifecycle of sixteen assurance regimes across fifteen industries as dated milestones: voluntary assurance, trigger incidents, mandates, standards, oversight of assurers, independence rules, access and publication rules. Three results. First, in the coded record, rules arrive within a few years of public failures while the drift before the first failure spans decades — a pattern consistent with responsiveness, not proof of it, since the triggers were selected with hindsight. Second, the party being assessed almost always keeps paying the assessor; independence arrived through standards, inspection of assessors, rotation, liability, and public ratings, and the two regimes that removed the payer conflict did so by routing payment through insurers or funding the assessor from a party with opposing incentives. Third, under the retrieved-&-confirmed evidence policy the 2026 frontier AI safety-evaluator population is strongest on scope control and method transparency and weakest on governance, access, and personnel; 13 of 24 organizations trigger no floor, 8 carry a conditional floor on a conflict dimension, and 3 a disqualifying floor; and frontier AI has reached five of seven lifecycle stages in four years while skipping the two every mature regime built: oversight of the evaluators themselves, and standalone independence rules.

1. Introduction

In September 2026 the chief executive of Anthropic committed, in an essay and an announcement on the same day, to giving a team of embedded third-party evaluators, naming METR as the example, ongoing employee-like access to Anthropic's systems and the right to publish key findings without editorial control, with redaction limited to what cannot be published for security reasons. Within hours the chief executive of OpenAI wrote that committing to independent evaluators with employee-like access was a good idea and that OpenAI would do the same; that statement does not name an evaluator and does not mention publication rights. Neither statement says who will pay the evaluators, who will select them, or what happens when an evaluator and a lab disagree about what counts as security-sensitive. A month earlier, Illinois enacted a statutory audit mandate for frontier developers, borrowing its independence test from financial audit: neither party may hold a financial interest in the other. The European Union's AI Act framework, supported by a voluntary Code of Practice, sets an access floor of twenty business days and minimally guardrailed model versions for external evaluation of systemic-risk models.

The organizations these instruments point at are few. The same six or seven groups appear in the system cards of OpenAI, Anthropic, and Google DeepMind. Some are nonprofits that refuse lab money; some are venture-backed companies whose investors also back the labs; one is a business unit of a frontier developer; one is a founder's side project whose chief scientist chairs a lab's safety committee. We are aware of no public scored, sourced account of them.

There is a long literature arguing that AI should learn from the assurance regimes of other industries (Raji et al. 2022; Manheim et al. 2025; Brundage et al. 2026). That argument is made. What is missing is measurement on both sides of the analogy: a coded reading of the AI evaluator population against the criteria other industries adopted, and a coded history of when and why those industries adopted them. This paper supplies both and reports what the comparison says.

Contributions:

1. Dataset A, an evidence-linked scorecard of 24 frontier AI safety evaluators on eight independence dimensions, built as an event-sourced graph so every score traces to its evidence, with a published rule set that turns each signal into a bound and a reader-selectable evidence policy that decides which signals count. One expected entrant is scored on the same rubric on an unranked watchlist. Signal and source counts at the pinned commit are on the status page.
2. Dataset B, a lifecycle dataset of sixteen assurance regimes coded as dated milestones of eleven kinds, with per-regime payer, access, and publication models.
3. A stage ladder that places frontier AI on the trajectory other regimes followed, and three empirical regularities about how assurance regimes acquire independence.
4. A reading of what those regularities imply for the design choices now open in AI.

2. Related work

Frameworks for third-party evaluation of AI systems have proliferated: audit ecosystem design (Raji et al. 2022), the ASPIRE criteria for external scrutiny (Anderljung et al. 2023), access taxonomies (Casper et al. 2024; Charnock et al. 2026), compliance reviews of safety frameworks (Homewood et al. 2025), and AI Assurance Levels (Brundage et al. 2026). The AI Evaluator Forum's AEF-1 standard specifies minimum operating conditions for independent evaluation. Empirical work on auditors exists for algorithmic bias audits: a field scan and practitioner survey (Costanza-Chock et al. 2022) and a coded analysis of 116 published New York City hiring-tool audits (Groves et al. 2025). Rating efforts score developers' safety frameworks (SaferAI; the Future of Life Institute index), not evaluators. Cross-industry precedent appears in most of these papers as illustration; the systematic search closest in spirit (Anderson-Samways 2024) catalogs US regulatory instruments rather than assurance-provider lifecycles. We are aware of no prior work that codes the frontier evaluator population against independence criteria with sourced evidence, and none that codes assurance-regime lifecycles as comparable dated events. A fuller annotated bibliography is in the repository.

3. Datasets

3.1 Dataset A: the evaluator scorecard

Population. The scope is third-party evaluation of frontier models for safety-relevant properties: dangerous capabilities, misuse, alignment and scheming, security, safeguards, and incident investigation, plus the public and standards bodies with a role in that layer; capability and performance leaderboards are out (RULES.md section 11, decision D-001). Within that scope an organization is included if, within the 24 months to the evidence clock, it was cited as an external evaluator or red team in a frontier system card or a government evaluation report; or it is named in a statute, code, or standard as an evaluator; or it operates a safety benchmark a frontier developer cites for a frontier model. Twenty-four ranked organizations as of v0.1, placed in three lists by a derived role (independent referees; government institutes; commercial and first-party, which holds vendors, lab units, and consortia funded by the labs they benchmark). Roles are derived from legal form and the role-incompatibility value, not typed. One expected entrant, an initiative announced days before the evidence freeze with no evaluations yet, is scored on the same rubric on a watchlist and excluded from rankings and from every statistic. A hypothetical composite of large assurance firms that v0 scored has been removed: a made-up organization cannot be evidenced. Two organizations v0 ranked, Epoch AI and the Princeton Holistic Agent Leaderboard, are retained out of scope as capability benchmarks, with their records unchanged and excluded from every statistic. A population check against the same criteria (data/exclusions.json) recorded 45 further candidates, of which 21 met a criterion and are candidates for the next batch, several of them evaluators named in current system cards (Deloitte, formerly Gryphon Scientific; Signature Science; Faculty.ai; Dyno Therapeutics; Trajectory Labs; Artificial Analysis; Vals AI; the Japanese, Singaporean, Korean, Canadian, and French institutes); 16 were out and 8 unclear.

Dimensions and anchors. Eight dimensions, each scored 0 to 4 against written anchors. The set draws on AEF-1's five operating conditions, on the auditor-independence rules Illinois SB 315 imports from financial audit, and on the Charnock et al. access taxonomy, with two additions the AI literature tends to omit: who owns the evaluator, and whether it sells remediation to the companies it grades. Anchors are in data/dimensions.json. Access is scored as access labs have actually granted, not as an institutional right: a low access score can mean exclusion, not incompetence.

Evidence model. A signal is one dated claim, for or against, on one dimension, citing at least one source with a URL and a retrieval date, and carrying a bound: an against-signal caps the value on its dimension at the anchor it supports, a for-signal floors it, and a rule code names the tie-break in RULES.md that says so. The value on a dimension is the smallest admissible cap or, with no cap, the largest admissible floor; where a floor exceeds a cap the assessment carries a written resolution naming the rule. Evidence status clamps what a bound may set: a signal whose sources are all confirmed may set any bound, one with a confirmed or unaudited source may set caps no lower than 1 and floors no higher than 3, and imported or unverifiable leads support only a 2. A 0 needs a quoted span; a 4 needs a span and a tier-1 or tier-2 source or two independent sources with one not the evaluator's own. Bounds that fail these tests are held at 1 or 3 and the card says which rule held them. Five evidence policies decide admissibility: leads included, standard (at least one confirmed source; the default), against interest (an evaluator's own statements count only against it), verified spans, and primary only (confirmed filings, indexes, and third-party ledgers with a span). A dimension with no admissible signal is unevidenced under that policy, shown as a dash and excluded from the score, with the coverage count beside it. A score is a weighted total over the evidenced dimensions, never stored, recomputed at build time under every policy and preset, and reported with a band. Integrity is enforced by a verifier: no orphan sources, no assessment without a signal, no stored value that disagrees with its derivation, no role that disagrees with the rule, no 4 on funding/governance/personnel without a tier-1 source on a cited signal, no record dated after the frozen evidence clock. The build writes every object into an append-only event log under a frozen clock; two builds on the same data produce a byte-identical log, and the provenance of every score names the events it rests on. Bench-authored docket drafts are not evidence and cannot be cited by signals.

Coding. Version 0.1 is single-coder from public sources between 12 and 15 September 2026, with AI assistance disclosed in the repository, and no second coder claimed; the check is public: every assessment, signal, bound, rule, row and source is published with its status, and corrections arrive through a reviewed contribution path. The rules were written before the bounds were assigned and are applied to every organization the same way; the assessments held at a supportable anchor by the span and second-source tests are counted on the status page at the pinned commit. Confidence tags (high, medium, low) mark thin records. Each assessment carries a computed evidence tier (the best source tier among its cited signals) and, on every card, a dissent: the strongest case from the record that the weakest dimension should be one notch lower, and one notch higher. A human second coder on every extreme is a gate for a citable tag, with agreement reported on the status page; the full population follows in v0.2.

3.2 Dataset B: assurance-regime lifecycles

Sixteen regimes: financial statement audit, credit ratings, electrical and consumer product safety, pharmaceutical safety and efficacy, aircraft certification, nuclear safety and safeguards, ship classification, cybersecurity assurance, boilers and pressure vessels, food safety, automobile crash safety, sustainability reporting assurance, dietary supplements, crypto exchange proof of reserves, platform and hiring-algorithm audits, and frontier AI evaluation. The first eleven are regimes that reached mandated assurance; the next four were added to avoid selecting on success and include one rollback (supplements), one collapse of voluntary assurance (crypto), one slow voluntary period (sustainability), and one young regime that wrote independence rules at the start (platform audits).

Milestones are coded by kind: voluntary_assurance, trigger, mandate, standards, accreditation (oversight of assessors), independence_rule, delegation_reform, payer_reform, access_expansion, publication_rule, delegation, payer_shift, rollback. One instrument is coded as one milestone at its primary operative kind; multi-provision laws are not double-counted across stages. Each regime also records its payer model, access model, and publication model. Year granularity. Every milestone cites a source; v0 uses secondary sources with years spot-checked, and each entry is flagged for primary verification.

4. Method

Instrument strength and trigger criterion. Every rule milestone carries a strength from 1 (disclosure or voluntary text) to 4 (structural change: separation of functions, rotation, a new inspecting body, payment rerouted, delegation reclaimed). Every trigger carries a harm class and must meet a criterion set before looking at what followed: ten or more deaths, losses above one billion dollars, or a documented integrity failure of the assurance itself.

Stage ladder. Seven stages are defined from milestone kinds: S1 voluntary assurance, S2 trigger, S3 mandate, S4 standards, S5 oversight of assessors, S6 independence rules (independence_rule, delegation_reform, payer_reform), S7 access and publication rules. A regime reaches a stage in the year of the first qualifying milestone. Delegation, payer shifts toward the assessed party, and rollbacks are recorded and displayed but do not advance a stage. Accreditation counts as oversight (S5), not as an independence rule (S6).

Lags. For each trigger we compute the years to the next milestone of any rule kind (mandate, standards, accreditation, independence_rule, delegation_reform, payer_reform, publication_rule). For each regime we compute the interval from its first milestone to its first S6 milestone. Same-year pairings are reported at lag 0; year granularity cannot resolve ordering within a year.

Paths. Each regime's milestones are ordered by year and written as a sequence of kinds with consecutive repeats collapsed. Similarity between frontier AI's sequence and each regime's opening is one minus the normalized Levenshtein distance over openings of comparable length.

Mechanisms. Each regime is coded before reform and now on who pays the assessor, who selects it, what it sees, what the public reads, and who oversees the assessor. The coding is categorical and coarse; matches on the full pattern are suggestive, not conclusive.

Scores. Dataset A values are weighted under four presets, each with a written derivation naming the persona and the instrument its weights follow, and reported with the distribution per dimension and per organization type. Independence has floors, not just averages, so organizations are reported in bands before they are ranked by number: an evidenced 0 on a conflict dimension (funding, governance, personnel, role incompatibility, scope control, publication rights) is a disqualifying floor, an evidenced 1 on one of those a conditional floor, otherwise no floor triggered; access depth and method transparency count in the number and never set a band, since a 0 there records what labs have granted or what has been published, not a conflict; the number ranks within a band, and the site hides it until the reader chooses weights. The lab-procurement preset is the confirmatory view: its weights were fixed in the seed script before the population pass but not registered outside this repository, so it is not called pre-registered. The other presets are sensitivity checks. Expectations H1 to H4 were stated in paper/README.md before the population pass; they are stated expectations, not a registration.

Everything above is computed by python -m bench industries, python -m bench timeline, python -m bench paths, and python -m bench scores.

5. Results

5.1 Where frontier AI sits (Figure 1)

Frontier AI reached S1 in 2022 (pre-release testing of GPT-4 by ARC Evals), S2 in 2025 (the FrontierMath funding disclosure) with a second, more serious trigger in 2026 (the OpenAI agent swarm incident and the first on-site third-party incident investigation), S3 in 2026 (Illinois SB 315, audits from 2028, with a financial-interest test embedded in the mandate), S4 in 2025 (AEF-1), and S7 in 2025 (the Code of Practice access floor, followed by the 2026 embedded-evaluator pledges). It has reached neither S5 nor S6: there is no body that accredits, inspects, or registers frontier AI evaluators, and no standalone independence rule — SB 315's financial-interest test exists only inside the mandate and has not been applied to any audit yet.

The empty cells are the informative ones, and they are the result. Eight of the fifteen non-AI regimes built an oversight layer for their assessors (a registry, an accreditation program, or an inspector of inspectors): financial audit (PCAOB, 2002), ship classification (IACS, 1968), boilers (National Board, 1919), product safety (OSHA NRTLs, 1988), pharmaceuticals (Good Laboratory Practice inspection, 1978), credit ratings (NRSRO, 1975), food safety (FSMA accredited certifiers, 2011), and cybersecurity (Common Criteria labs, 1999). AI has standards without anyone checking who applies them, and a mandate whose independence test has never been exercised.

Four years from voluntary assurance to mandate is the shortest interval in the dataset. The comparable intervals are seven years for automobile crash safety, nine for crypto (a mandate on safeguarding rather than on assurance), nineteen for sustainability assurance, forty-two for boilers, and seventy-eight for electrical product safety. Five of seven stages in four years outpaces every regime in the set — with the caveat that frontier AI is the only regime observed prospectively, while the others' early histories are reconstructed from secondary sources that may miss short-lived early stages, so the speed comparison is suggestive, not a finding.

5.2 Rules arrive after failures; drift before them is slow

Across twenty trigger events in the fifteen non-AI regimes, the lag from the trigger to the next rule of any kind has a median of two years and a maximum of ten. Seven of twenty are one year. The interval from a regime's first milestone to its first independence rule, where one exists, is 5 years (platform audits), 17 (nuclear), 92 (pharmaceuticals), 94 (aviation), 101 (credit ratings), 158 (financial audit), and 249 (ship classification).

The two clocks run at different speeds. Once a failure is public, regimes act within an election cycle. Before that, they drift for decades. This is a pattern in a hindsight-selected record, not causal evidence: the triggers were chosen partly because they were followed by rules, and an outcome-independent enumeration of qualifying incidents — including those followed by nothing — is needed before the lag pattern can be called a finding rather than a description. Frontier AI's first triggers arrived in 2025 and 2026. If the historical pattern holds, the rules that shape its assurance layer are being written now.

5.3 The assessed party keeps paying

In thirteen of the fifteen non-AI regimes, the party being assessed pays the assessor after reform, either directly (audit fees, certification fees, issuer fees, classification fees, user fees) or through delegation to its own staff. Independence, where it arrived, came from auxiliary mechanisms: standards (1939 auditing procedures after McKesson & Robbins), inspection of assessors (PCAOB, GLP), rotation (Sarbanes-Oxley partner rotation; EU firm rotation; DSA auditor rotation), separation of assessment from consulting (Sarbanes-Oxley), liability exposure (Dodd-Frank for rating agencies), and public comparative ratings (NCAP).

Two regimes removed the payer conflict. Boiler inspection routed inspection through insurers from 1866: the inspector is paid by the party that loses money if the boiler explodes. Automobile crash testing at IIHS is funded by insurers and published as ratings from 1995; manufacturers do not pay and cannot opt out. Both models have live AI analogues in proposal form (insurance-backed certification, insurer- or user-funded public ratings), and neither has an AI implementation yet.

5.4 Delegation is the recurring failure mode

Five regimes in Dataset B show the assessed party absorbing the assessment function: consulting revenue overtaking audit fees at accounting firms before Enron; sponsors running the trials regulators review; issuer-pays ratings; Organization Designation Authorization letting manufacturers' employees certify aircraft; and frontier AI, where developers run their own safety evaluations and select, scope, and pay the external testers cited in their system cards. In three of the four historical cases a public failure was followed by a reform that reclaimed part of the delegated function (Sarbanes-Oxley 2002, Good Laboratory Practice 1978, the 2020 aircraft certification reforms). Frontier AI's 2026 commitments move in the same direction, from lab-scoped engagements toward embedded evaluators with publication rights, but as pledges rather than rules.

5.5 The AI evaluator population in 2026

Under the lab-procurement preset and the retrieved-&-confirmed evidence policy (confirmed sources only), 13 of the 24 ranked organizations trigger no floor, 8 carry a conditional floor (an evidenced 1 on a conflict dimension), and 3 carry a disqualifying floor (an evidenced 0 on one). Bands are set by funding, governance, personnel, role incompatibility, scope control, and publication rights; access depth and method transparency count in the number and never set a band, because a 0 there records what labs have granted or what has been published, not a conflict (decision D-002). The no-floor group, ranked by number, opens with METR (79), Redwood Research (78, with personnel unevidenced), the EU AI Office (75), the UK AI Security Institute (73), and US CAISI (72, with publication rights unevidenced under the retrieved-&-confirmed policy), then RAND, Palisade, Transluce, Humane Intelligence, AVERI, Apollo Research, FAR.AI, and SaferAI: every government institute, and the nonprofits whose lab ties stop at pooled philanthropy. The conditional band holds SecureBio (funding 1 under the rule that a grant from a lab's majority owner is lab-linked money, and role incompatibility 1 for licensing evaluation products to labs), the Center for AI Safety (personnel 1: its director advises xAI), MLCommons (funding 1: member labs fund it), and the vendors EquiStamp, Andon Labs, Dreadnode, Irregular, and Nemesys. The disqualifying band holds Microsoft's red team (governance and personnel 0: a business unit of a developer), Gray Swan (role incompatibility 0: it sells the fix to the labs it grades), and Scale (funding, governance, and personnel 0 under the 20% ownership rule). The numbers inside a band rank, and a two-point gap means nothing; on the site the number is hidden until the reader chooses weights.

Across the 24, mean values under the retrieved-&-confirmed policy are scope control 2.71, method transparency 2.39, role incompatibility 2.30, funding 2.26, publication rights 2.23, personnel 1.95, governance 1.88, and access 1.79; 8 of 192 assessments are unevidenced under that policy and excluded from the means, and the leads-included means differ by at most 0.06. Governance and personnel fell from v0's 2.38 and 2.35 because legal form alone now floors governance at 2 and "no lab roles found" floors personnel at 2 (RULES G.1, P.6); access fell from 2.50 to 1.79 because access is scored as what labs have documented granting (A.1), and a released model evaluated through an API is 0. 10 of 24 score 2 or below on funding, which at anchor 2 means the lab pays per engagement.

By type: government bodies (n=3) average 3.0 on funding, governance, personnel, and access and 2.0 on publication: they see the most and publish the least. Nonprofits (n=11) average 2.73 on funding, 3.00 on scope, 2.55 on methods and on publication, 2.64 on role incompatibility, and 1.82 on access. Venture-backed evaluators (n=5) average 1.20 on funding, 0.80 on governance, 1.25 on personnel, and 1.80 on access. By list: the twelve independent referees split ten no-floor and two conditional; the three government institutes all trigger no floor; the nine commercial and first-party organizations split six conditional and three disqualifying, with the lowest means on every conflict dimension. Expectation H4 (paper/README.md) said the population would score higher on access and methods than on funding and personnel, and that philanthropic or public funding would score highest. Its second half holds: the no-floor group is public bodies and philanthropically funded nonprofits. Its first half fails under the rules: access ties governance as the weakest dimension in the population, below funding, which is what the September 2026 access pledges target. With n=24, a single coder, and 0 to 4 integer anchors, these are orderings of means, not statistical support.

Signals split 165 for and 166 against across the ranked population, 301 of 331 with a quoted span. Of 238 unique sources cited, 111 are the evaluators' own statements and 66 are filings or funder indexes. Under the against-interest policy 49 of 192 assessments are unevidenced; under primary only, 167 are. 3 assessments are held at a supportable anchor because an extreme lacks a span or a second source, and 10 conflicts between a floor and a cap are resolved by a named rule on the card. The scarcity of independent verification is itself a finding: the scores measure what the public record shows, and the public record is largely what the evaluators say about themselves; the primary-only column shows how little of it can be checked from outside the field.

5.6 Paths: which history AI's path resembles

Written as sequences, the regimes do not share a ladder. Financial audit reads mandate, rollback, voluntary, mandate, access, trigger, standards, delegation, trigger, oversight, independence. Nuclear opens with a mandate and adds voluntary peer review twenty years later. Dietary supplements opens with a rollback. Frontier AI's sequence so far is voluntary, delegation, voluntary, trigger, standards, access, trigger, mandate, access.

The openings most similar to AI's are credit ratings and crypto proof of reserves (distance 0.56, similarity 0.44): voluntary assurance, then the assessed party becomes the client, then a trigger. The comparison is over openings only, the sequences are short, the coding is coarse, and two regimes tie — so this is a hypothesis generator, not a result, and should not be quoted as a prediction about AI. It says the nearest analogues include the regime where regulators licensed a payer-conflicted assessor and then met 2008, and the regime whose voluntary assurance collapsed.

5.7 Fast responses are mostly weak responses

Pairing each trigger with the next rule and its strength changes the reading of the lag result. Sixteen of the twenty non-AI triggers were answered within three years, with a mean instrument strength of 2.88 on the 1 to 4 scale. Two of the sixteen were structural: the PCAOB after Enron, and inspection of testing labs under Good Laboratory Practice after the Industrial Bio-Test fraud. Two triggers were followed by nothing in the dataset (the Target breach; the 2025 analysis of hiring-tool audits). Frontier AI's two triggers pair with instruments of strength 2 and 1: the EU access floor and the embedded-evaluator pledges — with the caveat that year granularity cannot establish that the 2025 access floor followed the 2025 FrontierMath disclosure rather than preceding it.

5.8 Money by distance from a lab

A row-level ledger behind the funding and personnel dimensions changes what "no lab money" means. Of 24 ranked evaluators, 10 have at least one traced direct tie to a frontier lab (hop 0), and 15 have at least one traced tie to a lab or to a party with a direct tie to one (hop 0 or 1: an investor, board observer, employee, founder, or contractor), counting only evidential rows (confirmed or research-pass; quarantined rows excluded) and not counting credits consumed in evaluating a lab's own model, which are the substrate of the work rather than a payment and are recorded on entity pages without entering any count (decision D-004). The confirmatory figure is the narrower one: direct lab cash or contracts for evaluation work, which is six of 24. Of the ten hop-0 ties, six are lab cash for evaluation work, two are ownership (Meta's $14.3B purchase of a 49% Scale stake; Microsoft's red team as a business unit), and two are no-fee partnerships or memberships (Anthropic's Project Vend deployment at Andon Labs; Google's founding participation in MLCommons); the latter two are not evaluator funding. The figures moved during the verification passes of 2026-09-15, falling to 9 and 14 when a re-derivation of all 38 imported rows quarantined 23 whose cited pages did not support the recorded figures, and returning to 12 and 17 when three direct ties were confirmed against better sources (Meta to Scale, Anthropic to Andon, Google to MLCommons), this time on confirmed evidence. The release projection holds 84 transfer or collaboration rows: 49 confirmed, 14 unaudited, 5 differing, 14 unverifiable, and 2 superseded; 49 of 62 evidential inflow rows are confirmed, and 18 quarantined rows are excluded from exposure and from every dollar figure. One Coefficient Giving inflow is confirmed from its cited source (AVERI); Epoch's $24.5M matches Epoch's own funding list to the dollar once the July 2026 grant made after the index snapshot is excluded; Coefficient's principal is an Anthropic Series A investor by his own account. Dollar sums include only confirmed and unaudited USD rows; euro amounts are shown unconverted; component rows are detail rather than addends. The two evaluators with the most complete self-disclosure, Transluce and AVERI, are the ones whose disclosures move their own funding values down; under the bounds model an undisclosed funder base earns nothing either, because a dimension with no admissible signal renders as a dash rather than a number.

5.9 Mechanisms: the AI row

On the five mechanism questions, frontier AI today reads: the assessed party pays, the assessed party selects, access is shallow, the public reads summaries, and nobody oversees the assessors. Thirteen of the fifteen non-AI regimes have an oversight cell filled. The one other regime with AI's exact five-cell pattern is dietary supplements — on a coarse categorical coding, which makes the match suggestive rather than conclusive. The two regimes that moved payment and selection to a party with opposing exposure, boiler inspection and crash testing, are also the two where the assessed party never regained control of scope.

6. Discussion

What the history predicts. If frontier AI follows the pattern in Dataset B, the two or three years after its first public failures are when its assurance regime takes shape, and the shape will be set by rules written in response to those failures rather than by the voluntary arrangements that preceded them. The instruments in motion now (the EU framework, Illinois SB 315, the embedded-evaluator pledges) fit the pattern in timing. They do not yet fill the S5 cell, and S6 holds only an untested financial-interest test embedded in a mandate. Every mature regime in the dataset that kept a payer-conflicted model built an inspector of inspectors; AI has none, and the EU Code's "adequately qualified" and Illinois's "demonstrated competence" are the words regimes use before they build one.

Who pays the referee. The historical answer is that the referee is paid by the team, and the league makes that tolerable by licensing referees, watching them, rotating them, and letting the public see the score. The exceptions are the regimes where a party with opposing financial exposure pays. In AI, the philanthropic model (METR) works at small scale and does not scale to embedded teams at every developer; the lab-fee model (Apollo, SecureBio, Irregular, Gray Swan) scales and carries the conflict; the government model (UK AISI, CAISI, EU AI Office) has depth but publishes little and is exposed to political redirection. The insurer-routed and public-ratings models are untried.

Three design options the data point to, in increasing order of departure from current practice: (1) accredited lab-paid evaluation with a registry and inspection of evaluators, on the PCAOB and NRTL pattern; (2) mandatory publication of evaluator operating conditions per engagement, on the trial-registration pattern, which AEF-1 already specifies and no regulator yet requires; (3) an evaluator funded by parties exposed to model failure, on the boiler-insurance or IIHS pattern.

The large assurance firms. Illinois SB 315 will send frontier developers to auditors from 2028, and the firms that already audit their financial statements are the expected entrants. Version 0 scored a hypothetical composite of them; v0.1 does not, because a made-up organization cannot be evidenced. What the record does show is that Deloitte, as the successor to Gryphon Scientific, already appears in current system cards as a designer of biological-risk evaluations, so the audit-plus-consulting question that Sarbanes-Oxley settled for financial audit will arrive in AI with the same firms on both sides of it. The population check records it as an in-scope candidate for the next batch.

7. Limitations

Triggers were selected with hindsight in v0: the lag results describe a record biased toward responsiveness, and v0.2's outcome-independent criterion has not yet been applied to enumerate qualifying incidents that were followed by nothing. Until it is, Section 5.2 is a description of the coded record, not a finding about how regimes behave. Path similarity is computed over short sequences and coarse codings, over openings only, and two regimes tie for nearest — it is a hypothesis generator. Mechanism coding is single-coder and categorical; the supplements match in 5.9 rests on five coarse cells.

Dataset B uses year granularity and secondary sources in v0; ordering within a year is lost, and same-year trigger-rule pairings at lag 0 may not reflect the actual order of events. Regime selection is judgmental; the four added regimes reduce selection on success but do not remove it. Coding milestone kinds involves judgment at the margins (PDUFA is coded as a payer shift toward the assessed party rather than an independence reform). Frontier AI is the only regime observed prospectively, which flatters its speed relative to regimes whose early histories are reconstructed. Dataset A scores public records only; private contract terms could raise or lower any score, and evaluators can move their scores by publishing their terms. The scores are ordinal projections over anchored rubrics, not interval measurements — a 10-point gap is not twice the independence. Version 0 is single-coder, and much of the public record is self-published by the evaluators themselves.

The rules in RULES.md are the curator's rules: written before the bounds were assigned, applied to every organization the same way, and open to public correction, but not yet the product of a second coder or an external panel. Each signal's bound was assigned by the same single coder with AI assistance, and a stranger can disagree with a bound, a rule, or a resolution on the card rather than with the number. The evidence policies make the field's dependence on self-report visible; they do not remove it, and under the primary-only policy almost nothing in the population is evidenced, which is the central limitation of Dataset A and of the field.

The authors have relationships in the AI investment and research ecosystem, disclosed in the repository and on the site's method page, which is the same disclosure this paper asks of evaluators. The curation was drafted and reviewed with models from four developers that are labs in the ledger (Anthropic, OpenAI, xAI, Google); every value is re-derivable from public bounds without any of them, and the place a model's involvement could have tilted a value, a bound or a rule, is open to correction.

8. Conclusion

Frontier AI has compressed into four years a passage other industries took decades to make, and it has done so by adopting standards, mandates, and access rules while leaving the evaluators unwatched and independence rules untested. The history says failures come first and rules follow within a few years; AI has had its failures. The choice now is which of the three historical answers to the payer problem the field adopts, and whether it builds the oversight layer before or after the next incident. The datasets are open and rebuild from their evidence; we invite evaluators to change their scores by publishing their terms.

Figures and tables

• Figure 1. Stage ladder: paper/figures/stage-ladder.svg (generated by python -m bench timeline).
• Figure 2. Path strips with similarity to AI: paper/figures/paths.svg.
• Figure 3. Trigger-to-rule lag against instrument strength: paper/figures/responses.svg.
• Figure 4. Mechanism matrix: paper/figures/mechanisms.svg.
• Table 1. Trigger-to-rule lags: python -m bench industries.
• Table 2. Dataset A scorecard under four presets: python -m bench scores <preset>.
• Table 3. Per-dimension means by organization type: python -m bench scores lab --by-type.
• Table 4. What counts under each evidence policy: the status page at the pinned commit.

References

See paper/related-work.md. Dataset A sources are in data/sources/; Dataset B sources are inline in data/industries/.