Who Pays the Referee? Measuring Third-Party Evaluator Independence in Frontier AI Against the History of Assurance Regimes

Version 0 draft, 15 September 2026. All numbers regenerate from this repository at the pinned commit (python -m bench verify && python -m bench build && python -m bench industries && python -m bench paths). Dataset B milestone years are coded from secondary sources and need primary-source verification before submission. Dataset A is single-coder.

Abstract

Frontier AI developers now cite outside evaluators in nearly every model release, and regulators in the European Union and Illinois require "independent" external evaluation — Illinois defines the test, borrowing financial audit's rule that neither party may hold a financial interest in the other. We ask two questions. How independent are the evaluators the field relies on, measured against criteria that other industries eventually wrote into law? And where does frontier AI sit on the path those industries took from voluntary assurance to regulated independence? We answer with two open datasets. Dataset A scores 26 organizations that evaluate frontier models on eight independence dimensions (funding, governance, personnel, access depth, scope control, publication rights, method transparency, role incompatibility), each 0 to 4 with written anchors, each assessment linked to dated, sourced signals and rebuilt from an append-only event log; two expected entrants are scored on the same rubric on a watchlist, excluded from rankings and averages. Dataset B codes the lifecycle of sixteen assurance regimes across fifteen industries as dated milestones: voluntary assurance, trigger incidents, mandates, standards, oversight of assurers, independence rules, access and publication rules. Three findings. First, in the coded record, rules arrive within a few years of public failures while the drift before the first failure spans decades — a pattern consistent with responsiveness, not proof of it, since the triggers were selected with hindsight. Second, the party being assessed almost always keeps paying the assessor; independence arrived through standards, inspection of assessors, rotation, liability, and public ratings, and the two regimes that removed the payer conflict did so by routing payment through insurers or funding the assessor from a party with opposing incentives. Third, the 2026 frontier AI evaluator population is strongest on scope control and method transparency and weakest on funding, governance, and personnel, and it has reached five of seven lifecycle stages in four years while skipping the two every mature regime built: oversight of the evaluators themselves, and standalone independence rules.

1. Introduction

In September 2026 the chief executives of the two largest frontier AI developers committed, within hours of each other, to giving outside evaluators employee-level access to their systems and the right to publish findings without editorial control. The commitments name METR as the model evaluator. They do not say who will pay the evaluators, who will select them, or what happens when an evaluator and a lab disagree about what counts as security-sensitive. A month earlier, Illinois enacted a statutory audit mandate for frontier developers, borrowing its independence test from financial audit: neither party may hold a financial interest in the other. The European Union's AI Act framework, supported by a voluntary Code of Practice, sets an access floor of twenty business days and minimally guardrailed model versions for external evaluation of systemic-risk models.

The organizations these instruments point at are few. The same six or seven groups appear in the system cards of OpenAI, Anthropic, and Google DeepMind. Some are nonprofits that refuse lab money; some are venture-backed companies whose investors also back the labs; one is a business unit of a frontier developer; one is a founder's side project whose chief scientist chairs a lab's safety committee. We are aware of no public scored, sourced account of them.

There is a long literature arguing that AI should learn from the assurance regimes of other industries (Raji et al. 2022; Manheim et al. 2025; Brundage et al. 2026). That argument is made. What is missing is measurement on both sides of the analogy: a coded reading of the AI evaluator population against the criteria other industries adopted, and a coded history of when and why those industries adopted them. This paper supplies both and reports what the comparison says.

Contributions:

1. Dataset A, an evidence-linked scorecard of 26 frontier AI evaluators on eight independence dimensions, with 314 dated signals citing 122 public sources, built as an event-sourced graph so every score traces to its evidence. Two expected entrants are scored on the same rubric on an unranked watchlist.
2. Dataset B, a lifecycle dataset of sixteen assurance regimes coded as dated milestones of eleven kinds, with per-regime payer, access, and publication models.
3. A stage ladder that places frontier AI on the trajectory other regimes followed, and three empirical regularities about how assurance regimes acquire independence.
4. A reading of what those regularities imply for the design choices now open in AI.

2. Related work

Frameworks for third-party evaluation of AI systems have proliferated: audit ecosystem design (Raji et al. 2022), the ASPIRE criteria for external scrutiny (Anderljung et al. 2023), access taxonomies (Casper et al. 2024; Charnock et al. 2026), compliance reviews of safety frameworks (Homewood et al. 2025), and AI Assurance Levels (Brundage et al. 2026). The AI Evaluator Forum's AEF-1 standard specifies minimum operating conditions for independent evaluation. Empirical work on auditors exists for algorithmic bias audits: a field scan and practitioner survey (Costanza-Chock et al. 2022) and a coded analysis of 116 published New York City hiring-tool audits (Groves et al. 2025). Rating efforts score developers' safety frameworks (SaferAI; the Future of Life Institute index), not evaluators. Cross-industry precedent appears in most of these papers as illustration; the systematic search closest in spirit (Anderson-Samways 2024) catalogs US regulatory instruments rather than assurance-provider lifecycles. We are aware of no prior work that codes the frontier evaluator population against independence criteria with sourced evidence, and none that codes assurance-regime lifecycles as comparable dated events. A fuller annotated bibliography is in the repository.

3. Datasets

3.1 Dataset A: the evaluator scorecard

Population. Organizations that (a) have evaluated or red-teamed a frontier model before or shortly after release and been cited by a developer, government, or regulator for doing so, or (b) have a stated role in the emerging audit layer (standards bodies, regulators, announced entrants). Twenty-six ranked organizations as of v0, split by role in every table: fifteen independent referees, three government bodies, five vendors, two benchmarks, one first-party lab team. Two expected entrants (a hypothetical composite of large assurance firms; an initiative announced days before the evidence freeze with no evaluations yet) are scored on the same rubric on a watchlist, excluded from rankings and from every average.

Dimensions and anchors. Eight dimensions, each scored 0 to 4 against written anchors. The set draws on AEF-1's five operating conditions, on the auditor-independence rules Illinois SB 315 imports from financial audit, and on the Charnock et al. access taxonomy, with two additions the AI literature tends to omit: who owns the evaluator, and whether it sells remediation to the companies it grades. Anchors are in data/dimensions.json. Access is scored as access labs have actually granted, not as an institutional right: a low access score can mean exclusion, not incompetence.

Evidence model. A signal is one dated claim, for or against, on one dimension, citing at least one source with a URL and a retrieval date. An assessment is a value on one dimension citing at least one signal on that dimension. A score is a weighted total, never stored, recomputed at build time under named weight presets. Integrity is enforced by a verifier: no orphan sources, no assessment without a signal, no 4 on funding/governance/personnel without a tier-1 (filing or index) source on a cited signal, no record dated after the frozen evidence clock. The build writes every object into an append-only event log under a frozen clock; two builds on the same data produce a byte-identical log, and the provenance of every score names the events it rests on. Bench-authored docket drafts are not evidence and cannot be cited by signals.

Coding. Version 0 is single-coder from public sources between 12 and 15 September 2026, with no second coder claimed; the check is public: every assessment, signal, row and source is published with its status, and corrections arrive through a reviewed contribution path. Values are bounded by evidence status: a 0 or 4 needs a signal whose sources are all confirmed, a 1 or 3 needs a confirmed or unaudited source, and seven assessments in v0 are held at the nearest supportable anchor and marked evidence-limited (PROCESS section 12). Confidence tags (high, medium, low) mark thin records. Each assessment carries a computed evidence tier (the best source tier among its cited signals), displayed on the scorecard, so imported-only and self-only assessments are visible as such. A second-coder pass with disagreements logged in the rationale fields is scheduled before submission.

3.2 Dataset B: assurance-regime lifecycles

Sixteen regimes: financial statement audit, credit ratings, electrical and consumer product safety, pharmaceutical safety and efficacy, aircraft certification, nuclear safety and safeguards, ship classification, cybersecurity assurance, boilers and pressure vessels, food safety, automobile crash safety, sustainability reporting assurance, dietary supplements, crypto exchange proof of reserves, platform and hiring-algorithm audits, and frontier AI evaluation. The first eleven are regimes that reached mandated assurance; the next four were added to avoid selecting on success and include one rollback (supplements), one collapse of voluntary assurance (crypto), one slow voluntary period (sustainability), and one young regime that wrote independence rules at the start (platform audits).

Milestones are coded by kind: voluntary_assurance, trigger, mandate, standards, accreditation (oversight of assessors), independence_rule, delegation_reform, payer_reform, access_expansion, publication_rule, delegation, payer_shift, rollback. One instrument is coded as one milestone at its primary operative kind; multi-provision laws are not double-counted across stages. Each regime also records its payer model, access model, and publication model. Year granularity. Every milestone cites a source; v0 uses secondary sources with years spot-checked, and each entry is flagged for primary verification.

4. Method

Instrument strength and trigger criterion. Every rule milestone carries a strength from 1 (disclosure or voluntary text) to 4 (structural change: separation of functions, rotation, a new inspecting body, payment rerouted, delegation reclaimed). Every trigger carries a harm class and must meet a criterion set before looking at what followed: ten or more deaths, losses above one billion dollars, or a documented integrity failure of the assurance itself.

Stage ladder. Seven stages are defined from milestone kinds: S1 voluntary assurance, S2 trigger, S3 mandate, S4 standards, S5 oversight of assessors, S6 independence rules (independence_rule, delegation_reform, payer_reform), S7 access and publication rules. A regime reaches a stage in the year of the first qualifying milestone. Delegation, payer shifts toward the assessed party, and rollbacks are recorded and displayed but do not advance a stage. Accreditation counts as oversight (S5), not as an independence rule (S6).

Lags. For each trigger we compute the years to the next milestone of any rule kind (mandate, standards, accreditation, independence_rule, delegation_reform, payer_reform, publication_rule). For each regime we compute the interval from its first milestone to its first S6 milestone. Same-year pairings are reported at lag 0; year granularity cannot resolve ordering within a year.

Paths. Each regime's milestones are ordered by year and written as a sequence of kinds with consecutive repeats collapsed. Similarity between frontier AI's sequence and each regime's opening is one minus the normalized Levenshtein distance over openings of comparable length.

Mechanisms. Each regime is coded before reform and now on who pays the assessor, who selects it, what it sees, what the public reads, and who oversees the assessor. The coding is categorical and coarse; matches on the full pattern are suggestive, not conclusive.

Scores. Dataset A values are weighted under four presets and reported with the distribution per dimension and per organization type. Scores are gated by default: any 0 on a dimension caps the total at 40, any 1 caps it at 60 — independence has floors, not just averages. The lab-procurement preset is the pre-registered confirmatory view; the other presets are sensitivity checks. Raw compensatory scores are available as sensitivity analysis only.

Everything above is computed by python -m bench industries, python -m bench timeline, python -m bench paths, and python -m bench scores.

5. Results

5.1 Where frontier AI sits (Figure 1)

Frontier AI reached S1 in 2022 (pre-release testing of GPT-4 by ARC Evals), S2 in 2025 (the FrontierMath funding disclosure) with a second, more serious trigger in 2026 (the OpenAI agent swarm incident and the first on-site third-party incident investigation), S3 in 2026 (Illinois SB 315, audits from 2028, with a financial-interest test embedded in the mandate), S4 in 2025 (AEF-1), and S7 in 2025 (the Code of Practice access floor, followed by the 2026 embedded-evaluator pledges). It has reached neither S5 nor S6: there is no body that accredits, inspects, or registers frontier AI evaluators, and no standalone independence rule — SB 315's financial-interest test exists only inside the mandate and has not been applied to any audit yet.

The empty cells are the informative ones, and they are the result. Eight of the fifteen non-AI regimes built an oversight layer for their assessors (a registry, an accreditation program, or an inspector of inspectors): financial audit (PCAOB, 2002), ship classification (IACS, 1968), boilers (National Board, 1919), product safety (OSHA NRTLs, 1988), pharmaceuticals (Good Laboratory Practice inspection, 1978), credit ratings (NRSRO, 1975), food safety (FSMA accredited certifiers, 2011), and cybersecurity (Common Criteria labs, 1999). AI has standards without anyone checking who applies them, and a mandate whose independence test has never been exercised.

Four years from voluntary assurance to mandate is the shortest interval in the dataset. The comparable intervals are seven years for automobile crash safety, nine for crypto (a mandate on safeguarding rather than on assurance), nineteen for sustainability assurance, forty-two for boilers, and seventy-eight for electrical product safety. Five of seven stages in four years outpaces every regime in the set — with the caveat that frontier AI is the only regime observed prospectively, while the others' early histories are reconstructed from secondary sources that may miss short-lived early stages, so the speed comparison is suggestive, not a finding.

5.2 Rules arrive after failures; drift before them is slow

Across twenty trigger events in the fifteen non-AI regimes, the lag from the trigger to the next rule of any kind has a median of two years and a maximum of ten. Seven of twenty are one year. The interval from a regime's first milestone to its first independence rule, where one exists, is 5 years (platform audits), 17 (nuclear), 92 (pharmaceuticals), 94 (aviation), 101 (credit ratings), 158 (financial audit), and 249 (ship classification).

The two clocks run at different speeds. Once a failure is public, regimes act within an election cycle. Before that, they drift for decades. This is a pattern in a hindsight-selected record, not causal evidence: the triggers were chosen partly because they were followed by rules, and an outcome-independent enumeration of qualifying incidents — including those followed by nothing — is needed before the lag pattern can be called a finding rather than a description. Frontier AI's first triggers arrived in 2025 and 2026. If the historical pattern holds, the rules that shape its assurance layer are being written now.

5.3 The assessed party keeps paying

In thirteen of the fifteen non-AI regimes, the party being assessed pays the assessor after reform, either directly (audit fees, certification fees, issuer fees, classification fees, user fees) or through delegation to its own staff. Independence, where it arrived, came from auxiliary mechanisms: standards (1939 auditing procedures after McKesson & Robbins), inspection of assessors (PCAOB, GLP), rotation (Sarbanes-Oxley partner rotation; EU firm rotation; DSA auditor rotation), separation of assessment from consulting (Sarbanes-Oxley), liability exposure (Dodd-Frank for rating agencies), and public comparative ratings (NCAP).

Two regimes removed the payer conflict. Boiler inspection routed inspection through insurers from 1866: the inspector is paid by the party that loses money if the boiler explodes. Automobile crash testing at IIHS is funded by insurers and published as ratings from 1995; manufacturers do not pay and cannot opt out. Both models have live AI analogues in proposal form (insurance-backed certification, insurer- or user-funded public ratings), and neither has an AI implementation yet.

5.4 Delegation is the recurring failure mode

Five regimes in Dataset B show the assessed party absorbing the assessment function: consulting revenue overtaking audit fees at accounting firms before Enron; sponsors running the trials regulators review; issuer-pays ratings; Organization Designation Authorization letting manufacturers' employees certify aircraft; and frontier AI, where developers run their own safety evaluations and select, scope, and pay the external testers cited in their system cards. In three of the four historical cases a public failure was followed by a reform that reclaimed part of the delegated function (Sarbanes-Oxley 2002, Good Laboratory Practice 1978, the 2020 aircraft certification reforms). Frontier AI's 2026 commitments move in the same direction, from lab-scoped engagements toward embedded evaluators with publication rights, but as pledges rather than rules.

5.5 The AI evaluator population in 2026

Under the lab-procurement preset (gated), scores run from 81 (METR) to 38 (Scale AI), with Gray Swan at 40 and Microsoft AI Red Team at 41. The top scorers — METR, AVERI, SaferAI, and a three-way tie at 75 between the UK AI Safety Institute, the EU AI Office, and FAR.AI — are nonprofits or public bodies, consistent with H4's expectation that philanthropic or public funding scores highest. Nonprofits (n=12) average 3.0 or above on governance, scope, publication, methods, and product conflicts, 2.75 on funding, 2.58 on personnel, and 2.42 on access. Government bodies (n=3) score 3.0 on funding and 1.7 on publication: they see the most and publish the least. Venture-backed evaluators (n=5) average 1.0 on funding, governance, and product conflicts, 1.6 on personnel, and 2.8 on access. The one Big Tech unit is held at 1 on funding and governance under the evidence-limited rule (its cited sources are unaudited), where the record read alone would give 0.

Across all 26 ranked, mean scores by dimension are scope control 2.96, method transparency 2.88, role incompatibility 2.77, publication rights 2.62, access 2.50, governance 2.38, personnel 2.35, and funding 2.23. Twelve of 26 score 2 or below on funding, which at anchor 2 means the lab pays per engagement. Hypothesis H4 predicted the distribution would score higher on access and methods than on funding and personnel; the dimension means are ordered that way — access (2.50) and methods (2.88) above funding (2.23) and personnel (2.35) — but with n=26, single coding, and 0–4 integer anchors, that is an ordering of means, not statistical support. Access sits in the middle of the pack rather than among the strengths; the September 2026 access pledges target exactly the gap between access and the top dimensions.

Signals split 159 for and 135 against across the ranked population. The most frequent against-signal across organizations is that the developer pays for the evaluation of its own model. The most frequent for-signal is publication of a method or a tool. External validation is sparse: among the top five scorers, AVERI's seven cited sources are five self-published with none at tier 1, and self-published sources anchor a material share of high assessments across the population. The scarcity of independent verification is itself a finding — the scores measure what the public record shows, and the public record is largely what the evaluators say about themselves.

5.6 Paths: which history AI's path resembles

Written as sequences, the regimes do not share a ladder. Financial audit reads mandate, rollback, voluntary, mandate, access, trigger, standards, delegation, trigger, oversight, independence. Nuclear opens with a mandate and adds voluntary peer review twenty years later. Dietary supplements opens with a rollback. Frontier AI's sequence so far is voluntary, delegation, voluntary, trigger, standards, access, trigger, mandate, access.

The openings most similar to AI's are credit ratings and crypto proof of reserves (distance 0.56, similarity 0.44): voluntary assurance, then the assessed party becomes the client, then a trigger. The comparison is over openings only, the sequences are short, the coding is coarse, and two regimes tie — so this is a hypothesis generator, not a result, and should not be quoted as a prediction about AI. It says the nearest analogues include the regime where regulators licensed a payer-conflicted assessor and then met 2008, and the regime whose voluntary assurance collapsed.

5.7 Fast responses are mostly weak responses

Pairing each trigger with the next rule and its strength changes the reading of the lag result. Sixteen of the twenty non-AI triggers were answered within three years, with a mean instrument strength of 2.88 on the 1 to 4 scale. Two of the sixteen were structural: the PCAOB after Enron, and inspection of testing labs under Good Laboratory Practice after the Industrial Bio-Test fraud. Two triggers were followed by nothing in the dataset (the Target breach; the 2025 analysis of hiring-tool audits). Frontier AI's two triggers pair with instruments of strength 2 and 1: the EU access floor and the embedded-evaluator pledges — with the caveat that year granularity cannot establish that the 2025 access floor followed the 2025 FrontierMath disclosure rather than preceding it.

5.8 Money by distance from a lab

A row-level ledger behind the funding and personnel dimensions changes what "no lab money" means. Of 26 ranked evaluators, 12 have at least one traced direct tie to a frontier lab (hop 0), and 17 have at least one traced tie to a lab or to a party with a direct tie to one (hop 0 or 1: an investor, board observer, employee, founder, or contractor) — counting only evidential rows (confirmed or research-pass; quarantined rows excluded). The confirmatory figure is the first one — direct lab cash, contracts, or in-kind — not the broader sensitivity statistic. Both figures fell on 2026-09-15, from 12 and 17 to 9 and 14, when a verification pass re-derived all 38 imported rows and quarantined 23 whose cited pages did not support the recorded figures; a follow-up pass confirmed three more direct ties against better sources (Meta→Scale, Anthropic→Andon, Google→MLCommons), bringing the figures back to 12 and 17 — this time on confirmed evidence rather than un-re-derived imports. Of the twelve hop-0 ties, eight are lab cash or in-kind for evaluation work, two are ownership (Meta's $14.3B purchase of a 49% Scale stake; Microsoft's red team as a business unit), and two are no-fee partnerships or memberships (Anthropic's Project Vend deployment at Andon Labs; Google's founding participation in MLCommons). The confirmatory count is the first: eight of 26. One Coefficient Giving inflow is confirmed from its cited source (AVERI); seven more imported rows remain unsupported by their cited pages, and T07's $24.5M matches Epoch's own funding list to the dollar once the July 2026 grant made after the index snapshot is excluded. Coefficient's principal is an Anthropic Series A investor by his own account. Forty-eight of 82 inflow rows have been re-derived from the cited source; the rest are research-pass leads or quarantined, and the ledger names 36 distinct funders. Dollar sums include only confirmed and unaudited USD rows; euro amounts are shown unconverted, and contradicted, superseded, and unverifiable rows are quarantined from every figure. The two evaluators with the most complete self-disclosure, Transluce and AVERI, are also the ones whose disclosures move their own scores down, which is what an honest disclosure regime should look like.

The preceding audit-path narrative records interim review work; the release projection is the authoritative current state. It contains 84 transfer or collaboration rows: 49 confirmed, 14 unaudited, 5 differing, 14 unverifiable, and 2 superseded. Across evaluators, 49 of 62 evidential inflow rows are confirmed; 18 quarantined rows are excluded from exposure and every dollar figure. The direct tie counts remain 8 cash or in-kind, 2 ownership, and 2 no-fee partnerships or memberships; the latter two are not evaluator funding. One Coefficient Giving inflow is confirmed from its cited source (AVERI), and T07's $24.5M matches Epoch's own funding list to the dollar once the July 2026 grant made after the index snapshot is excluded. Dollar sums include only confirmed and unaudited USD rows; euro amounts are shown unconverted; component rows are detail rather than addends.

5.9 Mechanisms: the AI row

On the five mechanism questions, frontier AI today reads: the assessed party pays, the assessed party selects, access is shallow, the public reads summaries, and nobody oversees the assessors. Thirteen of the fifteen non-AI regimes have an oversight cell filled. The one other regime with AI's exact five-cell pattern is dietary supplements — on a coarse categorical coding, which makes the match suggestive rather than conclusive. The two regimes that moved payment and selection to a party with opposing exposure, boiler inspection and crash testing, are also the two where the assessed party never regained control of scope.

6. Discussion

What the history predicts. If frontier AI follows the pattern in Dataset B, the two or three years after its first public failures are when its assurance regime takes shape, and the shape will be set by rules written in response to those failures rather than by the voluntary arrangements that preceded them. The instruments in motion now (the EU framework, Illinois SB 315, the embedded-evaluator pledges) fit the pattern in timing. They do not yet fill the S5 cell, and S6 holds only an untested financial-interest test embedded in a mandate. Every mature regime in the dataset that kept a payer-conflicted model built an inspector of inspectors; AI has none, and the EU Code's "adequately qualified" and Illinois's "demonstrated competence" are the words regimes use before they build one.

Who pays the referee. The historical answer is that the referee is paid by the team, and the league makes that tolerable by licensing referees, watching them, rotating them, and letting the public see the score. The exceptions are the regimes where a party with opposing financial exposure pays. In AI, the philanthropic model (METR) works at small scale and does not scale to embedded teams at every developer; the lab-fee model (Apollo, SecureBio, Irregular, Gray Swan) scales and carries the conflict; the government model (UK AISI, CAISI, EU AI Office) has depth but publishes little and is exposed to political redirection. The insurer-routed and public-ratings models are untried.

Three design options the data point to, in increasing order of departure from current practice: (1) accredited lab-paid evaluation with a registry and inspection of evaluators, on the PCAOB and NRTL pattern; (2) mandatory publication of evaluator operating conditions per engagement, on the trial-registration pattern, which AEF-1 already specifies and no regulator yet requires; (3) an evaluator funded by parties exposed to model failure, on the boiler-insurance or IIHS pattern.

7. Limitations

Triggers were selected with hindsight in v0: the lag results describe a record biased toward responsiveness, and v0.2's outcome-independent criterion has not yet been applied to enumerate qualifying incidents that were followed by nothing. Until it is, Section 5.2 is a description of the coded record, not a finding about how regimes behave. Path similarity is computed over short sequences and coarse codings, over openings only, and two regimes tie for nearest — it is a hypothesis generator. Mechanism coding is single-coder and categorical; the supplements match in 5.9 rests on five coarse cells.

Dataset B uses year granularity and secondary sources in v0; ordering within a year is lost, and same-year trigger-rule pairings at lag 0 may not reflect the actual order of events. Regime selection is judgmental; the four added regimes reduce selection on success but do not remove it. Coding milestone kinds involves judgment at the margins (PDUFA is coded as a payer shift toward the assessed party rather than an independence reform). Frontier AI is the only regime observed prospectively, which flatters its speed relative to regimes whose early histories are reconstructed. Dataset A scores public records only; private contract terms could raise or lower any score, and evaluators can move their scores by publishing their terms. The scores are ordinal projections over anchored rubrics, not interval measurements — a 10-point gap is not twice the independence. Version 0 is single-coder, and much of the public record is self-published by the evaluators themselves.

The authors have relationships in the AI investment and research ecosystem, disclosed in the repository, which is the same disclosure this paper asks of evaluators.

8. Conclusion

Frontier AI has compressed into four years a passage other industries took decades to make, and it has done so by adopting standards, mandates, and access rules while leaving the evaluators unwatched and independence rules untested. The history says failures come first and rules follow within a few years; AI has had its failures. The choice now is which of the three historical answers to the payer problem the field adopts, and whether it builds the oversight layer before or after the next incident. The datasets are open and rebuild from their evidence; we invite evaluators to change their scores by publishing their terms.

Figures and tables

• Figure 1. Stage ladder: paper/figures/stage-ladder.svg (generated by python -m bench timeline).
• Figure 2. Path strips with similarity to AI: paper/figures/paths.svg.
• Figure 3. Trigger-to-rule lag against instrument strength: paper/figures/responses.svg.
• Figure 4. Mechanism matrix: paper/figures/mechanisms.svg.
• Table 1. Trigger-to-rule lags: python -m bench industries.
• Table 2. Dataset A scorecard under four presets: python -m bench scores <preset>.
• Table 3. Per-dimension means by organization type: python -m bench scores --by-type (see paper/NOTES.md for the pre-release numbers).

References

See paper/related-work.md. Dataset A sources are in data/sources/; Dataset B sources are inline in data/industries/.