Evaluator Bench

Where we are

Every mature high-stakes industry grew a third-party assurance layer, and most of them started the way AI has: voluntary, paid by the assessed party, with the assessed party choosing scope. The ladder marks the year each regime first reached each stage. Tap a cell to see the milestone behind it and its source; tap a regime name for its full history.

Caveat. Dataset B is coded from secondary sources at year granularity, with primary-source verification still to do, and its trigger incidents were selected with hindsight. The lag pattern below is a description of the coded record, not a finding about how regimes behave. The ladder records first arrivals, so a one-clause rule in one state fills a cell the same way Sarbanes-Oxley does; the three views after it keep the order and the strength of each move.

under 10 years into the regime 10 to 50 over 50 not reached rollback assessed party absorbed the assessment Download the static figure

Read financial audit left to right and you get 158 years from the first mandate to the first rule separating auditors from consultants. Read the AI row and you get four. Across regimes, rules have followed public failures within a few years in the coded record; the stretch before the first failure is where a regime's shape is decided. The oversight and independence-rule columns are the ones AI has not filled. Data and method: data/industries/, bench/timeline.py, paper/.

Paths, responses, and mechanisms

Three views that keep the nuance the ladder drops. Hover or tap any glyph, point, or cell to read what it stands for and where it comes from.

The order, not the calendar

Each regime as a sequence of moves. Reversals show up in red, the assessed party absorbing the assessment function in amber. The right column ranks regimes by how closely their opening moves match frontier AI's so far, using edit distance over the sequences. Two regimes tie for nearest, the sequences are short, and the coding is coarse: a hypothesis generator, not a prediction.

Paths: the order each regime did things, ignoring the calendar Consecutive repeats collapsed. Right column: similarity of each regime's opening to frontier AI's path so far (1 = identical, 0 = nothing shared). voluntary trigger mandate standards oversight independence access/publication delegation payer shift rollback Financial statement audit M1844 R1856 V1880 M1900 A1934 T1938 S1939 D1990 T2001 O2002 I2002 similarity 0.36, opening MRVMATSDTOI Ship classification V1760 O1968 T1999 I2009 similarity 0.22, opening VOTI Boilers and pressure vessels V1866 T1905 M1908 S1914 O1919 similarity 0.22, opening VTMSO Electrical and consumer product safety V1894 S1903 M1972 O1988 T2007 M2008 similarity 0.33, opening VSMOTM Pharmaceutical safety and efficacy M1906 T1937 M1938 T1961 D1962 M1962 S1962 T1976 O1978 P1992 I1998 A2007 similarity 0.22, opening MTMTDMS Aircraft certification M1926 T1956 M1958 S1965 D2005 T2018 I2020 similarity 0.22, opening MTMSDTI Credit rating agencies V1909 D1970 T1970 O1975 T2008 I2010 similarity 0.44, opening VDTOTI Nuclear safety and safeguards M1957 A1970 I1974 A1978 V1979 T1979 A1997 similarity 0.22, opening MAIAVTA Automobile crash safety V1959 T1965 M1966 S1967 A1979 similarity 0.33, opening VTMSA Food safety M1906 T1993 M1996 V2000 T2008 O2011 similarity 0.22, opening MTMVTO Cybersecurity assurance S1985 O1999 M2004 O2011 T2013 similarity 0.11, opening SOMOT Sustainability (ESG) reporting assuran S1997 V2003 T2015 M2022 S2024 similarity 0.33, opening SVTMS Dietary supplements (US) R1994 V2001 T2003 A2006 S2007 similarity 0.33, opening RVTAS Crypto exchange proof of reserves V2014 T2014 R2022 T2022 M2023 similarity 0.44, opening VTRTM Platform and hiring-algorithm audits T2018 M2021 A2022 I2023 T2025 similarity 0.22, opening TMAIT Frontier AI evaluation V2022 D2023 V2024 T2025 S2025 A2025 T2026 M2026 A2026 reference path: VDVTSATMA Source: data/industries/. Generated by bench.figures.

Scroll sideways to see the rest of the figure.

Fast is not the same as strong

Every trigger incident paired with the next rule that followed. Most responses arrive within three years, and most are mandates or standards rather than structural changes; the two structural responses in the set are the PCAOB after Enron and lab inspection under Good Laboratory Practice after the IBT fraud. Frontier AI's two responses so far score 2 and 1. Triggers were selected with hindsight, so the speed is a property of the coded record.

After a failure: how fast the next rule came, and how strong it was Each point is one trigger incident. Hollow points: no rule followed. Frontier AI in teal. 1 2 3 4 0 5 10 15 20 years from trigger to next rule instrument strength (1 disclosure to 4 structural) no rule

Scroll sideways to see the rest of the figure.

Who pays, selects, sees, publishes, and watches

The configuration the ladder cannot show. Read the AI row against the others: the only other regime with the same five-cell pattern is dietary supplements, the regime that removed pre-market review by statute. The match is on a coarse categorical coding, so it is suggestive rather than conclusive.

Mechanisms today: the incentive configuration of each regime Red marks the configuration that concentrates control in the assessed party. Hover a cell for the state before reform. Who pays Who selects What they see What the public reads Who watches assessors Financial statement audit assessed party assessed party deep public regulator * Ship classification assessed party assessed party deep summary or mark regulator * Boilers and pressure vessels opposing exposure opposing exposure deep summary or mark regulator * Electrical and consumer product safe assessed party assessed party deep summary or mark regulator * Pharmaceutical safety and efficacy assessed party regulator or state * deep public * regulator * Aircraft certification assessed party regulator or state * deep summary or mark * regulator Credit rating agencies assessed party * assessed party * deep * public regulator * Nuclear safety and safeguards assessed party regulator or state embedded * public regulator Automobile crash safety opposing exposure * opposing exposure * deep public * regulator * Food safety assessed party assessed party deep summary or mark * regulator * Cybersecurity assurance assessed party assessed party deep summary or mark regulator * Sustainability (ESG) reporting assur assessed party assessed party shallow public regulator * Dietary supplements (US) (same as AI) assessed party assessed party shallow summary or mark none Crypto exchange proof of reserves assessed party assessed party shallow public regulator * Platform and hiring-algorithm audits assessed party assessed party deep * summary or mark * regulator * Frontier AI evaluation assessed party assessed party shallow summary or mark none * changed since the regime's origin. Source: data/industries/ mechanisms. Generated by bench.figures.

Scroll sideways to see the rest of the figure.

The sixteen regimes

Sixteen assurance regimes coded as dated milestones. The path signature is the ordered sequence of moves (V voluntary, T trigger, M mandate, S standards, O oversight, I independence, A access or publication, D delegation, P payer shift, R rollback). Similarity compares each regime's opening to frontier AI's path so far.

RegimePathStagesFirst milestoneSimilarity to AIJurisdictionWho pays
Financial statement auditMRVMATSDTOI7/718440.36UK/US/EUAudited company pays the auditor; audit committee hires and fires (post-2002).
Ship classificationVOTI4/717600.22International/EUShipowner pays the classification society.
Boilers and pressure vesselsVTMSO5/718660.22USInsurer employs the inspector; owner pays the insurer.
Electrical and consumer product safetyVSMOTM5/718940.33USManufacturer pays the testing lab.
Pharmaceutical safety and efficacyMTMTDMSTOPIA6/719060.22USSponsor funds trials and, since 1992, pays user fees for review.
Aircraft certificationMTMSDTI4/719260.22USManufacturer pays for certification work, much of it delegated to its own employees.
Credit rating agenciesVDTOTI4/719090.44US/EUIssuer pays since the 1970s; investor-pays subscription model before that.
Nuclear safety and safeguardsMAIAVTA5/719570.22International/USOperators fund INPO; states fund IAEA and national regulators.
Automobile crash safetyVTMSA5/719590.33USInsurers fund IIHS; government funds NCAP; manufacturers do not pay for ratings.
Food safetyMTMVTO4/719060.22USProducer pays private certifiers; government inspection publicly funded.
Cybersecurity assuranceSOMOT4/719850.11International/USVendor or merchant pays the accredited assessor.
Sustainability (ESG) reporting assuranceSVTMS4/719970.33EU/InternationalReporting company pays the assurance provider.
Dietary supplements (US)RVTAS4/719940.33USManufacturer pays private verifiers; verification is voluntary.
Crypto exchange proof of reservesVTRTM3/720140.44International/EUExchange pays the attesting firm.
Platform and hiring-algorithm auditsTMAIT4/720180.22EU/USPlatform or employer pays the auditor.
Frontier AI evaluationVDVTSATMA5/72020refUS/EUMixed: philanthropy, lab fees, government budgets. No accreditation.