evaluator; distance from a lab: unattributed
US CAISI (NIST)
Scorecard
Pre-deployment agreements with five labs; 40+ evaluations; some classified via TRAINS taskforce.
Government, Gaithersburg, US. Government institute; listed with government institutes. Confidence high. Domains: cyber, bio, misuse, assurance.
No floor triggered 7/8 evidenced · Retrieved & confirmed Scores by preset: Lab procurement –; Regulator or auditor –; Public trust –; Equal weights –. Weakest evidenced dimension: method transparency 2/4. No evidenced dimension below 2 — not a pass. No floor triggered means no disqualifying conflict is on file, not a pass. Partial evidence (7/8).
The weighted numbers are hidden until you ask for them: two readers with different priorities see different numbers, and neither is the site's.
What would move the score. Regular public reporting per model and a standing budget line insulated from reorganization.
- Funding 3 to 4: score –
- Governance 3 to 4: score –
- Personnel 3 to 4: score –
Dissent, lower. Publication (R) at 2 could be 1. For the five partner labs, no per-model evaluation has been published; the public write-ups concern a foreign model tested without cooperation and two vulnerability disclosures the labs had already fixed. CSA records "no mandatory disclosure of findings". Under R.4, a public body that publishes nothing per model for the labs it has agreements with scores 1, and the DeepSeek report is the exception rather than the practice.
Dissent, higher. Publication (R) at 2 could be 3. CAISI has published a full per-model evaluation (DeepSeek V4 Pro), named vulnerabilities in ChatGPT Agent and Anthropic's Constitutional Classifiers, and states that it publishes findings from its assessments. No lab-edited summary or lab redaction is documented. That is closer to anchor 3 (publishes, redaction limited to security) than to the anchor-2 picture of broad lab redaction.
Each value is the tightest admissible cap or, with no cap, the highest admissible floor, under RULES.md. The binding signal is highlighted. A dash means no admissible signal sets a bound under the Retrieved & confirmed policy.
Funding 3/4
Anchor 3: Mostly philanthropic or public money; some lab-linked pooled funds.
evidence: tier 1 (filing/index)
Capped at 3 by caisi.10 (F.9): Public appropriation of about $10M in FY2026; free pre-deployment access from labs is in-kind. No confirmed primary-filing negative on lab money is on file.. Floors up to 3 from caisi.01, caisi.17 do not exceed the cap.
Curator: Anchor 3 pending a confirmed bounded negative (appropriations text); access from labs is in-kind.
- Public funding; formal access agreements with five developers. floors at 3 by F.9“CAISI, which operates within NIST at the Department of Commerce”
CAISI Frontier Testing Agreements Reach Five Labs (Cloud Security Alliance, 2026-05-20) tier 4 press (primary)unaudited Center for AI Standards and Innovation (NIST, 2026-08-08) tier 3 self (caisi)confirmed2026-09-14 - Public appropriation of about $10M in FY2026; free pre-deployment access from labs is in-kind. No confirmed primary-filing negative on lab money is on file. caps at 3 by F.9binding“As of FY2026, CAISI has approximately $15 million in funding: $10 million from FY2026 appropriations”
Funding for CAISI (Institute for Progress, 2026) tier 4 press (primary)confirmed Center for AI Standards and Innovation (NIST, 2026-08-08) tier 3 self (caisi)confirmed2026-09-14 - The FY2026 Commerce-Justice-Science joint explanatory statement provides no less than $55,000,000 for NIST's AI research and measurement science and, within it, up to $10,000,000 to expand NIST's AI efforts through the U.S. Center for AI Standards and Innovation: appropriated public money. floors at 3 by F.9“up to $ I 0,000,000 is to expand on N IST's A l e ffo 11s th ro ugh the U.S. Cente r fo r A l Standards and Innovation”
Division A - Commerce, Justice, Science, and Related Agencies Appropriations Act, 2026: joint explanatory statement (PDF) (U.S. House of Representatives, docs.house.gov (Committee on Appropriations), 2026-01-05) tier 1 filingconfirmed Funding for CAISI (Institute for Progress, 2026) tier 4 press (primary)confirmed Award search API, spending_by_award (POST), keywords 'Center for AI Standards and Innovation' / 'CAISI' / 'AI Standards and Innovation' (USAspending.gov, 2026-09-15) tier 1 indexconfirmed U.S. AI Safety Institute Signs Agreements Regarding AI Safety Research, Testing and Evaluation With Anthropic and OpenAI (NIST, 2024-08-29) tier 3 self (caisi)confirmed2026-09-15 - Document request: Add a confirmed bounded negative from a primary source to restore anchor 4.
Governance 3/4
Anchor 3: Nonprofit or public body with a COI policy.
evidence: tier 1 (filing/index)
Capped at 3 by caisi.09 (G.5): Appropriation of about $10M in FY2026 within NIST's AI line; the director resigned in July 2026 and the NIST director is acting head.; capped at 3 by caisi.11 (G.5): The CAISI director resigned on 20 July 2026, three months after appointment — the third leadership change in six months.. Floors up to 3 from caisi.15 do not exceed the cap. Not counted under this policy: caisi.03 (no confirmed source (unaudited)).
- Refocused in 2025 from broad safety research to demonstrable national-security risks; mandate is politically steerable. caps at 3 by G.5not counted under standard“repositioned the center as CAISI, shifting emphasis toward national security and cybersecurity risk reduction”
CAISI Frontier Testing Agreements Reach Five Labs (Cloud Security Alliance, 2026-05-20) tier 4 press (primary)unaudited2026-09-14 - Appropriation of about $10M in FY2026 within NIST's AI line; the director resigned in July 2026 and the NIST director is acting head. caps at 3 by G.5binding“$10 million from FY2026 appropriations”
Funding for CAISI (Institute for Progress, 2026) tier 4 press (primary)confirmed evaluators.csv (24 evaluators: Coefficient and SFF totals, lab money, government contracts, leadership) (Kevin Bass (GitHub), 2026-09-14) tier 2 ledgerimported2026-09-14 - The CAISI director resigned on 20 July 2026, three months after appointment — the third leadership change in six months. caps at 3 by G.5binding“resigned from his role as director just three months after he was picked for the job”
What Will It Cost for the US to Be Ready for the Next Big AI Breakthrough? (Institute for Progress, 2026) tier 4 press (primary)confirmed Trump's head of AI safety agency CAISI resigns after months on job (CNBC, 2026-07-20) tier 4 press (primary)confirmed2026-07-20 - CAISI is a NIST center whose staff are bound by the federal conflict-of-interest statute, 18 U.S.C. 208. floors at 3 by G.2“18 U.S. Code § 208 - Acts affecting a personal financial interest”
18 U.S. Code § 208 - Acts affecting a personal financial interest (Cornell Legal Information Institute (U.S. Code), 2026) tier 1 filingconfirmed Center for AI Standards and Innovation (NIST, 2026-08-08) tier 3 self (caisi)confirmed2026-09-15
Personnel 3/4
Anchor 3: Recusal policy and disclosure of lab ties.
evidence: tier 1 (filing/index)
Floored at 3 by caisi.14 (P.5): CAISI staff are NIST federal employees bound by the criminal conflict-of-interest statute (18 U.S.C. 208) and the post-employment statute (18 U.S.C. 207), which restricts former employees for one to two years after leaving.. No admissible signal caps it.
- Authorized Scale AI, a company 49% owned by Meta, to run evaluations on its behalf. informational (Section 9 (mis-dimensioned fact): delegating evaluations to a lab-owned contractor is not a personnel fact; none of the P anchors (board seats, equity, advisory roles, hiring) describes it. Informational on P; re-recorded on X as new signal caisi.16 (X.6).)“Scale AI SEAL : Safety, Evaluation, and Alignment Lab; first third-party evaluator authorized by US AISI”
Why the U.S. Needs an Independent AI Evaluation Framework for National Security (Scale AI, 2026-05-20) tier 3 self (scale)confirmed Third-Party Model Auditing (Longterm Wiki, 2026-01-29) tier 4 press (aggregator)confirmed Scale AI not winding down following Meta deal, interim CEO says (CNBC, 2025-06-18) tier 4 press (primary)confirmed2026-09-14 - CAISI staff are NIST federal employees bound by the criminal conflict-of-interest statute (18 U.S.C. 208) and the post-employment statute (18 U.S.C. 207), which restricts former employees for one to two years after leaving. floors at 3 by P.5binding“within 1 year after the termination of his or her service or employment as such officer or employee”
18 U.S. Code § 208 - Acts affecting a personal financial interest (Cornell Legal Information Institute (U.S. Code), 2026) tier 1 filingconfirmed 18 U.S. Code § 207 - Restrictions on former officers, employees, and elected officials (Cornell Legal Information Institute (U.S. Code), 2026) tier 1 filingconfirmed Center for AI Standards and Innovation (NIST, 2026-08-08) tier 3 self (caisi)confirmed2026-09-15
Access depth (lab-granted) 3/4
Anchor 3: Helpful-only or weights-level access, chain of thought, logs, on-site.
evidence: tier 4 (press)
mechanism: lab-controlled
Floored at 3 by caisi.12 (A.4): Five labs now give CAISI pre-deployment access (OpenAI and Anthropic since September 2025; Google DeepMind, Microsoft and xAI from 5 May 2026), including versions with guardrails stripped back; more than 40 evaluations completed, one of a foreign model without developer cooperation.. No admissible signal caps it. Not counted under this policy: caisi.02 (no confirmed source (unaudited)).
- More than 40 evaluations including unreleased models; classified CBRN and cyber work via the TRAINS taskforce. floors at 3 by A.4not counted under standard“completed more than 40 evaluations — including assessments of frontier models not yet available to the public”
CAISI Frontier Testing Agreements Reach Five Labs (Cloud Security Alliance, 2026-05-20) tier 4 press (primary)unaudited2026-09-14 - Five labs now give CAISI pre-deployment access (OpenAI and Anthropic since September 2025; Google DeepMind, Microsoft and xAI from 5 May 2026), including versions with guardrails stripped back; more than 40 evaluations completed, one of a foreign model without developer cooperation. floors at 3 by A.4binding“provide CAISI with model access that includes versions with safety guardrails stripped back”
CAISI Frontier Testing Agreements Reach Five Labs (Cloud Security Alliance (research note), 2026-05-05) tier 4 press (primary)confirmed2026-05-05
Scope control 3/4
Anchor 3: Evaluator sets scope and can add questions.
evidence: tier 3 (self)
Capped at 3 by caisi.13 (S.4): Participation is voluntary and agreements were jointly scoped with the two founding labs; the center can only assess models developers choose to share.. Floors up to 3 from caisi.06 do not exceed the cap.
- Government defines risk domains. floors at 3 by S.4“The government defines the risk domains it wants evaluated”
Why the U.S. Needs an Independent AI Evaluation Framework for National Security (Scale AI, 2026-05-20) tier 3 self (scale)confirmed2026-09-14 - Participation is voluntary and agreements were jointly scoped with the two founding labs; the center can only assess models developers choose to share. caps at 3 by S.4binding“CAISI’s authority to conduct these evaluations rests entirely on voluntary participation”
CAISI Frontier Testing Agreements Reach Five Labs (Cloud Security Alliance (research note), 2026-05-05) tier 4 press (primary)confirmed2026-05-05
Publication rights –
No admissible evidence under the standard policy.
With leads included: 2/4
evidence: tier 4 (press)
mechanism: statutory
Unevidenced under this policy: no admissible signal sets a bound. Not counted: caisi.04 (no confirmed source (unaudited)).
- Publishes little per model; findings mostly stay inside government. caps at 2 by R.4not counted under standard“voluntary agreements, unclassified evaluations, no mandatory disclosure of findings”
CAISI Frontier Testing Agreements Reach Five Labs (Cloud Security Alliance, 2026-05-20) tier 4 press (primary)unaudited2026-09-14
Method transparency 2/4
Anchor 2: Methods described in prose.
evidence: tier 3 (self)
Capped at 2 by caisi.07 (M.2): Methods partly published through NIST; per-evaluation methods not public..
- Methods partly published through NIST; per-evaluation methods not public. caps at 2 by M.2binding“CAISI published a write-up on how AI models can cheat on agentic evaluations”
Center for AI Standards and Innovation (NIST, 2026-08-08) tier 3 self (caisi)confirmed2026-09-14
Role incompatibility 3/4
Anchor 3: Tools are open or free to the ecosystem.
evidence: tier 3 (self)
floor and cap disagree
Conflict: floor 4 from caisi.08 (X.7) against cap 3 from caisi.16 (X.6); resolved at 3 by X.6: The specific co-production rule (X.6) decides over the general legal-form floor: 3.
Resolution: X.6 decides 3. The specific co-production rule (X.6) decides over the general legal-form floor: 3.
- No commercial products. floors at 4 by X.7“lead unclassified evaluations of AI capabilities that may pose risks to national security”
Center for AI Standards and Innovation (NIST, 2026-08-08) tier 3 self (caisi)confirmed CAISI Frontier Testing Agreements Reach Five Labs (Cloud Security Alliance (research note), 2026-05-05) tier 4 press (primary)confirmed2026-09-14 - CAISI partnered with Scale AI in February 2025 to jointly develop evaluation methods for frontier models, and authorized Scale SEAL as a third-party evaluator; Meta has held 49% of Scale AI since June 2025. caps at 3 by X.6binding“Scale partnered with the U.S. Center for AI Standards and Innovation (CAISI), jointly developing new evaluation methods”
Why the U.S. Needs an Independent AI Evaluation Framework for National Security (Scale AI, 2026-05-20) tier 3 self (scale)confirmed Scale AI not winding down following Meta deal, interim CEO says (CNBC, 2025-06-18) tier 4 press (primary)confirmed Third-Party Model Auditing (Longterm Wiki, 2026-01-29) tier 4 press (aggregator)confirmed2026-09-15
Values under each evidence policy
| Dimension | Leads included | Retrieved & confirmed | Against interest | Verified spans | Primary only |
|---|---|---|---|---|---|
| Funding | 3 | 3 | 3 | 3 | 3 |
| Governance | 3 | 3 | 3 | 3 | 3 |
| Personnel | 3 | 3 | 3 | 3 | 3 |
| Access depth (lab-granted) | 3 | 3 | 3 | 3 | – |
| Scope control | 3 | 3 | 3 | 3 | – |
| Publication rights | 2 | – | – | – | – |
| Method transparency | 2 | 2 | 2 | 2 | – |
| Role incompatibility | 3 | 3 | 3 | 3 | – |
| Score, Lab procurement | – No floor triggered, 8/8 | – No floor triggered, 7/8 | – No floor triggered, 7/8 | – No floor triggered, 7/8 | – No floor triggered, 3/8 |
| Score, Regulator or auditor | – No floor triggered, 8/8 | – No floor triggered, 7/8 | – No floor triggered, 7/8 | – No floor triggered, 7/8 | – No floor triggered, 3/8 |
| Score, Public trust | – No floor triggered, 8/8 | – No floor triggered, 7/8 | – No floor triggered, 7/8 | – No floor triggered, 7/8 | – No floor triggered, 3/8 |
| Score, Equal weights | – No floor triggered, 8/8 | – No floor triggered, 7/8 | – No floor triggered, 7/8 | – No floor triggered, 7/8 | – No floor triggered, 3/8 |
Traced money and ties
public: 2 rows (grant $25.0M). Confirmed rows: 1 of 2 Dollar sums: confirmed + unaudited USD rows only; imported figures are not re-derived and never summed. Second hop traced for 1 of 1 sources. Ties: none within two steps recorded. Checked and not found: 3.
Ledger
Money in
- T47 from US government (NIST appropriations): grant $10.00M, FY2026. appropriation within the NIST AI line confirmedtier 4 presshttps://ifp.org/funding-for-caisi/
- T79 from US government (NIST appropriations): grant $15.00M, FY2026. about $15M available: $10M appropriated plus a Technology Modernization Fund loan unauditedtier 4 presshttps://ifp.org/funding-for-caisi/
Checked and not found
- N09 Does Coefficient Giving fund US CAISI?
Searched Coefficient grants index, 2,911 rows, snapshot 2026-09-11: no grant from Coefficient Giving in its index. Prompted by imported lead: kevinnbass ledger row EV12. importedhttps://coefficientgiving.org/grants/ - N18 Does CAISI pay private evaluators such as METR or Apollo?
Searched CAISI announcements and spending search, snapshot 2026-09-14: no CAISI contract to METR, Apollo, or other private evaluators found. Prompted by imported lead: kevinnbass ledger row EV12. importedhttps://www.nist.gov/caisi - N26 no award or agreement naming the Center for AI Standards and Innovation, CAISI, or 'AI Standards and Innovation' in USAspending's award search (FY2024 to 2026-09-15, all award types)
Searched USAspending spending_by_award API, three keyword queries, all award types; NIST news releases on CAISI agreements, snapshot 2026-09-15: no award or agreement naming the Center for AI Standards and Innovation, CAISI, or 'AI Standards and Innovation' in USAspending's award search (FY2024 to 2026-09-15, all award types). confirmedhttps://api.usaspending.gov/api/v2/search/spending_by_award/
One hop away
Funding graph, focused
Every one-way chain to a lab and to an evaluator at full strength, everything else faded. Hover any name to trace its connections; click a name to open its page.