Evaluator Bench
Scale AI, 2026-05-20; retrieved 2026-09-14 tier 3 self (scale)confirmed

Why the U.S. Needs an Independent AI Evaluation Framework for National Security

https://scale.com/blog/ai-evaluation-framework-national-security

Scale partnered with CAISI Feb 2025; runs evaluations for governments. Cited from a search snippet; not re-fetched in full. [verify 2026-09-15] fetched 2026-09-15; page loads, citation matches

Signals citing this source

  • US CAISI (NIST), P (against): Authorized Scale AI, a company 49% owned by Meta, to run evaluations on its behalf.
  • US CAISI (NIST), S (for): Government defines risk domains.“The government defines the risk domains it wants evaluated”
  • US CAISI (NIST), X (against): CAISI partnered with Scale AI in February 2025 to jointly develop evaluation methods for frontier models, and authorized Scale SEAL as a third-party evaluator; Meta has held 49% of Scale AI since June 2025.“Scale partnered with the U.S. Center for AI Standards and Innovation (CAISI), jointly developing new evaluation methods”
  • Scale AI (SEAL / Scale Labs), F (against): Nearly half owned by Meta, itself a frontier developer.
  • Scale AI (SEAL / Scale Labs), G (against): Primary business is selling data and evaluation services to the labs; no public COI policy addressing the Meta relationship.“Scale has worked with the world's leading AI model builders”
  • Scale AI (SEAL / Scale Labs), A (for): Government-defined risk domains and access agreements; first private evaluator authorized by the US institute.
  • Scale AI (SEAL / Scale Labs), X (against): Sells data and evaluation services to labs.
  • Scale AI (SEAL / Scale Labs), S (against): Scope set by government or lab clients.“The government defines the risk domains it wants evaluated”
  • Scale AI (SEAL / Scale Labs), R (against): Results for governments not public.“The government provides subject matter expertise and interprets the results for policy and standards”
  • Scale AI (SEAL / Scale Labs), M (against): Benchmarks partly public (SEAL leaderboards); government evaluations closed.“Scale Labs builds the benchmarks and conducts the technical testing”

Ledger rows citing this URL

  • none