Evaluator Bench

Contribute

Scores here move only when evidence moves. A contribution is a source, a signal with an exact quote, a ledger row, a confirmed re-derivation, or a docket. Nobody edits a score directly, including the maintainers.

If you are a person

  • Ten minutes. Open any evaluator page, follow a source link, and check that the quoted span is there. If it is not, open an issue with the signal id.
  • An hour. Confirm an imported ledger row: open its source, compare the figure, set audit_status to confirmed or differs, add a line to the audit file, open a pull request.
  • An afternoon. Add evidence: a source you fetched yourself, a signal with a quote under 120 characters copied exactly, and if it changes an anchor, the new rationale.
  • If you work at an evaluator or a lab. You are welcome to contribute; say so in the pull request. Evaluators can publish their contract terms and the relevant dimensions move on their own. To file a formal response, use the right-of-reply issue template; it is filed as a signal with the date received.

Full recipes and rules of evidence: AGENTS.md, CONTRIBUTING.md, CONTRACT.md.

What happens to a pull request

  • CI runs the verifier (the CONTRACT), rebuilds the graph and site, checks the committed event log matches a clean build, and runs the tests.
  • A machine review re-fetches every source the PR cites, checks that each quoted span appears verbatim, asks a model whether each claim is supported, qualified, or unsupported, and flags any assessment change with no new signal on that dimension. The report is posted on the PR.
  • A maintainer reads the report and merges or asks for changes. Merged evidence appears on the site at the next build and in the next annual update paper.

If you are pointing an agent here

Copy this instruction to a coding agent with repository access:

Open https://github.com/yoheinakajima/evaluator-bench and read AGENTS.md. Choose one recipe (add evidence about an evaluator, confirm an imported ledger row, or draft a docket). Fetch every source yourself in this run, copy quoted spans exactly, run python -m bench verify && python -m bench build && pytest -q, commit the regenerated graph/ and dist/, and open a pull request using the template. Do not change scores, anchors, or the verifier. If you cannot complete a recipe without guessing, stop and report what is missing.

The agent must fetch sources itself, never sum across money measures, never turn a bounded absence into zero, name public roles only, and assert no motive. Its pull request is a queue item, not accepted evidence, until the checks and a maintainer pass it.

Dockets and certification

Contestable claims can be drafted as Epistemedia dockets from Bench evidence (bench docket build), validated with Epistemedia's own validator, and submitted through epistemedia.org. A docket is a draft until it is reviewed there by someone other than the drafter; drafts are not citable evidence and carry no certificate (see Dockets and the certification plan).

What we will not accept

Private communications, screenshots of paywalled pages, claims about a person's intent, non-public individuals, score edits without evidence, and deletions of rows (supersede them instead).