Contribute

If you are a person

• Ten minutes. Open any evaluator page, follow a source link, and check that the quoted span is there. If it is not, open an issue with the signal id.
• An hour. Confirm an imported ledger row: open its source, compare the figure, set audit_status to confirmed or differs, add a line to the audit file, open a pull request.
• An afternoon. Add evidence: a source you fetched yourself, a signal with a quote under 120 characters copied exactly, the anchor it supports as a bound, and the rule code that says so.
• If you work at an evaluator or a lab. You are welcome to contribute; say so in the pull request. Evaluators can publish their contract terms and the relevant dimensions move on their own. To file a formal response, use the right-of-reply issue template; it is filed as a signal with the date received.
• If you are named as a person. The same reply channel is yours. Bench records public roles only and asserts no motive; an open question about a role or a gift is a request for a document. People about whom the bench publishes a score-affecting claim are contacted about their card as the release goes public (RULES 12); everyone else named is covered by this reply channel. Until the outreach log shows a contact date, treat the card as provisional.

Full recipes and rules of evidence: AGENTS.md, CONTRIBUTING.md, CONTRACT.md, RULES.md.

What happens to a pull request

• CI runs the verifier (the CONTRACT), rebuilds the graph and site, checks the committed event log matches a clean build, and runs the tests. The verifier rejects a stored value that disagrees with its derivation, a signal without a bound, and a role that disagrees with the rule.
• A machine review re-fetches every source the PR cites, checks that each quoted span appears verbatim, asks a model whether each claim is supported, qualified, or unsupported, and flags any assessment change with no new signal on that dimension. The report is posted on the PR.
• A maintainer reads the report and merges or asks for changes. Merged evidence appears on the site at the next build and in the next annual update paper.

If you are pointing an agent here

Copy this instruction to a coding agent with repository access:

Open https://github.com/yoheinakajima/evaluator-bench and read AGENTS.md and RULES.md. Choose one recipe (add evidence about an evaluator, confirm an imported ledger row, or draft a docket). Fetch every source yourself in this run, copy quoted spans exactly, give every new signal a bound and the RULES.md rule code behind it, run python -m bench verify && python -m bench build && pytest -q, commit the regenerated graph/ and dist/, and open a pull request using the template. Do not change values, anchors, rules, or the verifier. If you cannot complete a recipe without guessing, stop and report what is missing.

The agent must fetch sources itself, never sum across money measures, never turn a bounded absence into zero, name public roles only, and assert no motive. Its pull request is a queue item, not accepted evidence, until the checks and a maintainer pass it.

Contested claims

Some claims here will be contested. We draft the contestable ones as proposals for independent review by a third party, Epistemedia, where a reviewer other than the drafter checks each quoted span; a draft is not evidence and none has been reviewed yet. The drafts and how review would work.

What we will not accept

Private communications, screenshots of paywalled pages, claims about a person's intent, non-public individuals, value edits without a signal, hypothetical organizations, and deletions of rows (supersede them instead).