GitHub has released ReviewBench, a public test set for AI tools that inspect proposed code changes. Its 219 pull requests span 19 programming languages, and the scoring system is designed to recognize a valid issue even when it was missing from the benchmark’s original answer key.
That matters because a code-review benchmark can undercount a capable system if its reference list is incomplete. ReviewBench scores findings against known issues, then separately asks a language-model judge to assess a reviewer’s unmatched findings. A reviewer can therefore earn credit for a new, valid problem rather than having it automatically treated as a false alarm.
GitHub assembled the reference findings from human code reviews, issues inferred from authors’ follow-up commits, static-analysis tools and several language models. It merged findings about the same underlying issue and judged them using a shared rubric: a finding must be true, relevant and non-trivial. GitHub says senior engineers who did not build the dataset independently relabeled every reference finding; their true-or-false judgments agreed with the benchmark labels 96.6% of the time.
Grounded precision asks what share of an agent’s reported issues match a finding already in the reference set; grounded recall asks what share of those known findings the agent catches. The augmented measures also have a language-model judge check unmatched findings, so a valid issue missing from the reference set can still count. Augmented recall’s denominator grows with the additional issues each agent surfaces, making it harder to compare systems directly; GitHub therefore uses grounded recall for its headline cross-system comparison.
The 219 pull requests come from 187 public repositories. GitHub analyzed 103.9 million pull requests to shape the benchmark’s language and repository-size distributions, but that is not the benchmark’s sample size. The set intentionally gives more weight to medium and larger changes than the full distribution, so it is not a random snapshot of every code review on GitHub.
The dataset, evaluation method, judge prompt and configuration, and self-serve runner are available through ReviewBench’s research preview. A developer can submit an agent in a container with their own model key, first test on 25 pull requests, then run all 219 across three rounds. Scores remain private until a maintainer reviews a submission.
GitHub says it uses ReviewBench before product experiments. For one Copilot code-review test, the benchmark predicted that an ensemble combining several independent model runs would improve precision and recall, increase comments and reduce review cost. GitHub says the later A/B test moved in the directions ReviewBench predicted. Compared with the production control, its online addressed-rate measure—the share of comments an LLM judged to prompt a corresponding code change—rose 8.0%. GitHub also reports a 13.6% rise in its online recall measure, which it describes as measuring how much additional human review is still needed. The post does not give that measure's formula; it is distinct from ReviewBench's grounded recall. Comments rose 61%, while cost per review fell 8.0%. These are GitHub-reported results for its own Copilot product, not an independent comparison of outside agents.
Sources
- GitHub’s ReviewBench announcement and methodology
- ReviewBench research preview
- Photo: Nemuel Sereti, Pexels (illustrative stock image)
- Pexels license