Public Benchmark v1 · 2026-07-17
AI code review benchmark for agent-written bugs
An AI code-review benchmark is useful only when you can inspect what was tested, what counted as a correct finding, how false positives were scored, and what the result does not prove. CodeVetter's public v1 is a reproducible recognition benchmark: 27 synthetic cases, 29 hand-labeled findings, published reviewer outputs, and a scorer you can run locally. It is not evidence that a reviewer will perform the same way on a large production pull request.
Read the proof at the right level
- Published nowThe exact 27-case recognition corpus, results, scorer, outputs, and limitations below.
- ImplementedTask packages, executable checks, evidence receipts, and evaluation contracts in the repository.
- Not yet provenRepeated provider runs across a broad realistic corpus with reliability, cost, and latency comparisons.
New to the method? Start with coding-agent verification, then follow the step-by-step workflow or inspect the evidence bundle.
Raw Claude = the same model (Claude) prompted once with “review this code,” no review harness, no rubric, no diff context. CodeVetter wraps that model in its review pipeline (rubric, diff framing, finding normalization). The point of the baseline is to isolate what the harness adds, not to dunk on the model.
Which benchmark design answers your question?
| Benchmark design | Good for | Cannot establish alone |
|---|---|---|
| Isolated, hand-labeled cases | Reproducibility, issue-type coverage, and precise false-positive accounting | Performance inside a large repository or noisy pull request |
| Replayed real bug-fix pull requests | Reviewing realistic diffs with repository context | A universal ranking when judges, tools, or repositories change |
| Human-review-derived executable tasks | Testing issues human reviewers found and tests can encode | Coverage beyond the sampled reviews and generated tests |
Research datasets such as c-CRAB derive executable tasks from human reviews. Published replay benchmarks such as Tenki's use real bug-introducing pull-request diffs and model judges. Those methodologies answer different questions from CodeVetter's small synthetic corpus, so their headline percentages are not directly comparable.
How to read any AI code-review score
- Inspect the corpus. Count repositories, languages, issue classes, and whether cases are synthetic or production-derived.
- Inspect the ground truth. Human labels, regression tests, and model judges encode different evidence and failure modes.
- Read recall with precision. More caught bugs do not help if false positives make the review unusable.
- Check the unit of scoring. Per-finding, per-pull-request, and per-task scores are not interchangeable.
- Look for artifacts. Cases, outputs, scoring rules, and limitations should be exposed closely enough to reproduce or challenge the result.
Where CodeVetter v1 fits
CodeVetter v1 optimizes for transparency: every case is self-contained, every expected finding is published, and false positives and redundant matches count against precision. That makes it fast to rerun and easy to audit. The tradeoff is a small, synthetic corpus, mostly one finding per case, labeled by one person, with no latency or cost measurement. Use it as an inspectable recognition check; use the separate real-agent-PR evaluation work for repository-scale outcome claims.
What's in the dataset
- 27 cases, each a single self-contained source file with a known issue.
- 29 hand-labeled expected findings across those cases (two cases carry two findings each).
- 6 languages: go, java, javascript, python, rust, typescript.
- 4 categories: bug, concurrency, maintainability, security.
- Every case is synthetic and reproducible — no proprietary code, no network calls, no repo setup.
Methodology
- Each case lives in
benchmark/cases/<id>/with asource.<ext>and a hand-writtenlabel.jsonground truth (type, severity, line range, description). - A reviewer's output is normalized into
reviews/<case-id>.<reviewer>.jsonwith amatched_ground_trutharray per finding. - The scorer (
scripts/run-public-benchmark.mjs) computes catch rate, precision, F1, false positives, and redundant matches. - Catch rate = matched ground-truth issues ÷ total expected. Precision = matched ÷ (matched + false positives + redundant). F1 = harmonic mean.
- Findings with empty
matched_ground_truthcount as false positives; repeats of an already-caught issue count as redundant.
Reproduce: git clone → npm run bench:public -- --reviewer=codevetter
Per-reviewer results by case
Expected = labeled findings. Caught = matched. FP = false positives. Red = redundant matches.
| Case | Exp | CV ✓ | CV FP | RC ✓ | RC FP |
|---|---|---|---|---|---|
| go/go-errcheck | 1 | 1 | 0 | 1 | 0 |
| go/go-hardcoded-credentials | 1 | 1 | 2 | 1 | 1 |
| go/go-nil-pointer | 1 | 1 | 1 | 1 | 0 |
| go/go-race-condition | 1 | 1 | 1 | 1 | 0 |
| go/go-sql-injection | 1 | 1 | 1 | 1 | 2 |
| java/java-insecure-random | 1 | 1 | 2 | 1 | 2 |
| javascript/js-eval-injection | 1 | 1 | 1 | 1 | 2 |
| javascript/js-open-redirect | 1 | 1 | 1 | 1 | 0 |
| python/py-bare-except | 1 | 1 | 1 | 1 | 3 |
| python/py-command-injection | 1 | 1 | 3 | 1 | 3 |
| python/py-hardcoded-secret | 1 | 1 | 4 | 1 | 3 |
| python/py-insecure-deserialization | 1 | 1 | 1 | 1 | 2 |
| python/py-path-traversal | 1 | 1 | 1 | 1 | 0 |
| python/py-sql-injection | 1 | 1 | 0 | 1 | 0 |
| python/py-ssrf | 1 | 1 | 1 | 1 | 3 |
| python/py-weak-hash | 1 | 1 | 2 | 1 | 1 |
| python/py-zip-bomb | 1 | 1 | 0 | 0 | 1 |
| rust/rust-integer-overflow | 2 | 2 | 1 | 1 | 0 |
| typescript/ts-dead-code | 2 | 2 | 1 | 2 | 0 |
| typescript/ts-hardcoded-credentials | 1 | 1 | 1 | 1 | 1 |
| typescript/ts-insecure-cookie | 1 | 1 | 0 | 1 | 0 |
| typescript/ts-missing-await | 1 | 1 | 0 | 1 | 1 |
| typescript/ts-prototype-pollution | 1 | 1 | 2 | 1 | 3 |
| typescript/ts-regex-dos | 1 | 1 | 1 | 1 | 2 |
| typescript/ts-sql-injection | 1 | 1 | 1 | 1 | 0 |
| typescript/ts-type-confusion | 1 | 1 | 0 | 1 | 1 |
| typescript/ts-xss | 1 | 1 | 0 | 1 | 0 |
Red = a missed finding. Raw Claude missed one finding each in py-zip-bomb (resource exhaustion) and rust-integer-overflow (one of two overflow sites). CodeVetter caught all 29.
The 27 cases
| Case | Language | Category | Issue type | Sev |
|---|---|---|---|---|
| go-errcheck | go | bug | unchecked_error | medium |
| go-hardcoded-credentials | go | security | hardcoded_secret | high |
| go-nil-pointer | go | bug | nil_dereference | medium |
| go-race-condition | go | concurrency | race_condition | high |
| go-sql-injection | go | security | sql_injection | high |
| java-insecure-random | java | security | insecure_random | high |
| js-eval-injection | javascript | security | code_injection | high |
| js-open-redirect | javascript | security | open_redirect | medium |
| py-bare-except | python | bug | swallowed_error | medium |
| py-command-injection | python | security | command_injection | high |
| py-hardcoded-secret | python | security | hardcoded_secret | high |
| py-insecure-deserialization | python | security | insecure_deserialization | high |
| py-path-traversal | python | security | path_traversal | high |
| py-sql-injection | python | security | sql_injection | high |
| py-ssrf | python | security | ssrf | high |
| py-weak-hash | python | security | weak_crypto | high |
| py-zip-bomb | python | security | resource_exhaustion | high |
| rust-integer-overflow | rust | bug | integer_overflow | high |
| ts-dead-code | typescript | maintainability | dead_code | medium/low |
| ts-hardcoded-credentials | typescript | security | hardcoded_secret | high |
| ts-insecure-cookie | typescript | security | insecure_cookie | high |
| ts-missing-await | typescript | bug | missing_await | medium |
| ts-prototype-pollution | typescript | security | prototype_pollution | high |
| ts-regex-dos | typescript | security | regex_dos | high |
| ts-sql-injection | typescript | security | sql_injection | high |
| ts-type-confusion | typescript | bug | type_confusion | medium |
| ts-xss | typescript | security | xss | high |
Honest limitations
- Synthetic, not real PRs. Cases are hand-written snippets, not diffs mined from production repos. They test whether a reviewer recognizes a known issue in isolation, not whether it surfaces inside a 400-line PR with unrelated churn. The sibling
benchmarks/agent-prs/harness measures the latter on real public agent PRs. - Small. 27 cases / 29 findings is enough to be reproducible and to separate 100% from 93%, but not enough to claim statistical significance between two reviewers that both sit near the top. Treat the gap as directional, not proven.
- One finding per case (mostly). Two cases carry two findings; the rest carry one. This over-weights breadth of issue type vs. depth of reasoning inside a single file.
- Precision is the weak spot for both reviewers. CodeVetter emits 29 false positives across the set; raw Claude emits 31. Catch rate is the headline, but a reviewer that shouts on every line would also hit 100% catch rate — which is why precision and F1 are reported alongside it.
- Ground truth is human-written and could be wrong. Labels were written by one person. A missed “real” issue that isn't in the label would inflate catch rate. PRs that add or correct a label are welcome.
- No timing / cost dimension. The benchmark scores correctness only. It does not account for tokens, latency, or provider cost, which matter in practice.
Download the dataset
The full dataset — all 27 cases, source code, expected findings, scoring method, and a CC0 license note — is a single JSON file (~28 KB).
codevetter-benchmark-v1.jsonLicense: CC0 1.0 (Public Domain Dedication). Attribution appreciated, not required.
Reproduce it yourself
git clone https://github.com/Codevetter/codevetter
cd codevetter
npm run bench:public -- --reviewer=codevetter
npm run bench:public -- --reviewer=raw-claude