# AI code review benchmark for agent-written bugs

> Canonical page: https://codevetter.com/benchmark

CodeVetter's public v1 benchmark is a reproducible recognition benchmark for agent-written bugs. It publishes 27 synthetic cases, 29 labeled findings, reviewer outputs, scoring rules, downloads, and explicit limitations. It answers whether the tested review pipeline recognizes these known issues under this fixture set. It does not prove that CodeVetter verifies arbitrary production pull requests.

## Read the result in context

The page should keep the current catch rate, precision, F1, false-positive accounting, case explorer, and raw baseline visible. The first screen should distinguish three layers:

1. **Published now:** the 27-case recognition benchmark and its exact results.
2. **Implemented qualification infrastructure:** task packages, executable checks, receipts, and evaluation contracts described by the repository.
3. **Not yet proven publicly:** repeated provider runs on a broad realistic corpus with reliability, cost, and latency comparisons.

## Choose the benchmark for the question

Recognition fixtures help isolate issue-type coverage and scoring behavior. Repository-task benchmarks such as SWE-bench evaluate whether a generated patch resolves a real issue inside a reproducible environment. A product-specific agent benchmark should additionally preserve the requested task, exact patch, environment, checks, evidence, and uncertainty. No single score answers all three questions.

## Inspect, do not just trust

Keep direct links to the downloadable dataset, per-case X-ray pages, outputs, and methodology. Explain how a reader can reproduce the scorer. Surface the limitations before the CTA: synthetic cases, narrow languages and categories, no timing or cost dimension, and precision as a material constraint.

## Where CodeVetter is going

Link the benchmark to the new coding-agent verification pillar and workflow. The next credible proof is not a larger marketing claim. It is qualified, repeated agent-task evidence with immutable receipt linkage and clear failure taxonomy.

**CTA:** Explore all cases, download the dataset, reproduce the scorer, then read the verification methodology.

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)