---
title: Catch-rate benchmark
description: Starter harness for measuring whether CodeVetter catches known bugs in agent-generated changes.
---

This is the starter harness for measuring whether CodeVetter catches known bugs in agent-generated changes.

Run:

```bash
npm run bench:catch-rate
npm run bench:catch-rate -- benchmarks/agent-prs/sample.json --reviewer=codevetter
npm run bench:catch-rate -- --reviewer=codevetter --baseline=baseline
npm run bench:catch-rate -- --reviewer=codevetter --baseline=coderabbit_free
npm run bench:catch-rate -- --reviewer=codevetter --baseline=claude_code_review
npm run bench:catch-rate -- --reviewer=codevetter --min-rate=0.8
npm run bench:catch-rate -- --reviewer=codevetter --max-false-positives=1
npm run bench:catch-rate -- --reviewer=codevetter --max-redundant-matches=0
npm run bench:catch-rate -- --reviewer=codevetter --min-severity-rate=high:1
npm run bench:catch-rate -- --reviewer=codevetter --require-rationales
npm run bench:catch-rate -- --json
npm run bench:catch-rate -- --format=markdown --out=artifacts/benchmark.md
npm run bench:new-case -- --id=owner-repo-pr-123 --title="Agent regresses checkout state" --repo=owner/repo --pr-url=https://github.com/owner/repo/pull/123
npm run bench:curation -- benchmarks/agent-prs/cases
# after adding at least one benchmarks/agent-prs/cases/*.json file
npm run bench:catch-rate -- benchmarks/agent-prs/cases --reviewer=codevetter
npm run test:benchmark
```

Fixture contract:

- Fixtures can be one combined JSON file with `cases[]`, one per-case JSON file, or a directory of per-case JSON files.
- `ground_truth`: hand-labeled issues that should be caught.
- `reviews.<reviewer>`: findings emitted by CodeVetter or a comparator.
- `matched_ground_truth`: issue ids the finding correctly catches.
- Findings with empty `matched_ground_truth` count as false positives.
- Strict mode is on by default and requires each case to include `source.repo`; pass `--no-strict` only for scratch fixtures.
- Publishable-fixture mode: `--require-rationales` requires non-placeholder `ground_truth[].evidence` and `reviews.<reviewer>[].match_rationale` on every matched finding.

Curation workflow:

1. Create one starter file per public PR with `npm run bench:new-case -- --id=<case-id> --title=<title> --repo=<owner/repo> --pr-url=<url>`.
2. Preserve source evidence in `source`: public PR URL, diff range, agent/tool name, raw diff artifact, CodeVetter output, CodeRabbit free-tier output, and Claude Code `/review` output.
3. Replace every `TODO` with hand-labeled evidence before publishable scoring.
4. Run `npm run bench:curation -- benchmarks/agent-prs/cases` until every case is ready.
5. Run `npm run bench:catch-rate -- benchmarks/agent-prs/cases --reviewer=codevetter --require-rationales`.

Curation readiness checks:

- public source metadata: repo, PR URL, diff range, and agent/tool name,
- preserved raw diff artifact,
- preserved CodeVetter, CodeRabbit free-tier, and Claude Code `/review` output artifacts,
- review arrays for `codevetter`, `coderabbit_free`, and `claude_code_review` (empty arrays are allowed when a tool produced no findings),
- hand-labeled ground truth with non-placeholder evidence,
- matched review findings with non-placeholder rationales.

Metrics:

- Overall catch rate: matched ground-truth issues divided by expected issues.
- By-severity catch rate: the same calculation grouped by `severity`.
- False positives: reviewer findings that do not match any ground-truth issue.
- Redundant matches: repeated matches to a ground-truth issue already caught by another finding in the same case.
- Precision: caught ground-truth issues divided by caught issues plus false positives plus redundant matches.
- F1: harmonic mean of catch rate and precision.
- Baseline comparison: `--baseline=<reviewer>` reports caught/rate/false-positive deltas.
- Gate mode: `--min-rate=<0..1>` exits non-zero when a selected reviewer falls below the threshold.
- False-positive gate mode: `--max-false-positives=<n>` exits non-zero when a selected reviewer reports more unmatched findings than allowed.
- Redundant-match gate mode: `--max-redundant-matches=<n>` exits non-zero when a selected reviewer repeats already-caught ground-truth matches too often.
- Severity gate mode: `--min-severity-rate=<severity>:<0..1>` exits non-zero when a selected reviewer falls below the threshold for that severity. Repeat the flag for multiple severities.
- Report output: `--format=text|json|markdown` controls stdout format. `--json` is equivalent to `--format=json`.
- Durable artifact output: `--out=<path>` writes JSON or Markdown reports to disk. Text mode writes a Markdown report to `--out` while keeping the human text summary on stdout.
- Rationale gate: `--require-rationales` exits non-zero if hand-labeled bugs or matched findings lack evidence/rationale text.

The included sample is not a publishable benchmark. It only proves the harness shape. Before making external claims, curate 20-30 real public agent PRs with known bugs, preserve raw diffs and review outputs, and record why each finding matches or misses ground truth.

`npm run test:benchmark` exercises the CLI gates against the sample fixture and generated temporary fixtures: rationale enforcement, TODO-placeholder rejection, directory/per-case fixture loading, curation readiness reporting, false-positive limits, duplicate match accounting, JSON metrics, and Markdown artifact output.

## Results — public 27-case set (2026-07-11)

| Reviewer | Catch rate | Precision | F1 |
|---|---|---|---|
| codevetter (pipeline + similarity dedup) | **1.000** (29/29) | **0.433** | **0.604** |
| codevetter (pipeline, pre-dedup-fix) | **1.000** (29/29) | 0.299 | 0.460 |
| raw-claude (single prompt baseline) | 0.931 (27/29) | 0.397 | 0.557 |

The dedup fix (same-file near-line token-similarity clustering in
`dedupe_findings`, calibrated on real duplicate/non-duplicate pairs from the
first run) cut raw findings 95→65 and flipped the head-to-head: CodeVetter now
beats the single-prompt baseline on catch rate, precision, AND F1. Remaining
noise is mostly process/verification findings (intentional for agent-PR
review, penalized by defect-only ground truth) plus a few boundary duplicate
pairs below the similarity threshold — deliberately not tuned further to
avoid overfitting 27 cases.

Protocol: every case ran through the REAL production pipeline (risk tiers,
specialists, coordinator, dedup) via the ignored Rust harness
`diag_benchmark_generate_codevetter_reviews`; raw outputs live in
`benchmark/reviews-raw/`. Ground-truth mapping was proposed mechanically
(`scripts/map-benchmark-reviews.mjs`) and then hand-judged per finding under
one rule: a finding matches only if its CORE CLAIM identifies the defect;
process findings (no-tests, shipped-with-comment) never match.

Reading: CodeVetter catches everything raw Claude catches plus the two it
missed — but emits 95 findings for 29 defects. The precision loss decomposes
as 41 redundant restatements of already-caught defects (the coordinator dedup
does not collapse same-defect findings on small diffs — actionable) and 25
unmatched findings, ~20 of which are process/verification findings that are
intentional for agent-PR review but score as false positives against
defect-only ground truth. With redundants collapsed, precision would be
0.537 (F1 ≈ 0.70); excluding process findings too, 0.85.

These are synthetic single-file cases — the planned real agent-PR case
curation still stands before external head-to-head claims.