How to verify AI-generated code

To verify AI-generated code, start from the requested behavior and finish with reproducible evidence. Do not begin by asking a second model whether the patch looks good. That can help triage risk, but it cannot establish that the software works.

1. Freeze the task and revision

Write down the requested outcome, the repository revision before the run, and the exact agent change. If the task is ambiguous, record the ambiguity instead of allowing the verifier to silently choose a friendlier interpretation.

2. Identify the behavioral boundary

Name what users or callers can observe: a browser state transition, an API response, an authorization denial, a persisted record, or a regression that must remain fixed. Prefer checks against this boundary over checks that merely mirror the implementation.

3. Run the smallest authoritative checks

Use repository-owned tests and commands first. Add a focused browser or API check when the existing suite does not exercise the changed behavior. Capture the command, environment, exit state, and bounded output. A passing command without identity or context is weak evidence.

4. Separate failures

Distinguish an agent regression from an existing repository failure, an environment problem, a timeout, and an unverified requirement. This prevents infrastructure noise from becoming a false product verdict.

5. Package the evidence

Bind the task, change, checks, outputs, artifacts, and verdict into a machine-readable bundle. A reviewer should be able to answer: what ran, against which revision, what changed, what failed, and what remains unknown?

6. Re-check after a fix

A fix is not complete because the diff changed. Re-run the failing check and the smallest relevant regression set. Preserve both the failure and the later pass so the evidence shows closure rather than only the final state.

CodeVetter follows this task-to-evidence shape and is designed to fail closed when required evidence is absent. Its current public benchmark demonstrates a narrower recognition task and publishes its own limitations; it should not be read as a production-wide guarantee.

Try it: catch a real class of AI-code bug

Suppose an agent is asked to preserve an explicit false setting while keeping the default and label behavior. A truthiness fallback can look reasonable in a review and still violate that requirement. This small, synthetic example is adapted from CodeVetter's published explicit-false task; it illustrates the check, rather than a measured agent success rate.

With Node.js installed, save this as verify.mjs and run node verify.mjs:

import assert from 'node:assert/strict';

function transform(input) {
  return {
    enabled: input.enabled || true,
    label: input.label ?? 'Default',
  };
}

assert.equal(transform({ enabled: false }).enabled, false);
assert.equal(transform({ label: 'Keep this' }).label, 'Keep this');
assert.equal(transform({}).enabled, true);

The first assertion fails: false || true becomes true. Change only input.enabled || true to input.enabled ?? true and run the same file again. All three assertions should pass. The default and label checks protect behavior that the fix must preserve.

For your own patch, replace these assertions with checks at the changed API, browser or storage boundary. A unit example cannot establish that a deployed route or permission-dependent workflow works. Record the failing revision, the fixed revision and the same check's output for both.

Reproduce the CodeVetter corpus check

To inspect the product's evidence mechanism, clone the repository and follow its corpus setup and qualification guide. From the repository root, run:

pnpm corpus:qualify --task preserve-explicit-false --json

Qualification executes the committed baseline and known-good change in fresh temporary workspaces and checks that the intended failure and later pass repeat. It does not call a model provider or demonstrate that an agent found the fix. Inspect the task identity, required check, preserved-label regression and cleanup state in the receipt before treating it as evidence.

Decide what is ready to ship

A passing example covers only its declared requirement. Before shipping your change, also exercise relevant integration paths, permission denials and failure recovery. If a required environment or check is unavailable, record it as unverified and state what remains to run. Keep the evidence bundle alongside the patch so another developer can assess the result.

CTA: Use the checklist with your next agent patch, then inspect how CodeVetter structures verification evidence.

A public-repository case

Inspect a React hydration visibility regression: one CodeVetter review finding, two failing visibility cases before repair, and four passing checks afterward. The article links pinned public source, a redacted evidence summary and a focused replay. This is one observed case, not a catch-rate benchmark.