CodeVetter
Download
← CodeVetter

Verification field guide

Coding-agent verification

Verify code inside the agent loop with a runnable pagination example: define checks, reproduce failure and repair, preserve regressions, and decide when to stop.

Coding-agent verification asks whether an agent completed the requested software task and which executable checks support that answer. Put the check inside the agent loop: define the behavior, preserve the patch, run the check, then use the result to decide whether to repair, stop or gather missing evidence.

The five-part loop

  1. Task: write the requested outcome and the behavior that must stay intact.
  2. Change: record the base revision and the exact agent-produced patch.
  3. Execution: run repository-owned checks against the changed boundary.
  4. Evidence: retain the command, environment, output and source identity.
  5. Verdict: state what passed, failed or remains unverified.

Review can help find a suspicious change. Execution establishes whether the checked behavior fails. A green command is useful only when it exercises the requirement at the right boundary.

Worked example: forward a pagination cursor

An agent is asked to update a list wrapper so the caller's cursor reaches an integration client, while preserving the requested limit and returned response. The published synthetic task supplies the starting source, acceptance contract, check driver and known-good change.

The starting wrapper calls client.list({ limit }), dropping its cursor argument. The known-good change calls client.list({ limit, cursor }). Before changing code, turn the request into these three checks:

  • Requested behavior: call listNext(client, 25, 'cursor-2') and check that the integration receives cursor: 'cursor-2'. The check is named pagination-cursor-forwarded.
  • Preserved input: the same call must pass limit: 25. The regression check is pagination-limit-preserved.
  • Preserved output: the wrapper must return the exact response object from the integration. The regression check is list-response-preserved.

The authoritative boundary here is the wrapper's call to client.list and its returned value. The check captures that call with a small test client. It does not contact a live integration or prove that an external API accepts the cursor. In your repository, add the real integration check when that is part of the requested task.

Run the failure-and-fix check

Use Node.js 24 or newer and the repository's pinned pnpm 10.33.2. In a fresh clone, run:

git clone https://github.com/Codevetter/codevetter.git codevetter-cursor-check
cd codevetter-cursor-check
git switch --detach 83b09906d44eab9ddd15bbc8264d10781e3d2474
corepack pnpm install --frozen-lockfile
corepack pnpm corpus:qualify --task forward-pagination-cursor --json

The corpus guide explains setup and the full receipt contract. Qualification runs the committed baseline twice and the known-good replacement twice in fresh temporary workspaces. It does not launch a coding agent or call a model provider. It executes trusted corpus check code without an operating-system sandbox.

For this pinned task, the expected receipt has qualified: true, baseline.status: "intended_failure", known_good.status: "pass" and cleanup.status: "complete". The command exits 0 when all four attempts and cleanup satisfy the contract. The baseline is supposed to fail the cursor check while passing both regression checks; an arbitrary failure does not qualify it. The known-good version must pass all three checks on both attempts.

Keep the complete JSON receipt. Its task ID and SHA-256 identities bind the manifest, fixture, acceptance contract and known-good change. The attempt outcomes and result hashes distinguish repeated expected behavior from a single green run. Record your local Node version and command alongside it.

Decide whether to repair, recheck or stop

  • The changed behavior fails: give the agent the failing requirement and bounded output. After a repair, rerun the same check and the relevant regressions against the new patch. Preserve the earlier failure.
  • The requested behavior passes but a regression fails: continue the repair. Forwarding the cursor while changing the limit or response is incomplete work.
  • All declared checks pass: stop the repair loop for those requirements. Review any remaining integration, permission or recovery requirements before release; the fixture's pass does not cover them.
  • Setup fails, a check times out, attempts disagree or evidence is missing: keep the outcome inconclusive. Fix or obtain the missing environment or evidence, then rerun. Do not ask the agent to alter working product code just to hide a test-environment failure.

This synthetic replay qualifies a task and its checks. It does not measure an agent's task-completion rate, time saved or return on investment. Verification can guide the next action for an exercised requirement; showing that it improves agent-loop effectiveness requires a separate comparison on representative tasks with the same scoring rules.

Use the loop on your next change

Run the published verification recipe, then replace the sample requirement with your own task, source revision and authoritative checks. Keep an evidence bundle with the patch.

For an observed public-repository failure, inspect the React hydration case. It links the reviewed source, before-and-after visibility checks and a focused replay. For local product availability, install CodeVetter.