# CodeVetter - full content > Execution-backed verification for coding-agent changes, with local checks, portable evidence, explicit limitations, and a measurable verdict. Generated: 2026-10-27 --- # CodeVetter > Verify whether a coding agent completed its task. CodeVetter binds the exact change to executable checks, evidence, limitations, and a measurable verdict. Canonical: https://codevetter.com/ Execution-backed verification and evaluation for coding agents. ## What it is - Reproducible runtime evidence for agent changes - Deterministic and calibrated graders - Machine-readable verification bundles through CLI and MCP - Local SQLite storage and a native macOS viewer - Open source (ISC) ## Who it's for Engineers evaluating or shipping coding-agent changes who need executable evidence before accepting a result — without uploading the repository to a CodeVetter server. ## Current availability - Apple-silicon macOS build through GitHub Releases - Packaged local CLI and stdio MCP sidecars inside the desktop distribution - No separately published npm package, hosted verification API, Homebrew cask, or other platform installer - Optional provider-backed review sends selected context directly to the provider configured by the user - No CodeVetter checkout, paid API, hosted verification service, or team subscription ## Agent entrypoints - https://codevetter.com/llms.txt - https://codevetter.com/api/ai - https://codevetter.com/index.md - https://codevetter.com/docs.md - https://codevetter.com/benchmark.md - https://codevetter.com/inspiration.md - https://codevetter.com/pricing.md - https://codevetter.com/.well-known/ai-catalog.json --- # About CodeVetter > CodeVetter is an execution-backed verification system for coding agents. Local-first, open-source, and built by Sarthak Agrawal. Canonical: https://codevetter.com/about CodeVetter is an execution-backed verification and evaluation system for coding agents. It determines whether an agent completed a software task correctly using reproducible runtime evidence — not another LLM opinion. The review pipeline runs locally in a native macOS app; your repository never hits a CodeVetter server. ## What it does The core loop is: task → agent change → executable verification → evidence → measurable verdict. CodeVetter runs tests, type checks, builds, and profiling against the agent's diff, then produces a portable evidence bundle that proves whether the task is done. ## Open source CodeVetter is open-source under the ISC license. The source lives at github.com/Codevetter/codevetter. The current Apple-silicon macOS build and updater archive are published via GitHub Releases. ## Built by Sarthak Agrawal — AI infrastructure and product engineer. More at sarthakagrawal.dev. ## Public benchmark CodeVetter maintains a public benchmark of reproducible verification cases, a scorer, and published results with documented limitations. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Download CodeVetter > Download the current Apple-silicon macOS release of CodeVetter from GitHub Releases, or inspect and build the open-source repository. Canonical: https://codevetter.com/download CodeVetter is distributed as a native SwiftUI/AppKit macOS application through GitHub Releases. It runs locally, requires no CodeVetter account, and stores product state in local SQLite. A signed Sparkle update feed is available. In-app installation and relaunch are still being verified. To update manually, quit CodeVetter, download the latest DMG, and replace CodeVetter.app in Applications; keep your existing application data. ## Supported release artifacts - macOS 14 or newer on Apple Silicon: `CodeVetter-1.16.6-arm64.dmg` (drag-to-Applications installer) and `CodeVetter-1.16.6-arm64.zip` (the Sparkle update archive), published on v1.16.6. Developer ID signed and notarized. The current public release does not contain Intel macOS, Windows, Linux, or Homebrew installers. The repository remains available for source inspection and local builds. [Open the latest release](https://github.com/Codevetter/codevetter/releases/tag/v1.16.6). ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeVetter FAQ > Clear answers about CodeVetter's execution-backed agent verification, local evidence, provider boundary, benchmark, and current macOS availability. Canonical: https://codevetter.com/faq ## What is CodeVetter? CodeVetter is an execution-backed verification and evaluation system for coding agents. Its core loop is task, agent change, executable verification, evidence, and a measurable verdict. CLI, MCP, and machine-readable verification bundles are primary product surfaces; the desktop app is a local viewer. ## Does CodeVetter upload repositories to its own server? No. CodeVetter has no hosted review backend. Repository context, review history, and product state remain on the local machine. When a user explicitly invokes an external model provider, that provider receives the request under its own policy. ## Which coding agents and model providers can it work with? The desktop work surface can supervise installed Codex and Claude CLIs. Review supports user-configured Anthropic, OpenAI, and OpenRouter providers. Provider availability and account policy remain external to CodeVetter. ## What evidence does a review retain? Findings can retain source anchors, command or test evidence, verification state, and portable proof exports. Missing, waived, stale, and unverified stages stay explicit rather than being presented as successful. ## Is there a public benchmark? Yes. The public synthetic benchmark contains 27 hand-labeled cases and 29 expected findings. Its cases, labels, reviewer outputs, and scorer are published so the result can be reproduced. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Comparing local desktop verification with cloud PR review > Compare CodeVetter, CodeRabbit, and Greptile on where your code is processed and what each publishes as reproducible evidence, sourced from each vendor's… Canonical: https://codevetter.com/compare ## The short answer CodeVetter is a local-first desktop application that verifies AI-written code by executing it on your machine. CodeRabbit and Greptile are cloud pull-request bots that analyse your diff after you push it. The difference that decides most of these evaluations is where your code is processed. This page compares the three on two axes we can verify from public evidence: **where the code goes, and whether the tool publishes a reproducible benchmark you can run yourself.** Every statement about another vendor is quoted from that vendor's own public documentation, with the date it was checked. ## Where your code is processed This is the one hard architectural difference rather than a feature gap. All three send code to a large language model provider. What differs is what sits between your repository and that provider. **CodeVetter.** Reviews execute on your machine. From [`codevetter.com/privacy`](https://codevetter.com/privacy) (last updated 2026-09-01): > "CodeVetter is a native macOS app. Reviews run on your machine. The repo you point it at, the diff being reviewed, your notes, and the review history all live in a local SQLite database in the app data directory. None of that goes to a CodeVetter-owned server — there isn't one." The same page is equally explicit about what does leave: *"When you run a review, CodeVetter sends your code + the review prompt to whichever provider (Anthropic, OpenAI, OpenRouter, your own gateway) you've picked. Their privacy policy applies."* There is no CodeVetter server in the path. There is still a model provider in the path, and it is one you choose and contract with directly. **CodeRabbit.** The default product is cloud. Self-hosting exists and is documented, with two qualifications worth reading before assuming it applies to you. From `docs.coderabbit.ai/self-hosted/overview`, checked 2026-09-05: > "Self-hosted CodeRabbit runs the CodeRabbit review agent inside your own infrastructure instead of CodeRabbit's cloud." > "The self-hosted option is available for CodeRabbit Enterprise customers with 500 or more user seats." > "Pull request orchestration and review results stay within your environment; code and prompt data leave it only to reach the LLM provider." That last sentence is the honest version of the whole axis, and it applies to CodeVetter too: self-hosting relocates the orchestration, not the inference. **Greptile.** Cloud by default, with a self-hosted deployment that the docs describe in stronger terms than CodeRabbit's. From `greptile.com/docs/security/selfhost`, checked 2026-09-05: > "The entirety of Greptile's service can be self-hosted in an air-gapped environment." And from `greptile.com/docs/deployment-options`, checked 2026-09-05: > "Use Cloud if you want Greptile running in minutes with zero infrastructure management. Use Self-Hosted if you need data sovereignty, air-gapped environments, or custom LLM providers." The same page documents Docker Compose for teams up to 100 developers and Kubernetes above that — so unlike CodeRabbit's 500-seat Enterprise floor, the self-hosted path is documented for small teams. ### The axis, side by side | | CodeVetter | CodeRabbit | Greptile | | --- | --- | --- | --- | | **Default deployment** | Local desktop app | Vendor cloud | Vendor cloud | | **Vendor server in the path** | None — *"there isn't one"* (`/privacy`) | Yes, unless self-hosted | Yes, unless self-hosted | | **Self-hosted option** | N/A — already local | Enterprise, documented at 500+ seats | Documented from small-team Docker Compose upward, incl. air-gapped | | **Model provider** | Yours: Anthropic, OpenAI, OpenRouter, or your own gateway | Yours, under self-host ("bring your own model") | Yours: OpenAI, Anthropic, Bedrock, Azure OpenAI, Vertex AI | | **Code reaches an LLM provider** | Yes | Yes | Yes | *All competitor rows quoted or summarised from the vendor documentation cited above, checked 2026-09-05.* ## What each tool publishes as evidence The second axis is not "who publishes a benchmark." **All three do.** The question a reader actually needs answered is narrower: *can you re-run it on your own code?* - **CodeVetter** publishes a benchmark with its cases, scorer and outputs at [`/benchmark`](https://codevetter.com/benchmark). It is not restated here. - **Greptile** publishes a five-tool, 50-PR, five-repository comparison at `greptile.com/benchmarks` with a written protocol, and says of it: *"All PRs come from public, verifiable repositories, so you can inspect the sources and reproduce the runs on your own."* The same page dates the work: *"this evaluation was conducted in July 2025, and these tools evolve quickly, so performance may change over time."* Checked 2026-09-05. - **CodeRabbit** publishes benchmark and evaluation write-ups on its blog and points at a third-party benchmark, Martian's Code Review Bench, describing Martian's *"methodology and code"* as *"fully open source."* Checked 2026-09-05. **The third-party harness is real, and it is worth being precise about what it lets you do.** Martian's Code Review Bench is published in full at [`github.com/withmartian/code-review-benchmark`](https://github.com/withmartian/code-review-benchmark) under an MIT licence. Its README is unambiguous: *"We open-source everything: the PRs, the golden comments, the LLM judge prompts, the evaluation pipeline, and a continuously-updated online benchmark that avoids training data leakage."* Anyone can run it — *"Running a tool that isn't on the leaderboard is open to anyone."* But read what running it means: *"Fork the 50 benchmark PRs into a GitHub org where your tool is installed."* **The harness runs on Martian's public pull-request corpus, not on your repository.** That is the distinction this section exists to draw. A public benchmark tells you how a tool scored on fifty pull requests that someone else chose. It does not tell you how it behaves on yours, and no vendor on this page — CodeVetter included — ships something that does. **A caution that belongs on this page more than any performance figure.** Both CodeRabbit and Greptile publish a post claiming the #1 position on the *same* benchmark, on the *same* metric — F1, which is the metric Martian's leaderboard is ordered by. CodeRabbit's post reports a January–February 2026 sampling window. Greptile's is dated 30 July 2026, and reproduces the leaderboard table it is citing. Both posts are accurate. Code Review Bench samples pull requests continuously and Martian says so: *"Martian updates the leaderboard continuously as it analyzes new pull requests."* Neither vendor is misrepresenting anything, and this page is not suggesting otherwise. The lesson is structural: **a "#1" on a live leaderboard is a timestamp, not a property of the product.** A comparison page that quoted either ranking as current would be presenting a six-month-old snapshot as a fact — which is precisely why there are no rankings or scores below. This page therefore publishes **no** catch-rate, precision, recall, latency or cost figure for any of the three. That is a deliberate omission, not a gap. ## When a cloud PR bot is the better choice Plainly: often. - **Review has to cover everyone's pull requests, not yours.** A cloud bot reviews every PR in a repository with no per-developer install. CodeVetter is a desktop app someone has to run. - **You want review where review already happens.** CodeRabbit attaches to the pull request, IDE and CLI, with incremental follow-ups on new commits. - **You need whole-repository context.** Greptile indexes the codebase to review a diff against the rest of the system, which a local pre-push check on one change set does not attempt. - **Nobody has a Mac.** CodeVetter publishes an Apple silicon build only ([`/download`](https://codevetter.com/download)); the cloud tools are platform-independent. If your constraint is coverage across a team, a cloud PR bot is the better tool and this page is not going to argue otherwise. ## Head-to-head detail The full comparisons live on their own pages and are not duplicated here. - **[CodeVetter vs. CodeRabbit](https://codevetter.com/codevetter-vs-coderabbit)** — verification evidence against automated PR review, and where the two overlap in a workflow. - **[CodeVetter vs. Greptile](https://codevetter.com/codevetter-vs-greptile)** — local verification against repo-aware review, including how Greptile's TREX layer executes code during review, so Greptile is not a static-only reviewer. ## How this comparison was built **Sources.** CodeVetter claims come from [`/privacy`](https://codevetter.com/privacy) and [`/benchmark`](https://codevetter.com/benchmark). CodeRabbit claims come from `docs.coderabbit.ai` (`/self-hosted/overview`) and `coderabbit.ai/blog` (`/coderabbit-tops-martian-code-review-benchmark`). Greptile claims come from `greptile.com/docs` (`/security/selfhost`, `/deployment-options`), `greptile.com/benchmarks`, and `greptile.com/content-library/greptile-martian-code-review-benchmark`. The third-party benchmark is `github.com/withmartian/code-review-benchmark`. **All pages checked 2026-09-05.** **What we did not test.** We did not run CodeRabbit or Greptile. We did not run any of the three against a shared case set. We did not verify vendor claims about their own internals beyond quoting what they publish, and we did not evaluate any self-hosted deployment. No figure on this page is a measurement we made. **What would make a real head-to-head.** The same immutable cases, disclosed configuration, repeated runs, and one scorer applied to all three. Until that exists, treat every vendor comparison table — including the one above — as a map of stated capabilities, not measured performance. ### How to decide in your own repository 1. Pick five representative agent tasks and freeze their acceptance criteria. 2. Run each product in its normal supported configuration. 3. Preserve every finding, executed check, skipped check, and provider setting. 4. Score task completion and regressions with repository-owned checks. 5. Report latency, cost, false positives, and missing evidence separately. ### Where to go next - [Review vs. verification](https://codevetter.com/ai-code-review-vs-verification) — the concepts, no vendor - [CodeVetter vs. CodeRabbit](https://codevetter.com/codevetter-vs-coderabbit) - [CodeVetter vs. Greptile](https://codevetter.com/codevetter-vs-greptile) - [CodeVetter vs. the verification CLIs](https://codevetter.com/codevetter-vs-verification-clis) - [The public benchmark](https://codevetter.com/benchmark) — cases, scorer, outputs, limitations - [How verification works](https://codevetter.com/coding-agent-verification) - [Privacy](https://codevetter.com/privacy) — what runs locally and what reaches a provider - [Download CodeVetter](https://codevetter.com/download) ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # AI code review benchmark for agent-written bugs > Public AI code review benchmark for agent-written bugs: 27 cases, 29 labeled findings, catch rate, precision, downloads, and stated limitations. Reproducible. Canonical: https://codevetter.com/benchmark CodeVetter's public v1 benchmark is a reproducible recognition benchmark for agent-written bugs. It publishes 27 synthetic cases, 29 labeled findings, reviewer outputs, scoring rules, downloads, and explicit limitations. It answers whether the tested review pipeline recognizes these known issues under this fixture set. It does not prove that CodeVetter verifies arbitrary production pull requests. ## Read the result in context The page should keep the current catch rate, precision, F1, false-positive accounting, case explorer, and raw baseline visible. The first screen should distinguish three layers: 1. **Published now:** the 27-case recognition benchmark and its exact results. 2. **Implemented qualification infrastructure:** task packages, executable checks, receipts, and evaluation contracts described by the repository. 3. **Not yet proven publicly:** repeated provider runs on a broad realistic corpus with reliability, cost, and latency comparisons. ## Choose the benchmark for the question Recognition fixtures help isolate issue-type coverage and scoring behavior. Repository-task benchmarks such as SWE-bench evaluate whether a generated patch resolves a real issue inside a reproducible environment. A product-specific agent benchmark should additionally preserve the requested task, exact patch, environment, checks, evidence, and uncertainty. No single score answers all three questions. ## Inspect, do not just trust Keep direct links to the downloadable dataset, per-case X-ray pages, outputs, and methodology. Explain how a reader can reproduce the scorer. Surface the limitations before the CTA: synthetic cases, narrow languages and categories, no timing or cost dimension, and precision as a material constraint. ## Where CodeVetter is going Link the benchmark to the new coding-agent verification pillar and workflow. The next credible proof is not a larger marketing claim. It is qualified, repeated agent-task evidence with immutable receipt linkage and clear failure taxonomy. **CTA:** Explore all cases, download the dataset, reproduce the scorer, then read the verification methodology. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeVetter docs > The local-first knowledge system for the CodeVetter repository. Canonical: https://codevetter.com/docs/ This is the canonical knowledge system for the CodeVetter repository. The committed Markdown here is the source of truth; [Blume](https://useblume.dev/) is only the presentation and search layer. - **Short current view**: [`STATUS.md`](https://github.com/Codevetter/codevetter/blob/main/STATUS.md) - **Deep timeline + feature log**: [`PROJECT_STATUS.md`](https://github.com/Codevetter/codevetter/blob/main/PROJECT_STATUS.md) - **Agent bootloader**: [`AGENTS.md`](https://github.com/Codevetter/codevetter/blob/main/AGENTS.md) - **Product readme**: [`README.md`](https://github.com/Codevetter/codevetter/blob/main/README.md) ## Product - [overview.md](https://codevetter.com/docs/product/overview/) — what CodeVetter is, durable scope, capability matrix, strategy. - [capabilities.md](https://codevetter.com/docs/product/capabilities/) — UI, CLI, MCP, underlying-tool, and future-capability glossary. - [mcp-setup.md](https://codevetter.com/docs/product/mcp-setup/) — verified CLI installation and repository-scoped MCP onboarding. - [surfaces.md](https://codevetter.com/docs/product/surfaces/) — nav tabs, URL-only routes, removed surfaces. - [synthetic-user-qa.md](https://codevetter.com/docs/product/synthetic-user-qa/) — runtime proof layer for agent-written code. - [trex-change-preview.md](https://codevetter.com/docs/product/trex-change-preview/) — exact PR or commit-range verification against an existing preview. ## Architecture - **Start here:** [how-it-works.md](https://codevetter.com/docs/architecture/how-it-works/) — end-to-end pedagogical overview of the whole system (components, review flow, key decisions). - [overview.md](https://codevetter.com/docs/architecture/overview/) — layers, critical invariants, what was removed. - [ipc-and-commands.md](https://codevetter.com/docs/architecture/ipc-and-commands/) — the IPC bridge and command map. - [data-model.md](https://codevetter.com/docs/architecture/data-model/) — SQLite tables and persistence boundaries. - [review-pipeline.md](https://codevetter.com/docs/architecture/review-pipeline/) — review → fix → re-review → proof. - [graph-and-history.md](https://codevetter.com/docs/architecture/graph-and-history/) — canonical structural graph + release history workbench. - [repo-unpacked.md](https://codevetter.com/docs/architecture/repo-unpacked/) — evidence-backed repo briefs. - [mcp-sidecar.md](https://codevetter.com/docs/architecture/mcp-sidecar/) — opt-in local MCP server. - [trex-mcp-projection.md](https://codevetter.com/docs/architecture/trex-mcp-projection/) — future agent-triggered verification boundary without weakening the read-only MCP. - [history-evidence-import.md](https://codevetter.com/docs/architecture/history-evidence-import/) — importing provider-side outcomes. - [verification-workbench.md](https://codevetter.com/docs/architecture/verification-workbench/) — additive evidence identities, retention, managed-run, intent, and performance records. - [native-migration-matrix.md](https://codevetter.com/docs/architecture/native-migration-matrix/) — no-regression map and release gates for the sole native macOS app. - [native-rust-boundary.md](https://codevetter.com/docs/architecture/native-rust-boundary/) — measured hybrid ownership split between in-process reads and supervised execution. ### Decisions - [mcp-sdk.md](https://codevetter.com/docs/architecture/decisions/mcp-sdk/) — chose `rmcp` 2.2.0 for the MCP sidecar. - [oss-integration.md](https://codevetter.com/docs/architecture/decisions/oss-integration/) — OSS integration evaluation. - [structural-graph-contract.md](https://codevetter.com/docs/architecture/decisions/structural-graph-contract/) — first-party code-graph capability and release contract. ## Development - [setup.md](https://codevetter.com/docs/development/setup/) — prerequisites, install, run. - [testing.md](https://codevetter.com/docs/development/testing/) — native, Rust, CLI/MCP, package, and code-health gates. - [performance.md](https://codevetter.com/docs/development/performance/) — native, Rust, MCP, and exact-package performance gates. - [native-macos.md](https://codevetter.com/docs/development/native-macos/) — native Evidence Workbench toolchain, ownership, and qualification boundaries. - [benchmark.md](https://codevetter.com/docs/development/benchmark/) — catch-rate benchmark. - [agent-task-corpus.md](https://codevetter.com/docs/development/agent-task-corpus/) — immutable coding-agent task packages and fail-closed readiness. - [verification-receipts.md](https://codevetter.com/docs/development/verification-receipts/) — ingest project-runner evidence, compare regressions, and explain test blast radius. - [configuration.md](https://codevetter.com/docs/development/configuration/) — native identity, settings, data, and updater configuration. - [docs.md](https://codevetter.com/docs/development/docs/) — how to write, validate, and render docs. ## Operations - [automation-contract.md](https://codevetter.com/docs/operations/automation-contract/) — privacy-safe product, release, reliability, and Foundry evidence contracts. - [release-pipeline.md](https://codevetter.com/docs/operations/release-pipeline/) — desktop release chain. - [signing-and-updates.md](https://codevetter.com/docs/operations/signing-and-updates/) — Developer ID signing, notarization, and Sparkle auto-update inputs. - [landing-deploy.md](https://codevetter.com/docs/operations/landing-deploy/) — Cloudflare Pages deploy. - [ci.md](https://codevetter.com/docs/operations/ci/) — CI workflow and order. ### Jobs - [weekly-quality.md](https://codevetter.com/docs/operations/jobs/weekly-quality/) — Monday cron canary. ### Runbooks - [cut-a-release.md](https://codevetter.com/docs/operations/runbooks/cut-a-release/) — ship a new desktop version. - [deploy-landing.md](https://codevetter.com/docs/operations/runbooks/deploy-landing/) — deploy the Astro site. ## Knowledge - [failed-approaches.md](https://codevetter.com/docs/knowledge/failed-approaches/) — things that broke and the constraints they left. - [competitive-landscape.md](https://codevetter.com/docs/knowledge/competitive-landscape/) — AI code review competitive landscape. - [performance-lab-publication.md](https://codevetter.com/docs/knowledge/performance-lab-publication/) — published optimization experiments, external PRs, and evidence boundaries. - [tooling-decisions.md](https://codevetter.com/docs/knowledge/tooling-decisions/) — what CodeVetter embeds instead of building, what it refuses, and why. Hub for the five evaluations below. - [tooling-secrets-and-supply-chain.md](https://codevetter.com/docs/knowledge/tooling-secrets-and-supply-chain/) — secret scanners and dependency/vulnerability scanners. - [tooling-coverage.md](https://codevetter.com/docs/knowledge/tooling-coverage/) — coverage collection across Rust/TS/Python, the LCOV transport decision, diff-coverage traps. - [tooling-agent-benchmarks.md](https://codevetter.com/docs/knowledge/tooling-agent-benchmarks/) — public agent benchmark corpora, graded on determinism and licensing. - [tooling-documentation.md](https://codevetter.com/docs/knowledge/tooling-documentation/) — the DeepWiki reject and what would fit instead. - [tooling-sandboxing.md](https://codevetter.com/docs/knowledge/tooling-sandboxing/) — isolation for running untrusted agent code, determinism controls, SARIF analyzers. - [codebase-context-tools-landscape.md](https://codevetter.com/docs/knowledge/codebase-context-tools-landscape/) — April 2026 survey of codebase indexing and context tooling. ### Learnings - [README.md](https://codevetter.com/docs/knowledge/learnings/README/) — learning roadmap and coverage map. - [new-things.md](https://codevetter.com/docs/knowledge/learnings/new-things/) — platform + stack concepts. - [telemetry-and-indexing.md](https://codevetter.com/docs/knowledge/learnings/telemetry-and-indexing/) — the usage pipeline. - [verification-and-judgment.md](https://codevetter.com/docs/knowledge/learnings/verification-and-judgment/) — the verification stack. ## Archive Superseded docs kept for git history. **Do not treat as current.** These are excluded from the rendered Blume site (see `blume.config.ts`), so the links below point at the files in the repo on GitHub. - [DECISIONS.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/DECISIONS.md) — older decision log. - [LESSONS.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/LESSONS.md) — older lessons. - [PRD-*.md](https://github.com/Codevetter/codevetter/tree/main/docs/archive) — scoped PRDs for shipped slices. - [stale-competitive-landscape-2026-03.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/stale-competitive-landscape-2026-03.md) — full March competitor survey + April OSS GTM plan. - [stale-architecture-2026-04.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/stale-architecture-2026-04.md) — pre-desloppification architecture (removed `packages/`/`workers/`). - [stale-development-2026-04.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/stale-development-2026-04.md) - [stale-configuration-2026-04.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/stale-configuration-2026-04.md) - [stale-testing-2026-04.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/stale-testing-2026-04.md) - [stale-performance-harness-2026-09.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/stale-performance-harness-2026-09.md) — retired React/Tauri performance harness and historical measurements. - [stale-native-agent-island-2026-09.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/stale-native-agent-island-2026-09.md) — retired helper prototype; any future status island is a side quest. - [stale-repo-unpacked-2026-09.md](https://github.com/Codevetter/codevetter/blob/main/docs/archive/stale-repo-unpacked-2026-09.md) — retired React/Tauri implementation notes. - [planning-codebase/](https://github.com/Codevetter/codevetter/tree/main/docs/archive/planning-codebase) — pre-desloppification `.planning/codebase/` docs. ## Open questions Tracked in [`STATUS.md`](https://github.com/Codevetter/codevetter/blob/main/STATUS.md) under "Unresolved questions". Do not invent answers in docs; mark gaps explicitly. ## Agent workflows - [Agent workflow skills](https://codevetter.com/docs/development/agent-skills/) — review, testing, performance and value evaluation. --- # CodeVetter changelog > Verified CodeVetter product updates, written as concise user-visible outcomes. Canonical: https://codevetter.com/changelog This curated changelog records shipped user-visible outcomes. Planned work remains in [GitHub Issues](https://github.com/Codevetter/codevetter/issues). ## 2026-07-26 — Focused on executable verification - CodeVetter now centers on proving whether coding agents completed a task correctly with reproducible runtime evidence. - CLI, MCP, and machine-readable verification bundles are primary surfaces; the desktop app remains a local viewer. ## 2026-07-25 — Complete local verification loop - Reviews can retain local sessions, recover managed runs, record intent closure, and produce evidence-owned artifact previews. - A public agent-PR corpus and redacted performance receipts support reproducible regression and reliability comparisons. ## 2026-07-24 — Deterministic Review and Agent PR X-Ray - Broad reviews cover changed files through bounded, checkpointed verification units. - Completed reviews can export fail-closed JSON, Markdown, and self-contained offline HTML evidence. ## 2026-07-21 — Conversation-first Work - Codex and Claude sessions gained provider-aware controls, attention states, searchable history, and safe archiving. - A local Plan, Build, Review, Verify, Done board keeps sessions attached to repository context. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeVetter privacy > What CodeVetter stores, what it never sends to a server, and how API keys are handled. Canonical: https://codevetter.com/privacy Last updated: 2026-09-28. ## Local-first by design CodeVetter is a native macOS app. Reviews run on your machine. The repo you point it at, the diff being reviewed, your notes, and the review history all live in a local SQLite database in the app data directory. None of that goes to a CodeVetter-owned server — there isn't one. ## What hits third parties - **The LLM provider you configure.** When you run a review, CodeVetter sends your code + the review prompt to whichever provider (Anthropic, OpenAI, OpenRouter, your own gateway) you've picked. Their privacy policy applies. - **Auto-updater.** The app checks GitHub Releases for new versions. That request includes your platform string and the current version. Disable in settings if you prefer. ## API keys Provider API keys are yours and stay on your machine. CodeVetter reads them from your local application settings or from the environment it is launched with; they are never transmitted to CodeVetter, and they leave your machine only inside the Authorization header of a request you initiate to the provider you chose. ## Crash and usage telemetry None by default. The first launch does not phone home; there is no anonymous usage analytics endpoint baked into the app. ## Public website analytics The public codevetter.com marketing and benchmark pages use PostHog and Microsoft Clarity to understand page visits and interactions. Those pages have no repository upload or CodeVetter account surface. This does not add telemetry to the desktop application. ## Optional product updates If you enter your email in the optional product-updates form and give consent, the SaaS Maker subscription service stores your address and consent record to manage that subscription. The form sends your address to SaaS Maker; CodeVetter does not receive it through App Health or link it to desktop-app activity. The form is optional, and the desktop application does not include it. ## Deletion Uninstall the app and delete the data directory (`~/Library/Application Support/CodeVetter` on macOS) to remove all local state. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeVetter terms of use > Terms of use for CodeVetter desktop software and codevetter.com. Execution-backed verification for coding-agent changes, with local checks, portable… Canonical: https://codevetter.com/terms CodeVetter is open-source desktop software distributed under the ISC License. Source and release binaries are provided as-is. Code review and verification output is assistive, not authoritative. Users remain responsible for the software they ship and should not treat CodeVetter as a substitute for human review or security testing. When a user invokes Claude, Codex, Gemini, or another external CLI or provider, that provider's terms apply to the external request. CodeVetter does not store repositories on a central server. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Coding-agent verification > Learn how coding-agent verification connects a task, agent change, executable checks, evidence, and a measurable verdict — and how to verify code inside the… Canonical: https://codevetter.com/coding-agent-verification Coding-agent verification answers a stricter question than code review: did an agent complete the requested software task correctly, and what executable evidence supports that verdict? A useful verification loop begins with the task, preserves the agent's change, runs checks against the changed behavior, collects the resulting evidence, and emits a verdict that another person or system can inspect. ## Review is an input, not the verdict Static review can identify suspicious code, but a plausible finding is not the same as a reproduced failure. Likewise, a green test command is not automatically proof: the test may not exercise the requested behavior, the environment may differ, or an existing failure may be mistaken for a regression. Verification links the task, revision, environment, command, output, and result instead of collapsing them into a confidence score. ## The five-part loop 1. **Task:** preserve the requested outcome and acceptance boundary. 2. **Change:** bind the exact repository revision and agent-produced patch. 3. **Execution:** run declared checks in a bounded environment. 4. **Evidence:** retain commands, outputs, artifacts, and failure classifications. 5. **Verdict:** report what passed, failed, or remained unverified without turning missing evidence into success. CodeVetter is being built around this loop. Its CLI, MCP boundary, and machine-readable verification bundle are the primary product surfaces; the desktop app is a local viewer. The current public benchmark is deliberately narrower: it contains 27 synthetic cases and 29 labeled findings for reproducible recognition scoring. It is useful evidence about that benchmark and is not proof of performance on arbitrary production repositories. ## When verification matters Verification is most valuable when an agent changes authorization rules, browser state, API contracts, persistence, concurrency, or regression-sensitive behavior. These tasks can look reasonable in a diff while failing at runtime. A good verifier makes the failure reproducible and makes uncertainty visible. ## What to inspect next Read the step-by-step verification workflow, compare review with verification, inspect the verification evidence bundle, and then open the public benchmark to see the cases, outputs, scorer, and limitations. **CTA:** Inspect the public benchmark or download CodeVetter to examine agent-written changes locally. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # How to verify AI-generated code > Verify AI-generated code with a runnable Node.js failure-and-fix example, executable acceptance checks, regression checks and reproducible evidence. Canonical: https://codevetter.com/verify-ai-generated-code To verify AI-generated code, start from the requested behavior and finish with reproducible evidence. Do not begin by asking a second model whether the patch looks good. That can help triage risk, but it cannot establish that the software works. ## 1. Freeze the task and revision Write down the requested outcome, the repository revision before the run, and the exact agent change. If the task is ambiguous, record the ambiguity instead of allowing the verifier to silently choose a friendlier interpretation. ## 2. Identify the behavioral boundary Name what users or callers can observe: a browser state transition, an API response, an authorization denial, a persisted record, or a regression that must remain fixed. Prefer checks against this boundary over checks that merely mirror the implementation. ## 3. Run the smallest authoritative checks Use repository-owned tests and commands first. Add a focused browser or API check when the existing suite does not exercise the changed behavior. Capture the command, environment, exit state, and bounded output. A passing command without identity or context is weak evidence. ## 4. Separate failures Distinguish an agent regression from an existing repository failure, an environment problem, a timeout, and an unverified requirement. This prevents infrastructure noise from becoming a false product verdict. ## 5. Package the evidence Bind the task, change, checks, outputs, artifacts, and verdict into a machine-readable bundle. A reviewer should be able to answer: what ran, against which revision, what changed, what failed, and what remains unknown? ## 6. Re-check after a fix A fix is not complete because the diff changed. Re-run the failing check and the smallest relevant regression set. Preserve both the failure and the later pass so the evidence shows closure rather than only the final state. CodeVetter follows this task-to-evidence shape and is designed to fail closed when required evidence is absent. Its current public benchmark demonstrates a narrower recognition task and publishes its own limitations; it should not be read as a production-wide guarantee. ## Try it: catch a real class of AI-code bug Suppose an agent is asked to preserve an explicit `false` setting while keeping the default and label behavior. A truthiness fallback can look reasonable in a review and still violate that requirement. This small, synthetic example is adapted from CodeVetter's [published explicit-false task](https://github.com/Codevetter/codevetter/tree/3a0e322aa21a20bf522392ae21b88f4bd40bb237/benchmarks/agent-tasks/sample/tasks/preserve-explicit-false); it illustrates the check, rather than a measured agent success rate. With Node.js installed, save this as `verify.mjs` and run `node verify.mjs`: ```js import assert from 'node:assert/strict'; function transform(input) { return { enabled: input.enabled || true, label: input.label ?? 'Default', }; } assert.equal(transform({ enabled: false }).enabled, false); assert.equal(transform({ label: 'Keep this' }).label, 'Keep this'); assert.equal(transform({}).enabled, true); ``` The first assertion fails: `false || true` becomes `true`. Change only `input.enabled || true` to `input.enabled ?? true` and run the same file again. All three assertions should pass. The default and label checks protect behavior that the fix must preserve. For your own patch, replace these assertions with checks at the changed API, browser or storage boundary. A unit example cannot establish that a deployed route or permission-dependent workflow works. Record the failing revision, the fixed revision and the same check's output for both. ## Reproduce the CodeVetter corpus check To inspect the product's evidence mechanism, clone the repository and follow its [corpus setup and qualification guide](https://codevetter.com/docs/development/agent-task-corpus). From the repository root, run: ```sh pnpm corpus:qualify --task preserve-explicit-false --json ``` Qualification executes the committed baseline and known-good change in fresh temporary workspaces and checks that the intended failure and later pass repeat. It does not call a model provider or demonstrate that an agent found the fix. Inspect the task identity, required check, preserved-label regression and cleanup state in the receipt before treating it as evidence. ## Decide what is ready to ship A passing example covers only its declared requirement. Before shipping your change, also exercise relevant integration paths, permission denials and failure recovery. If a required environment or check is unavailable, record it as unverified and state what remains to run. Keep the [evidence bundle](https://codevetter.com/verification-evidence-bundle) alongside the patch so another developer can assess the result. **CTA:** Use the checklist with your next agent patch, then inspect how CodeVetter structures verification evidence. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # AI code review vs. coding-agent verification > Validate AI-generated code with a runnable failure-and-fix check. Compare review with executable verification and inspect the evidence. Canonical: https://codevetter.com/ai-code-review-vs-verification ## Try an AI code validation check Requirement: preserve an explicit `false` setting, the default, and the existing label. A truthiness fallback (`input.enabled || true`) breaks that requirement; a nullish fallback (`input.enabled ?? true`) preserves it. The published synthetic task repeats the intended baseline failure twice and the known-good pass twice. [Inspect the executed qualification receipt](https://codevetter.com/examples/explicit-false-qualification.json), then [run the same failure-and-fix check locally](https://codevetter.com/verify-ai-generated-code#worked-example). The receipt qualifies this task and its checks; it does not measure an agent's success rate. AI code review and coding-agent verification solve related but different problems. Review searches a change for risks and explains suspicious code. Verification determines whether the requested behavior works and preserves evidence for the verdict. Teams using coding agents usually need both. | Question | AI code review | Agent verification | | --- | --- | --- | | Primary input | Diff and repository context | Task, exact change, environment, and checks | | Main output | Findings and explanations | Pass, fail, or unverified with evidence | | Strong at | Breadth, suspicious patterns, maintainability clues | Reproducing behavior and proving closure | | Main limitation | A plausible finding may not reproduce | A check can miss risks outside its behavioral boundary | | Best use | Risk discovery and reviewer focus | Acceptance, regression protection, and auditability | ## Why review alone is not enough A reviewer can notice an authorization branch that looks unsafe, but only an authoritative check can show whether an unauthorized request is accepted. A model can praise a state update while a browser journey still loses user input. Review remains valuable because it can propose where to test; it should not be promoted into runtime proof. ## Why tests alone are not enough A green suite can be irrelevant to the requested task, stale, or incomplete. Verification strengthens test output by binding it to the task, revision, environment, and expected behavior. It also preserves explicit unknowns instead of treating an absent check as a pass. ## A practical combined loop 1. Review the diff and repository context to identify risk. 2. Translate material risks and acceptance criteria into focused checks. 3. Run repository-owned tests plus the smallest missing behavioral check. 4. Package findings and execution results separately. 5. Re-run after fixes and retain the before-and-after evidence. CodeVetter's current direction is this combined evidence loop, with execution-backed verification as the authority. The public recognition benchmark remains useful for one review dimension and openly states that it does not prove production PR performance. **CTA:** Start with the verification workflow, then inspect the public benchmark and its limitations. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Coding-agent verification evidence bundles > See the identities, execution records, artifacts, uncertainty, and re-check linkage a coding-agent verification bundle should contain. Canonical: https://codevetter.com/verification-evidence-bundle A verification evidence bundle is the portable record behind a verdict. It should let a person, CI system, or later evaluation understand exactly what was checked without relying on a screenshot or a model summary. ## Required identities The bundle should bind the task, repository, base revision, agent change, verifier version, environment, and attempt. Without those identities, a passing output can be accidentally attached to the wrong patch or repeated under different conditions. ## Execution records For each check, retain the normalized command or runner identity, start and finish state, exit result, timeout or cancellation state, and bounded output. Record artifacts such as screenshots or reports by path and digest when they contribute to the verdict. Do not hide environment failures inside a generic test failure. ## Verdict and uncertainty A useful verdict separates passed requirements, reproduced failures, regressions, pre-existing failures, infrastructure failures, and requirements that remain unverified. The bundle should fail closed when an authoritative check is missing instead of filling the gap with a confidence score. ## Redaction and portability Evidence should be useful without leaking secrets or entire private repositories. Retain only the necessary excerpts and metadata, enforce output bounds, redact sensitive values, and make truncation explicit. JSON is useful for automation; Markdown and self-contained HTML are useful for human inspection. ## Re-check linkage When a failure is fixed, the new bundle should reference the earlier attempt and preserve the failing evidence. That creates a closure trail: failed behavior, corrective change, passing check, and regression result. CodeVetter's product direction makes the machine-readable verification bundle a primary surface. Completed reviews can already export fail-closed JSON, Markdown, and self-contained offline HTML evidence, while the broader agent-task evaluation contracts remain explicit about qualification and missing proof. **CTA:** Inspect the benchmark data format or download CodeVetter to review evidence locally. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeVetter vs Greptile: local verification or repository-aware review? > Compare CodeVetter's local execution-backed verification with Greptile's repository-aware AI code review and TREX execution workflow. Canonical: https://codevetter.com/codevetter-vs-greptile Greptile is a repository-aware code-review product that comments in the pull-request workflow and can hand findings to coding agents. CodeVetter is an execution-backed verifier that binds a requested task and exact change to checks, artifacts, and a measurable verdict. Choose based on the authority you need: review guidance or reproducible task evidence. ## The short comparison | Question | CodeVetter | Greptile | | --- | --- | --- | | Primary job | Verify whether an agent task worked and retain evidence | Review pull requests with repository context | | Main surface | Local desktop viewer, CLI/MCP, JSON/Markdown/HTML evidence | GitHub/GitLab review workflow, CLI, and agent handoff | | Execution | Runs declared repository checks as the verdict boundary | TREX is Greptile's execution layer; standard review remains comment-oriented | | Deployment | Local product with no required CodeVetter-hosted code service | Hosted cloud, with enterprise self-hosting documented | | Public comparison evidence | Own reproducible fixture benchmark and limitations | Own benchmark and product claims; no shared head-to-head run with CodeVetter | | Current CodeVetter availability, checked August 2026 | Apple-silicon macOS build through GitHub Releases; no CodeVetter subscription is currently declared, and model-provider costs are separate | Free Starter tier; Pro is listed at $30 per seat per month with included review credits | ## When Greptile is the better fit Greptile is built around continuous pull-request review with repository context. Its official pages describe full-codebase analysis, configurable rules, GitHub and GitLab support, comments that can be sent to coding agents, a CLI, and enterprise self-hosting. Its newer TREX surface can run code during review, which makes the boundary closer to verification than a purely static reviewer. Choose Greptile when the team wants review embedded in every pull request, shared administration, and a managed or self-hosted organization workflow. ## When CodeVetter is the better fit Choose CodeVetter when the primary artifact is a portable verification record. The product direction begins with the task and exact patch, executes repository-owned checks, classifies failures, preserves bounded outputs and artifacts, and emits pass, fail, or unverified without turning missing evidence into confidence. It is especially useful for agent evaluation, repeated regression comparison, and local or air-gapped evidence review. ## How TREX changes the comparison It would be inaccurate to describe Greptile as static-only. Greptile's official material now presents TREX as an execution layer. The remaining distinction is the product center of gravity and evidence contract: Greptile starts from managed pull-request review, while CodeVetter makes task identity, immutable execution evidence, and a portable verdict the primary product surface. Teams should test both against their own acceptance checks. ## No invented head-to-head result CodeVetter's public benchmark currently compares its review pipeline with a raw Claude baseline on 27 synthetic cases and 29 labeled findings. Greptile has not been run through that harness. Greptile also publishes its own benchmark material, but separately designed benchmarks cannot be treated as a common leaderboard. This page therefore makes no claim that either tool catches more bugs, produces fewer false positives, runs faster, or costs less per successful verification. ## A practical evaluation 1. Select five representative agent tasks with frozen acceptance criteria. 2. Run each product with its normal supported configuration. 3. Preserve every finding, executed check, skipped check, and provider setting. 4. Score task completion and regressions with repository-owned checks. 5. Report latency, cost, false positives, and missing evidence separately. ## Sources and freshness Product capabilities and pricing were checked on 7 August 2026 against Greptile's official documentation, pricing, security material, and CodeVetter's repository and public benchmark. Recheck the linked sources before procurement. **Next:** Inspect CodeVetter's benchmark and evidence-bundle format, then test the same representative tasks in your own repository. ## Primary sources - [www.greptile.com](https://www.greptile.com/) - [www.greptile.com](https://www.greptile.com/docs/code-review/key-features) - [www.greptile.com](https://www.greptile.com/blog/trex) - [www.greptile.com](https://www.greptile.com/pricing) - [www.greptile.com](https://www.greptile.com/security) - [codevetter.com](https://codevetter.com/benchmark) ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeVetter vs CodeRabbit: verification evidence or automated PR review? > Compare CodeVetter's execution-backed coding-agent verification with CodeRabbit's automated PR, IDE, and CLI code review workflow. Canonical: https://codevetter.com/codevetter-vs-coderabbit CodeRabbit and CodeVetter address different checkpoints. Choose CodeRabbit when you want an automated reviewer inside pull requests, an IDE, or a CLI. Choose CodeVetter when you need to bind an agent task and exact change to executable checks, retained evidence, and a pass, fail, or unverified verdict. A team can use both: review to discover risk, verification to prove behavior. ## The short comparison | Question | CodeVetter | CodeRabbit | | --- | --- | --- | | Primary job | Execution-backed verification of an agent change | Automated AI code review and development workflow | | Main surface | Local desktop viewer plus CLI/MCP and portable evidence bundles | Pull requests, IDE, CLI, and planning workflow | | Main output | Task-linked checks, artifacts, uncertainty, and measurable verdict | Review summaries, findings, suggestions, fixes, and conversations | | Repository access | No CodeVetter-hosted server; local execution, with any chosen model provider governed separately | Connected Git provider; official docs say review code may be shared with OpenAI or Anthropic, with enterprise self-hosting available | | Published proof | Reproducible 27-case recognition benchmark with data, scorer, outputs, and limitations | Product documentation and customer workflow evidence; not run in CodeVetter's public benchmark | | Current CodeVetter availability, checked August 2026 | Apple-silicon macOS build through GitHub Releases; no CodeVetter subscription is currently declared, and model-provider costs are separate | Free plan plus paid Pro and Pro+ plans with per-developer limits | ## When CodeRabbit is the better fit CodeRabbit is designed to meet developers where review already happens. Its official documentation covers automatic pull-request reviews, incremental follow-ups, IDE and CLI review, codebase knowledge, linked repositories, planning, and one-click fixes. If the immediate need is consistent review comments across a team and Git provider, that is its natural workflow. Its data boundary is also different. CodeRabbit's documentation says proprietary code can be shared with OpenAI or Anthropic for review and describes storage controls and enterprise self-hosting. Teams should evaluate those controls against their own repository and compliance requirements. ## When CodeVetter is the better fit CodeVetter starts from the requested task rather than the review comment. It records the exact change, runs declared checks in a bounded environment, separates agent regressions from environment and pre-existing failures, and preserves a machine-readable verdict. That matters when a plausible-looking patch must demonstrate browser behavior, API behavior, authorization, persistence, concurrency, or regression closure. CodeVetter's desktop app is a local viewer. The primary product direction is the CLI/MCP verification boundary and portable evidence bundle. Model-backed analysis may still use the provider the operator chooses; the important distinction is that CodeVetter does not require a CodeVetter-hosted code service. ## What the public benchmark does and does not prove CodeVetter publishes 27 synthetic cases with 29 labeled findings, reviewer outputs, scoring rules, and explicit limitations. The current baseline compares CodeVetter with a raw Claude review under that fixture set. CodeRabbit has not been run through the same harness, so this page makes no catch-rate, precision, speed, or cost claim against CodeRabbit. A valid head-to-head result would require the same immutable cases, configuration disclosure, repeated runs, and scorer for both products. ## A practical combined workflow 1. Let CodeRabbit review a pull request and identify suspicious code. 2. Translate material findings and task acceptance criteria into focused executable checks. 3. Run CodeVetter against the exact agent change. 4. Keep review findings and execution evidence separate. 5. Re-run failed behavior after the fix and preserve both attempts. ## Sources and freshness Product capabilities and pricing were checked on 7 August 2026 against CodeRabbit's official documentation and CodeVetter's repository and public benchmark. Product plans change; use the linked primary sources for the current details. **Next:** Inspect the public benchmark, read the review-versus-verification guide, or download CodeVetter for local evaluation. ## Primary sources - [docs.coderabbit.ai](https://docs.coderabbit.ai/) - [docs.coderabbit.ai](https://docs.coderabbit.ai/management/plans) - [docs.coderabbit.ai](https://docs.coderabbit.ai/faq) - [codevetter.com](https://codevetter.com/benchmark) ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeVetter articles > Editorial articles on execution-backed coding-agent verification, reproducible evidence, and verification receipts. Canonical: https://codevetter.com/articles Editorial articles on execution-backed coding-agent verification, reproducible evidence, and verification receipts. - [Agent evaluation vs tests vs code review: what each can actually prove](https://codevetter.com/articles/agent-evaluation-vs-tests-vs-code-review) — Understand the distinct proof boundaries of agent evaluation, software testing, and static AI code review when checking agent-written software changes. - [What an evidence receipt for a coding agent should contain](https://codevetter.com/articles/coding-agent-evidence-receipt-spec) — Learn what a machine-readable evidence receipt for AI coding agents must contain, including identity bindings, execution records, taxonomy, and failure trails. - [How to verify AI-generated code changes beyond code review](https://codevetter.com/articles/verifying-ai-code-changes-beyond-code-review) — Static AI code review cannot prove software works. Learn how to bind task intent, run executable checks, and capture evidence for AI code changes. - [Why reproducible evidence matters when an AI agent says a task is done](https://codevetter.com/articles/why-reproducible-evidence-matters-when-an-ai-agent-says-a-task-is-done) — When an AI coding agent says a task is done, how do you verify it? Learn why reproducible execution evidence matters more than LLM-as-judge opinions. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Tools that shaped CodeVetter > A detailed, source-linked ledger of the tools and ideas that shaped CodeVetter's approach to code context, agent workflows, and executable verification. Canonical: https://codevetter.com/inspiration A source-linked ledger of products and open-source tools whose work informed CodeVetter's research and design. Inclusion is independent appreciation, not endorsement, affiliation, code reuse, or feature parity. - [Aider](https://codevetter.com/inspiration/aider) — A small, ranked repository map gives an agent useful context without dumping the whole tree. (design reference) - [Greptile](https://codevetter.com/inspiration/greptile) — A code review must follow a change beyond the diff. (design reference) - [CodeRabbit](https://codevetter.com/inspiration/coderabbit) — Review is more useful when it arrives in the developer’s normal change workflow. (design reference) - [Qodo / PR-Agent](https://codevetter.com/inspiration/qodo) — A review should be close to tests and the intent behind a change. (research reference) - [Ellipsis](https://codevetter.com/inspiration/ellipsis) — The useful review loop continues from finding to attempted fix. (research reference) - [Sourcery](https://codevetter.com/inspiration/sourcery) — Fast review can combine code quality signals with readable feedback. (research reference) - [Bito](https://codevetter.com/inspiration/bito) — Repository architecture and external project context can change the meaning of a diff. (research reference) - [Graphite Agent](https://codevetter.com/inspiration/graphite) — Review quality depends on the shape and timing of the change. (research reference) - [Panto AI](https://codevetter.com/inspiration/panto) — Design documents, tickets, and code history can be relevant to review. (research reference) - [GitHub Copilot code review](https://codevetter.com/inspiration/github-copilot) — A code host can make automated review a normal request on a pull request. (research reference) - [DeepWiki](https://codevetter.com/inspiration/deepwiki) — A repository can be made legible quickly through structured explanation. (design reference) - [Sourcegraph](https://codevetter.com/inspiration/sourcegraph) — Finding the right code is a first-class engineering workflow. (design reference) - [Augment Code](https://codevetter.com/inspiration/augment-code) — Fresh repository context should follow the developer across tools. (research reference) - [Arbor](https://codevetter.com/inspiration/arbor) — Deterministic graph traversal is a useful answer to structural questions. (design reference) - [LogicStamp Context](https://codevetter.com/inspiration/logicstamp) — Architecture facts can be compiled into deterministic, diffable contracts. (research reference) - [Repomix](https://codevetter.com/inspiration/repomix) — A repository packet should be portable and deliberate about what it includes. (research reference) - [Swimm](https://codevetter.com/inspiration/swimm) — Documentation is more useful when it remains connected to code. (research reference) - [Google Code Wiki](https://codevetter.com/inspiration/google-code-wiki) — Repository explanations should be refreshed as code changes. (research reference) - [OpenDeepWiki](https://codevetter.com/inspiration/opendeepwiki) — Repository documentation can be self-hosted and available to agents. (research reference) - [Claude Code](https://codevetter.com/inspiration/claude-code) — An agent can discover context through targeted search and reads. (design reference) - [OpenAI Codex](https://codevetter.com/inspiration/codex) — An agent’s patch and its execution trace should be available for inspection. (design reference) - [Cursor](https://codevetter.com/inspiration/cursor) — Codebase navigation can feel immediate inside a coding workflow. (design reference) - [Windsurf](https://codevetter.com/inspiration/windsurf) — An agent’s context has to be assembled from several live signals. (research reference) - [Continue](https://codevetter.com/inspiration/continue) — The developer should be able to choose models and context sources. (research reference) - [Cline](https://codevetter.com/inspiration/cline) — Planning and acting benefit from a visible boundary. (research reference) - [Gemini CLI](https://codevetter.com/inspiration/gemini-cli) — A terminal agent can keep planning, tools, and code changes in one inspectable loop. (research reference) - [Tree-sitter](https://codevetter.com/inspiration/tree-sitter) — Syntax trees make source relationships repeatable and fast enough for local tooling. (in the product) - [ast-grep](https://codevetter.com/inspiration/ast-grep) — Structural search can express focused review rules more clearly than text matching. (in the product) - [Semgrep](https://codevetter.com/inspiration/semgrep) — Security and bug patterns can be written as reviewable code-shaped rules. (research reference) - [CodeQL](https://codevetter.com/inspiration/codeql) — Deep code queries can reveal data flow and security relationships. (research reference) - [ripgrep](https://codevetter.com/inspiration/ripgrep) — Fast, plain text search is often the right first step in code exploration. (research reference) - [GitNexus](https://codevetter.com/inspiration/gitnexus) — A code graph can be exposed as a useful interface for agents. (research reference) - [gitoxide](https://codevetter.com/inspiration/gitoxide) — Git history can be explored through an owned Rust interface. (research reference) - [SCIP](https://codevetter.com/inspiration/scip) — An interchange format can connect symbol and reference information across tools. (research reference) - [codesize](https://codevetter.com/inspiration/codesize) — A tiny heuristic can still point a reviewer toward risky code. (research reference) - [Unsupervised Finder](https://codevetter.com/inspiration/unsupervised-finder) — Search a space of possible explanations before asking an agent to tell a story. (design reference) - [DeepWork](https://codevetter.com/inspiration/deepwork) — Long agent tasks need explicit steps, artifacts, and quality gates. (design reference) - [Karpathy’s LLM Wiki](https://codevetter.com/inspiration/llm-wiki) — Knowledge can accumulate as a maintained, inspectable artifact. (design reference) The [wider research archive](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) also names surveyed alternatives that did not yield a specific product lesson. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Agent evaluation vs tests vs code review: what each can actually prove > Understand the distinct proof boundaries of agent evaluation, software testing, and static AI code review when checking agent-written software changes. Canonical: https://codevetter.com/articles/agent-evaluation-vs-tests-vs-code-review ## Understanding Quality Gates in the Era of AI Coding Agents As AI coding agents take on complex software tasks, technical leaders face a critical question: how do we know an agent-authored pull request is safe to merge? In traditional software development, quality assurance relies on **automated test suites** (unit, integration, and end-to-end tests) and **human code review**. With generative AI, organizations have added **static AI code review**, where an LLM inspects diffs. More recently, **execution-backed agent evaluation** systems have emerged to assess whether an agent completed a requested task correctly. However, teams frequently blur the boundaries between these mechanisms, treating them as interchangeable. This leads to misplaced confidence—such as assuming a green test suite proves an agent understood the prompt, or believing an AI review comment proves a bug was fixed. To build a reliable software delivery pipeline for AI-generated code, teams must understand what static code review, software testing, and execution-backed agent evaluation can—and cannot—actually prove. ## The Core Comparison Matrix The table below outlines the fundamental differences across all three mechanisms: | Dimension | Static AI / Human Code Review | Automated Testing (Unit / Integration / E2E) | Execution-Backed Agent Evaluation | | --- | --- | --- | --- | | **Primary Input** | Code diff, repository files, style rules | Source code, test scripts, test data | Task intent, repository revision, patch, executable checks | | **Primary Output** | Text findings, style suggestions, risk flags | Pass/fail counts, stack traces, coverage reports | Portable evidence receipt, failure taxonomy, completion verdict | | **What It Establishes** | Source-level findings and maintainability judgments | Whether specific assertions pass under defined inputs | Evidence for the acceptance criteria and regression checks actually exercised | | **Core Limitation** | Cannot execute code; plausible diffs can fail | Cannot determine if tests match prompt intent | Requires explicit executable checks; bounded by runner | | **Failure Mode** | False positives or false praise from text | Green suite passing on an incomplete task | Incomplete checks can still miss defects; missing evidence must remain explicit | ## 1. Static Code Review: Evaluating Appearance and Maintainability Static code review—performed by engineers or AI models—inspects source text. It evaluates structure, style, and architectural patterns without executing the application. ### What Static Code Review Can Prove - **Conformity to Style and Conventions:** Verifies that variable naming, organization, and formatting match repository standards. - **Surface Risk Identification:** Flags known anti-patterns, such as hardcoded credentials or missing error-handling blocks. - **Maintainability and Readability:** Evaluates whether code is modular and understandable for future maintainers. ### What Static Code Review Cannot Prove - **Operational Correctness:** A diff can be beautifully formatted and completely broken at runtime due to an unhandled null pointer or async race condition. - **Task Completion:** A reviewer inspecting a diff cannot verify if an API endpoint handles real HTTP headers or whether a migration executes cleanly. ## 2. Automated Testing: Verifying Specific Code Assertions Software testing executes code within a controlled runner to verify that specific functions satisfy predefined code assertions. ### What Testing Can Prove - **Assertion Validity:** Proves that specified code paths yield expected return values under defined test inputs. - **Regression Detection for Covered Code:** Demonstrates that existing behaviors continue to function after new code is added. - **Performance Benchmarks:** Measures execution wall time, memory consumption, or CPU utilization for specific benchmark functions. ### What Testing Cannot Prove - **Alignment with Task Intent:** A coding agent can satisfy test runners by altering test assertions or deleting failing tests entirely. The test suite passes, but the business requirement is unsatisfied. - **Correctness of Un-Tested Requirements:** If a task requires handling an edge case that lacks a written test, a passing test suite provides zero proof that the edge case works. ## 3. Execution-Backed Agent Evaluation: Verifying Task Completion Execution-backed agent evaluation helps bridge the "intent-to-execution" gap in autonomous software development. It connects the prompt, the exact patch, isolated local execution, and structured evidence collection. Someone still has to establish that the checks represent the requested behavior. ``` [User Task Prompt] ──> [Agent Patch] ──> [Executable Verification] ──> [Evidence Receipt] │ ├── Unit / Integration Tests ├── Headless Browser Flows └── Local API Assertions ``` ### What Agent Evaluation Can Prove - **Covered Acceptance Criteria:** Records whether mapped checks pass at a behavioral boundary, such as headless browser state or API endpoints. Unchecked requirements remain unverified. - **Failure Classification:** Compares failure evidence with baseline runs and environment observations. A taxonomy label alone does not establish causation. - **Auditability and Re-Checks:** Can link an initial failure to a passing re-check when the runner records both attempts and their source identities. The link is bounded evidence, not a guarantee that every regression is resolved. ### What Agent Evaluation Cannot Prove - **Universal Code Quality:** A passing evaluation covers its checks, inputs, environment, and revision. It cannot establish universal runtime correctness or architectural quality. - **Coverage Beyond Executable Checks:** Agent evaluation cannot verify behavior for which no executable check or browser automation can be constructed. ## Why Relying on Any Single Mechanism Fails Engineering organizations relying on only one mechanism face predictable failure modes: 1. **Code Review Alone:** Merges "plausible-looking code", leading to runtime outages, state corruption, or API breaks. 2. **Tests Alone:** Results in "passing suites with drift," where agents satisfy test runners by altering assertions or ignoring un-tested edge cases. 3. **Agent Evaluation Alone:** Can miss untested behavior and maintainability problems even when all selected checks pass. ## Building a Unified Quality Pipeline for AI Code Technical teams should integrate all three mechanisms into a unified, complementary quality loop: ``` ┌─────────────────────────────────────────┐ │ 1. STATIC CODE REVIEW │ │ Discover risks, check style & structure │ └────────────────────┬────────────────────┘ │ ▼ ┌─────────────────────────────────────────┐ │ 2. EXECUTABLE AGENT EVALUATION │ │ Check mapped criteria & browser/API │ └────────────────────┬────────────────────┘ │ ▼ ┌─────────────────────────────────────────┐ │ 3. AUTOMATED REGRESSION SUITE │ │ Re-run tests to catch side effects │ └─────────────────────────────────────────┘ ``` 1. **Use Static Review for Triage:** Let static AI code review scan incoming diffs to identify structural risks and maintainability concerns. 2. **Translate Acceptance Criteria into Executable Checks:** Ensure changes are exercised against real behavioral boundaries, such as Playwright browser flows or local API tests. 3. **Generate Machine-Readable Evidence Receipts:** Capture execution telemetry, failure taxonomies, and resource bounds into portable JSON receipts. 4. **Require Closure Trails Before Merging:** Require the agent to apply a fix and generate a passing re-check receipt linked to the original failure before approving the pull request. ## Next Action for Engineering Leadership Audit your engineering team's current pull request process for AI-generated code. Establish an execution-backed evaluation layer that binds user prompt intent to isolated, executable checks and structured evidence receipts. To inspect how local execution-backed verification functions in practice, download CodeVetter or review our public benchmark methodology. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # What an evidence receipt for a coding agent should contain > Learn what a machine-readable evidence receipt for AI coding agents must contain, including identity bindings, execution records, taxonomy, and failure trails. Canonical: https://codevetter.com/articles/coding-agent-evidence-receipt-spec ## Why Informal Agent Self-Reports Fail Auditability When an autonomous coding agent completes a task, it typically posts a pull request accompanied by an informal Markdown summary describing what it claims to have changed and tested. While readable, these text summaries are unsuited for technical auditability or automated verification. An agent self-report suffers from four fundamental flaws: 1. **Non-Verifiable Claims:** An agent asserting that "all tests passed" provides no proof that tests were actually executed against the checked-in commit SHA. 2. **Missing Execution Context:** Summaries rarely record the precise environment variables, node versions, database states, CLI flags, or mock settings used during execution. 3. **Selective Omission:** A summary can omit unexecuted suites, skipped assertions, timeouts, or intermediate failures. 4. **Lack of a Stable Contract:** Free-form summaries lack the schema needed for reliable automated validation. Structured, machine-readable **verification evidence receipts** make these claims easier to inspect. They complement source review and runner controls; the receipt format itself does not make an agent safe. ## General Receipt-Design Principles vs. Current Implementations A verification evidence receipt is a versioned, portable artifact recording the evidence its producer collected, along with missing measurements and trust limitations. In general receipt design, an evidence specification defines structured schemas that bind source code identities to execution telemetry. In practical software systems, these fall into two distinct categories: - **Experimental / Ingested Project Receipts:** Ingestion schemas (such as `codevetter.project-verification-receipt/v1`) that parse raw external runner outputs (Playwright JSON, JUnit XML, LCOV) without mutating local state or asserting full system authority. - **Persisted Native / Local Receipts:** Local execution receipts (such as `codevetter.local-check/v1`) emitted and stored in local databases during supervised verification passes. The sections below propose a general design checklist, not a single shipped CodeVetter schema. The experimental project-receipt analyzer does not execute tests, discover commands, or persist receipts into the desktop database. It cannot supply measurements absent from its input. Hashes identify content; they do not authenticate the producer or make storage immutable. ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ VERIFICATION EVIDENCE RECEIPT SCHEMA (v1) │ ├─────────────────────────────────────────────────────────────────────────┤ │ 1. Identity & Scope │ Base Commit, Patch SHA, Task ID, Runner Profile │ │ 2. Execution Inventory │ Commands, Exit Codes, Output Hash, Duration │ │ 3. Failure Taxonomy │ Agent Regressions vs. Environment vs. Pre-Exist│ │ 4. Resource Telemetry │ Peak RSS, Wall Time, CPU, Egress, Sandbox State │ │ 5. Explicit Bounds │ Unverified Requirements, Missing Test Paths │ │ 6. Fix Closure Trail │ Linkage to Initial Failure Receipt & Re-check │ └─────────────────────────────────────────────────────────────────────────┘ ``` Below is a detailed breakdown of what each section of a robust evidence receipt specification must contain. ## 1. Identity and Scope Bindings An evidence receipt must be linked unambiguously to a specific source tree and task description. The receipt begins by pinning identity references: - **Repository & Revision Identity:** Absolute Git commit SHA of the base repository, target branch, and exact SHA-256 digest of the agent's patch file. - **Task Intent Reference:** A unique task identifier, along with a digest or text snapshot of the original prompt and acceptance criteria. - **Environment & System Profile:** OS kernel version, architecture (e.g., `darwin-arm64`), runtime engine version (e.g., Node.js, Rust), and installed CLI versions. - **Verifier Profile:** The specific verifier name, executable version, and schema version (e.g., `codevetter.local-check/v1`). Checking these identities against the candidate revision helps detect "replay drift"—where an older receipt is attached to a new pull request. Declared identity fields alone do not prove which code the producer executed. ## 2. Execution Records and Inventory The execution section records every command invoked during the verification pass, documenting exact inputs and outputs: - **Command Identity:** The selected command and safe arguments, with credential-bearing values redacted. Record omissions explicitly rather than copying secrets into a receipt. - **Terminal Exit States:** Numeric exit codes, process signal terminations (e.g., `SIGKILL`), and cancellation flags. - **Bounded Output Captures:** SHA-256 hashes of stdout and stderr logs, accompanied by bounded text excerpts (e.g., capped at 64 KiB) capturing the failure tail. - **Test Inventory Details:** Individual counts for total discovered tests, passed tests, failed tests, skipped tests, and suite durations. - **Stable Failure Signatures:** Normalized, hashed representations of error messages and stack traces, stripping non-deterministic elements like timestamps. ## 3. Failure Taxonomy and Classification A simple pass/fail flag collapses important operational distinctions. A receipt can record the following classifications when baseline and runner evidence support them; otherwise attribution should remain uncertain: - **Agent-Introduced Defect:** The test failed due to a logical error or broken assertion directly within the code written by the agent. - **Side-Effect Regression:** A test outside the immediate scope of the requested task failed after the patch was applied, indicating unexpected breakage. - **Pre-Existing Failure:** The check failed identically on the base commit SHA before the agent's patch was applied. - **Operational / Environment Failure:** Execution was aborted due to infrastructure conditions, such as missing dependencies or port conflicts. - **Transient Recovery:** A test that failed initially but passed upon immediate re-execution, flagged for flaky behavior tracking. ## 4. Resource, Egress, and Safety Boundaries Executing arbitrary agent code presents security and resource risks. A receipt should describe the controls and observations available to assess those risks, without treating incomplete telemetry as proof of safety: - **Resource Consumption:** Peak process-tree Resident Set Size (RSS memory in MiB), sampled CPU usage, wall-time duration, and process spawn counts. - **Network Egress Evidence:** Observed requests or escapes, collection coverage, and the policy actually enforced. An absent network field is not evidence of zero egress; destination details should be minimized or redacted where sensitive. - **Sandbox State:** The configured isolation boundaries and evidence that enforcement was active. A producer's declaration alone is not independent attestation. ## 5. Explicit Uncertainty and Limitation Bounds Missing evidence is not proof of success. If an agent modifies an API route that lacks test coverage, the evidence receipt must make that gap explicit. The limitations section records: - **Unverified Acceptance Criteria:** User requirements that could not be mapped to an executable check. - **Unexecuted Code Paths:** Files modified by the agent that were not exercised by any executed test runner (supported by LCOV or Cobertura XML reports). - **`no_confidence` States:** Sections where telemetry collection was partial or where test reports lack revision-bound selection or resource measurements. ## 6. Fix Linkage and Closure Trails When an agent run fails verification, a fix cycle begins. A production-grade evidence receipt specification supports receipt chaining to establish an auditable history of resolution: 1. **Initial Failure Receipt:** Emitted when the agent's first patch fails an executable check, containing the failing signature and diff. 2. **Fix Candidate Linkage:** The corrective agent run references the initial failure receipt by its unique SHA-256 bundle identity. 3. **Targeted Re-Check Execution:** The new verification run re-executes the exact failing check identified in the initial receipt. 4. **Re-Check Receipt:** Records that the targeted check passed at the new revision and links the retained attempts. Broader regression checks and incomplete coverage remain separate from that result. ## Next Action for Platform Engineers Incorporate structured evidence receipts into your agent execution infrastructure. Replace informal Markdown summaries with a versioned, schema-validated JSON receipt specification that captures base commit identities, command inventory, failure taxonomy, resource bounds, and explicit limitation boundaries. To inspect executable receipt schemas and CLI ingestion tooling, review CodeVetter's open verification documentation or download the local desktop viewer. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # How to verify AI-generated code changes beyond code review > Static AI code review cannot prove software works. Learn how to bind task intent, run executable checks, and capture evidence for AI code changes. Canonical: https://codevetter.com/articles/verifying-ai-code-changes-beyond-code-review ## Moving Beyond Static Opinions in AI Code Verification Large language models and autonomous coding agents can generate hundreds of lines of code in seconds. However, engineering velocity breaks when developers are asked to approve pull requests they did not author and cannot easily reason about. The default response across many software organizations has been to insert another AI model as a static code reviewer. Static AI code review is valuable for discovering obvious risk patterns, flagging style inconsistencies, or explaining complex diffs. But static review cannot prove that software actually works. An LLM reviewing a diff reads source code as text; it does not execute the program, initialize a database, manage browser state, or process HTTP headers. A pull request generated by an agent can look clean, include docstrings, pass linter checks, and still fail silently at runtime. Relying solely on static code review to verify autonomous code changes introduces a dangerous gap between plausible-looking source text and verified operational reality. To establish confidence in AI-generated code, software teams must move beyond code review to execution-backed verification. ## The Structural Limits of Code Review for Agent Outputs To understand why code review alone is insufficient for AI-generated code, consider how autonomous agents construct changes. When an agent attempts a task—such as updating an authentication flow or modifying a database query—it operates against statistical likelihoods and context windows. This creates specific failure modes that static code review routinely misses: 1. **Plausible Lookalike Code:** Agents frequently generate code patterns that mirror surrounding repository style but use invalid assumptions regarding library APIs, internal helper contracts, or async execution timing. 2. **Unseen Edge-Case Behavior:** A diff modifying an authorization check may look mathematically sound in isolation while failing under specific session states, cookie configurations, or concurrent requests. 3. **Decoy and Scope Drift:** Agents tasked with fixing a bug can inadvertently edit adjacent, working functions or rewrite working tests to fit a broken implementation. 4. **Environment-Dependent Mutations:** Changes involving DOM interactions, local storage, network retries, or database transactions cannot be verified by inspecting syntax trees. When a human or an LLM reviews these diffs statically, they evaluate plausibility. But software correctness is not a matter of plausibility; it is a matter of executable behavior under defined constraints. ## The 5-Part Executable Verification Loop Execution-backed verification bridges the gap between agent intent and operational truth. Rather than asking a model whether a diff looks correct, executable verification subjects the change to deterministic checks and captures reproducible evidence. The following is a recommended workflow, not a claim that every CodeVetter receipt path implements every phase. In particular, experimental project-receipt ingestion consumes existing runner evidence; it does not execute checks, discover commands, or persist results into the desktop database. A verification loop consists of five distinct phases: ``` [Task Intent] ──> [Agent Change] ──> [Executable Checks] ──> [Captured Evidence] ──> [Measurable Verdict] ``` ### 1. Freeze Task Intent and Scope Verification begins by binding the exact requested behavior and its acceptance boundaries. If the original task description is ambiguous, the verifier records that ambiguity explicitly rather than allowing the agent or evaluator to invent a friendly interpretation. ### 2. Bind the Repository Revision and Change The exact base Git commit SHA, the workspace state, and the agent's patch must be pinned together. Verification must run against a clean, pinned revision so that external workspace drift cannot alter the outcome. ### 3. Execute Bounded Behavioral Checks The verifier runs authoritative, deterministic checks against the modified codebase. These checks prioritize repository-owned unit and integration tests, supplemented by focused browser journeys or API calls when existing test suites do not exercise the changed behavior. ### 4. Capture Raw Execution Evidence Record the evidence the runner actually captures: command identity, exit codes, bounded and redacted output, resource measurements, network observations, and failure signatures where available. Mark missing measurements and collection limits explicitly. A command-level memory measurement is not automatically a process-tree measurement, and missing network telemetry does not mean zero egress. ### 5. Render a Measurable Verdict Use the chosen verifier's documented status vocabulary. CodeVetter's experimental project-receipt analyzer reports `passed`, `failed`, or `no_confidence` separately for correctness, performance, safety, inventory, and overall status. Missing evidence remains explicit; a passing dimension does not establish that unchecked requirements are safe. Unsupported receipt formats are rejected before analysis. ## Establishing Behavioral Boundaries: Browser State and APIs Code review evaluates syntax; verification evaluates behavioral boundaries. When verifying agent changes, execution must target the exact interface boundaries exposed to users or system callers. ### Browser Journeys and User Interface State For web applications, subtle frontend regressions—such as lost input focus, incorrect state hydration, broken event handlers, or race conditions during async fetches—frequently pass static diff inspection. Executable verification uses headless browser automation (such as Playwright) to run end-to-end flows against local development servers, capturing DOM states, console errors, and visual evidence. ### API Contracts and Authorization When an agent updates API endpoints or middleware, verification requires executing actual HTTP requests against isolated local endpoints. Checks must explicitly test both valid payload paths and boundary conditions, such as unauthorized access attempts and malformed inputs. ## Failure Taxonomy: Separating Regressions from Environment Noise A common flaw in automated verification systems is treating every failed command as an agent bug. If a test fails because a local database port was busy, or because an external API timed out, blaming the coding agent distorts quality metrics. A useful verification workflow records failure classifications when the evidence supports them. The labels below are design guidance, not proof of causal attribution: - **Agent Defect / Failure:** The executed check failed explicitly due to broken logic or unhandled exceptions introduced by the agent's change. - **Possible Regression:** A previously passing test failed after the change. Check comparable baseline conditions, repeatability, and environment differences before attributing the failure to the patch. - **Pre-Existing Failure:** The test was already failing on the base commit prior to the agent's run. - **Operational / Environment Failure:** Execution was aborted due to infrastructure conditions, such as missing binaries, port conflicts, or memory limits. - **Unverified Requirement:** No authoritative test or runner existed to exercise the requested acceptance criteria. Keep uncertain attribution explicit. Before/after observations alone cannot rule out environment changes or flaky tests. ## Fix Closure and Re-Check Linkage When verification identifies a failure in an agent's patch, the failure evidence becomes structured feedback for a corrective agent iteration. To document a bounded fix and re-check trail: 1. **Preserve the Initial Failure:** The failing run, command, exit code, and failure signature are retained in a persisted evidence record. 2. **Apply the Corrective Patch:** The agent applies a targeted fix in an isolated, clean workspace. 3. **Re-Run the Failing Check:** The verifier re-executes the exact test that previously failed to confirm resolution. 4. **Link Before-and-After Evidence:** Link the original failure to the passing re-check, including revision and environment identities. This establishes the recorded check's outcome, not the absence of every regression. Preserve broader checks and unresolved limitations separately. ## Next Action for Technical Teams Do not rely on second-model LLM review alone to approve agent pull requests. Evaluate your workflow against the five-part loop: freeze task intent, isolate the revision, run behavioral checks, capture evidence, and inspect failure-to-pass re-checks alongside remaining coverage gaps. To examine how local verification functions in practice, download CodeVetter or inspect our public benchmark methodology. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Why reproducible evidence matters when an AI agent says a task is done > When an AI coding agent says a task is done, how do you verify it? Learn why reproducible execution evidence matters more than LLM-as-judge opinions. Canonical: https://codevetter.com/articles/why-reproducible-evidence-matters-when-an-ai-agent-says-a-task-is-done The adoption of AI coding agents has shifted the bottleneck of software engineering. We are no longer constrained by the speed at which we can draft new features. Instead, the primary constraint has become verification. When an autonomous coding agent completes a task and reports "done," how do we actually know the change is correct? In many workflows, the answer is manual inspection. A developer must read the generated diff, pull the branch, run the test suite, and click through the application to ensure nothing broke. This process is time-consuming and negates the velocity gains the agent was supposed to provide. To bridge this trust gap, the industry must move beyond trusting an agent's self-reported completion and beyond asking a second model for its opinion. We need reproducible execution evidence. ## The Limits of LLM-as-Judge and Static Review As the volume of agent-generated code increases, a common response has been to automate the review process using the same technology. This approach involves feeding the diff to another language model and asking it to identify defects. Hosted pull request bots analyze the structure of the change and flag potential issues. While static AI review can catch syntax errors or stylistic inconsistencies, it suffers from a fundamental limitation: it is not executable proof. An LLM reading a diff is providing an opinion based on pattern matching. It does not know if the code will compile. It does not know if a new API endpoint correctly parses the payload. Furthermore, static review tools lack the broader execution context of the repository. They view the diff in isolation, perhaps with some surrounding file context, but they do not run the application. When an agent hallucinates a variable name, a static reviewer might miss the error if the hallucination looks plausible. Executable outcomes outrank model opinion. When the goal is to determine whether a software task was completed correctly, a deterministic test failure is infinitely more valuable than a probabilistic model's approval. ## Defining Reproducible Execution Evidence If static review is insufficient, what is the alternative? The answer lies in reproducible execution evidence. Execution evidence is generated by running the code in a deterministic environment and observing its behavior. It moves the verification process from the realm of "looks correct" to "provably works." Reproducible evidence means that any developer, or any other agent, can take the same code, run the same checks, and achieve the exact same results without relying on a hidden cloud state. In practical terms, reproducible execution evidence includes: - **Build Logs:** Does the code compile without errors? - **Test Results:** Do the unit and integration tests pass? Did the agent's change break an unrelated test? - **Runtime Behavior:** For web applications, does the server start? Can a script interact with the newly created API? By capturing this evidence deterministically, we create a durable, machine-readable record of the software's state. This record serves as the source of truth for whether the agent succeeded. ## The Verification Loop To effectively utilize execution evidence, it must be integrated into a tight loop. At CodeVetter, we structure this around a specific process: **task → agent change → executable verification → evidence → measurable verdict**. 1. **Task:** The workflow begins with a clear intent, such as fixing a bug. 2. **Agent Change:** The coding agent executes the task, modifying the local repository. 3. **Executable Verification:** Instead of immediately asking a human to review the diff, the system runs deterministic checks. This might involve running a Playwright script. 4. **Evidence:** The verification step produces concrete artifacts—JSON test reports, build logs, and traces. 5. **Measurable Verdict:** The evidence is synthesized into a clear outcome. Did the tests pass? Was the bug fixed? If the verdict is a failure, the evidence is fed back to the agent as a concrete diagnostic. The agent is given the exact compiler error or the precise test assertion that failed. This deterministic feedback loop allows the agent to iteratively correct its work. ## A Concrete Example: The API Boundary Consider a web development task: a developer asks an agent to update a backend to support a new query parameter on a search endpoint. The agent modifies the controller, adds the logic, and reports the task as complete. A static AI reviewer might look at the diff, confirm that the syntax is valid, and approve the pull request. However, an execution-backed verification system takes a different approach. It runs a local test script that starts the server and sends an HTTP request with the new parameter. Imagine the agent made a subtle error: it parsed the parameter as a string, but the database function expects an integer. The static reviewer missed this because it didn't trace the type through the application hierarchy. But the executable verification catches it instantly. The request returns a `500 Internal Server Error`, and the test framework captures the stack trace indicating a type mismatch. The system captures this execution evidence and provides a measurable verdict: Failure. The developer (or the agent itself) can now see exactly what went wrong, supported by a reproducible runtime trace. There is no ambiguity. * ## Local, Bounded, and Reversible Workflows A critical aspect of reproducible evidence is where and how it is generated. For verification to be fast, secure, and iterative, it should be local to the repository. Many AI coding tools rely on hosted infrastructure—pushing code to a cloud service where a PR bot analyzes the diff. This introduces latency, requires granting third-party access to proprietary source code, and creates a disconnect between the developer's local environment and the verification environment. A local, bounded workflow means the execution evidence is gathered exactly where the agent made the changes: on the developer's machine or within an isolated, deterministic local sandbox. - **Privacy and Security:** Code never leaves the local machine for verification. There is no need to upload proprietary logic to a third-party server just to run a test suite. - **Speed:** Local execution eliminates network latency. A developer can ask an agent for a change, and the verification loop can run in seconds, right in the same workspace. - **Reversibility:** By keeping the workflow local and bound to the repository's version control system, changes are easily reversible. If the executable verification shows a catastrophic failure, the local Git worktree can be reset instantly. At CodeVetter, our architecture reflects this principle. The verification engine is a native desktop binary and Rust core that operates over local SQLite. It does not require a hosted server to evaluate whether a local application functions correctly. * ## Moving from Trust to Verification The goal of AI coding agents is not to write code that looks good; it is to write code that works. As long as we rely on manual inspection or non-executable model opinions to bridge the trust gap, we will be bottlenecked by the limits of static analysis. By demanding reproducible execution evidence, engineering teams can change their relationship with coding agents. Instead of acting as full-time code reviewers, developers can become orchestrators—setting the tasks, defining the boundaries, and letting deterministic verification handle the proof. If you are incorporating autonomous agents into your workflow, the most important question to ask is not "What model is the agent using?" but rather, "How will I execute and verify the result?" ### Practical Next Action To start shifting toward execution-backed verification, pick one critical component of your application—such as an authentication flow or a core data transformation function. Write a deterministic, local test script (using a framework like Vitest or Playwright) that exercises this component from end to end. The next time you use a coding agent to modify that component, do not review the diff first. Run the script. Let the execution evidence tell you whether the agent succeeded. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # AI code validation: requirements, tests, and evidence > AI code validation, verification, and review are three different jobs. Learn what validating agent-generated code actually requires and how to run a… Canonical: https://codevetter.com/ai-code-validation ## Try an AI code validation check Requirement: preserve an explicit `false` setting, the default, and the existing label. A truthiness fallback (`input.enabled || true`) breaks that requirement; a nullish fallback (`input.enabled ?? true`) preserves it. The published synthetic task repeats the intended baseline failure twice and the known-good pass twice. [Inspect the executed qualification receipt](https://codevetter.com/examples/explicit-false-qualification.json), then [run the same failure-and-fix check locally](https://codevetter.com/verify-ai-generated-code#worked-example). The receipt qualifies this task and its checks; it does not measure an agent's success rate. "AI code validation" is one of those search terms that covers three different jobs, and the method differs for each: **review** asks whether a change looks risky, **validation** asks whether it satisfies the stated requirements, and **verification** asks whether executed evidence proves the required behavior. Most tools and most searches conflate all three. ## The three jobs, separated | Question | What it checks | What evidence it produces | | --- | --- | --- | | AI code review | Does the diff contain suspicious or risky code? | Findings and comments on the patch | | AI code validation | Does the change satisfy what was requested? | A pass/fail against declared acceptance criteria | | Coding-agent verification | Did the required behavior actually work when exercised? | Executed checks, artifacts, and a reproducible verdict | Validation and verification overlap but are not interchangeable: a patch can satisfy its acceptance criteria on paper and still fail at runtime. The strongest answer comes from binding the requirement to a check that exercises it. ## What validating agent-generated code requires 1. **A frozen statement of intent.** The task the agent was given, verbatim — not a paraphrase reconstructed after the diff exists. 2. **Declared acceptance criteria.** "The endpoint returns 404 for deleted records" is checkable; "make deletion work" is not. If the criteria are ambiguous, record the ambiguity rather than letting the validator pick a generous reading. 3. **Checks that exercise behavior.** A review comment that a query "looks right" is not validation of the query. Running the endpoint, driving the browser flow, or executing the repository's own tests is. 4. **Failure classification.** A red check can mean an agent regression, a pre-existing failure, an environment problem, or an unverified requirement. Collapsing them into "failed" loses exactly the information validation exists to produce. ## Why a second model does not validate the first The most common "AI code validation" workflow is asking another LLM to read the diff. That is review, not validation — the second model reasons about plausibility, not execution. It cannot confirm that the migration applies, the route responds, or the permission check denies what it should. A model reading code is a useful triage step; it is not evidence that the software works. ## A practical validation pass For an agent-produced change, run this sequence before trusting it: 1. Write down the task and the exact revision being validated. 2. Convert each acceptance criterion into the smallest check that exercises it — prefer repository-owned tests and commands. 3. Run the checks in a bounded environment and record the command, exit state, and output. 4. Classify each failure: regression, pre-existing, environment, or unverified. 5. Re-run after the fix; keep both the failure and the later pass so the record shows closure. ## Where CodeVetter fits CodeVetter is built around the verification end of this spectrum: it binds a task and an exact agent change to executed checks and emits a portable evidence bundle with a pass, fail, or unverified verdict. Its public benchmark currently demonstrates a narrower recognition task — 27 synthetic cases with 29 labeled findings — and publishes its own limitations rather than claiming production-wide coverage. ## What to inspect next For the step-by-step workflow read the verification guide; for the conceptual boundary between review and verification see the comparison essay; for evidence shape see the bundle format; for measured results see the public benchmark. **CTA:** Validate your next agent change with executable checks, or inspect how CodeVetter structures its evidence bundle. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Anime List optimization case study > Anime List: A two-file import-boundary change materially reduced one local Vite navigation, while a separate production build showed almost no… Canonical: https://codevetter.com/benchmark/optimization/anime-list A two-file import-boundary change materially reduced one local Vite navigation, while a separate production build showed almost no shipped-bundle movement. ## Result - Decision: retained - Observation date: 2026-08-14 - Primary metric: Local response bytes — 35.8% fewer - Measurement note: 11,871,129 → 7,616,611 bytes - Availability: Historical research receipt. The experimental browser harness is not part of the compact current release. ## Tested boundary - Flow: One mobile home-page navigation in an isolated local Vite checkout - Source: components/Navigation.tsx, components/ui/dropdown-menu.tsx - Revision: db1faf0e0c27996c3eacfb9f1e3a4c0a65958938 - Correctness: One exact repository-owned Vitest test and the selected Playwright flow passed. - Patch cost: 2 files, 62 gross changed lines, net 58, 0 production dependencies ## Decision path - **Radix package subpaths (rejected):** 9.3% fewer bytes did not clear the 10% materiality floor. - **Lucide removed from the initial flow boundary (retained):** 10 samples per side plus warmups confirmed the local transfer reduction. - **Local developer-flow result only (qualified):** The production gzip change was 0.3%, so no consumer speed claim is made. ## Limitations - The keep proves one local config-disabled Vite development navigation, not production traffic, representative devices, CDN behavior, or end-user impact. - The correctness scope is one exact repository-owned recommendation test plus the selected Playwright flow; it is not the full Anime List suite. - The production build check shows only a 0.313 percent gzip reduction, so this candidate should be presented as local developer-flow optimization rather than a consumer performance case study. - The host agent still invents and applies source patches; CodeVetter owns evidence, boundaries, verification, decisions, and replanning. - No production, cloud, registry, package installation, migration, deployment, paid-model, or secret operation was used. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Free AI optimization case study > Free AI: A one-pass model-selection loop removed an intermediate array and improved every tested registry size without growing the patch. Canonical: https://codevetter.com/benchmark/optimization/free-ai A one-pass model-selection loop removed an intermediate array and improved every tested registry size without growing the patch. ## Result - Decision: confirmed - Observation date: 2026-08-14 - Primary metric: Largest tested registry — 29.8% faster - Measurement note: 0.016568 → 0.011624 ms/op at 79 models - Availability: Uses the current local Vitest performance path. ## Tested boundary - Flow: test/select-model-performance.spec.ts — model selection scales across the supported registry size - Source: src/router/select-model.ts:171 - Revision: 23ed8566bb1b1d8d86ee1f9890aa4ad5bba9cd18 - Correctness: 241 tests in 37 files passed, plus typecheck and changed-file lint. - Patch cost: 1 file, 45 gross changed lines, net -7, 0 production dependencies ## Decision path - **Allocation hotspot in selectCandidates (observed):** 84.1% of initial sampled allocation bytes pointed at the source. - **Build the ranked list directly (tested):** 10 interleaved samples per side covered three registry sizes. - **Faster with less code (confirmed):** The candidate removed seven net lines, added no dependency, and passed the bounded change-cost policy. ## Limitations - This proves one synthetic local model-registry workload up to the repository's current 79-model registry; it does not establish production request latency or customer impact. - Sampled heap bytes include allocations that may later be collected and are not retained-heap measurements. - Peak RSS is sampled process-tree evidence and includes the runtime and test runner. - No production endpoint, provider API, cloud resource, package installation, deployment, paid model, or secret operation was used. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Starboard optimization case study > Starboard: A plausible incremental token scanner passed focused correctness but ran slower, so CodeVetter rejected it and restored the original code. Canonical: https://codevetter.com/benchmark/optimization/starboard A plausible incremental token scanner passed focused correctness but ran slower, so CodeVetter rejected it and restored the original code. ## Result - Decision: rejected - Observation date: 2026-08-14 - Primary metric: Largest tested catalog — 5.3% slower - Measurement note: 47.496 → 50.026 ms/op - Availability: Uses the current local Vitest performance path. ## Tested boundary - Flow: src/__tests__/project-recommendations-performance.test.ts — project recommendations scale across local catalog sizes - Source: src/lib/project-recommendations.ts:101 - Revision: 7af23ce31ac10c7c552713f92985934de116abee - Correctness: 13 focused tests passed before performance rejected the candidate. - Patch cost: 1 file, 19 gross changed lines, net 5, 0 production dependencies ## Decision path - **Tokenization appeared in the CPU profile (observed):** 10.8% of CPU samples pointed at meaningfulTokens. - **Incremental regular-expression scan (tested):** The one-file experiment stayed within the change-cost budget and passed focused correctness. - **Slower and too noisy to trust (rejected):** The candidate was slower, sample spread was high, allocation evidence was incomplete, and the source was restored. ## Limitations - Wall-time sample spreads were 86.332% and 91.255%, so host or startup noise may dominate. - The candidate was slower at the largest input and heap evidence was incomplete; no optimization claim is made. - No production, cloud, deployment, package installation, paid model, or secret operation was used. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Optimization benchmarks and case studies > Dated CodeVetter optimization trials with tested flows, repeated measurements, correctness boundaries, patch cost, rejected candidates, and proof JSON. Canonical: https://codevetter.com/benchmark/optimization These dated local trials have separate evidence boundaries rather than one blended headline. Each case study names the exact flow, revision, source candidate, patch cost, correctness scope, repeated measurement, decision, availability, and limitations. - [Anime List](https://codevetter.com/benchmark/optimization/anime-list) — retained: A two-file import-boundary change materially reduced one local Vite navigation, while a separate production build showed almost no shipped-bundle movement. - [Free AI](https://codevetter.com/benchmark/optimization/free-ai) — confirmed: A one-pass model-selection loop removed an intermediate array and improved every tested registry size without growing the patch. - [Starboard](https://codevetter.com/benchmark/optimization/starboard) — rejected: A plausible incremental token scanner passed focused correctness but ran slower, so CodeVetter rejected it and restored the original code. A rejected experiment remains visible. Historical research capability is labeled separately from the compact runtime path currently shipped in CodeVetter. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeVetter vs open-source agent-verification CLIs > Compare CodeVetter with open-source agent-verification CLIs like Verdict, PatchDrill, ProofGate, evigate, didrun, and verity — quoted from their own READMEs. Canonical: https://codevetter.com/codevetter-vs-verification-clis An open-source wave of agent-verification CLIs now sits between code review and CI: Verdict, PatchDrill, ProofGate, evigate, didrun, and verity each check one narrow, deterministic claim about agent-written code. CodeVetter is a local verification workbench that reviews an agent change, exercises the changed behavior, and packages the evidence into a portable bundle. This page compares the shapes, quoting each project's own README, checked 2026-09-18. ## The short comparison | Question | CodeVetter | The verification CLIs | | --- | --- | --- | | Primary job | Verify whether an agent task worked and retain evidence | One deterministic gate per tool: intent-vs-diff, proof requirements, or claims-vs-events | | Main surface | Local desktop viewer plus CLI/MCP and portable evidence bundles | CLI or GitHub Action, usually wired into CI or a hook | | Verdict basis | Executed checks against changed behavior, findings, and uncertainty | Structural or recorded facts — never a model's judgement of the diff | | Platform | Apple-silicon macOS | Platform-independent | | Where code goes | Local machine; a chosen model provider only for review findings | Local or CI; most make no model call at all | | Current CodeVetter availability, checked August 2026 | Apple-silicon macOS build through GitHub Releases; no CodeVetter subscription is currently declared, and model-provider costs are separate | Open source | ## What each CLI actually checks From each repository's own README, checked 2026-09-18: - **Verdict** (`github.com/verdict-ci/verdict`) — "the merge-time fidelity gate for AI-written code." It reads a pull request's declared intent — the issue, PR description, or linked spec — and compares it against the actual diff, catching intent-mismatch, silent-scope, unbacked-claim, doc-drift, and missing-impl. It runs as a GitHub Action on every PR and as a CLI. - **PatchDrill** (`github.com/seungdori/patchdrill`) — "the deterministic proof layer between code review and CI." It answers "what proof should exist for THIS diff before merge — and what's missing?" with no model call and no network. Its detectors cover leaked secrets, prompt injection planted in files an agent will read, workflow escalation, missing proof, and dependency drift across roughly 25 ecosystems. - **ProofGate** (`github.com/aevryone/proofgate`) — a "trust-but-verify merge gate" that blocks only on structural or executed facts: committed secrets, empty or always-true tests, security functions stubbed to a constant, imports that resolve nowhere. Claims inferred from PR prose are warnings, never blockers. - **evigate** (`github.com/shiki-yusuke/evigate`) — a local CLI that parses a session transcript into tool-observed events and agent-declared claims, then returns proven, contradicted, or unknown verdicts. A claim is proven only when the recorded evidence supports it; ambiguous evidence is reported as unknown rather than guessed. - **didrun** (`github.com/nelsonwerd/didrun`) — "flight records for agent-written code." It records what an agent actually executed — the command, the exit code, the output, and the exact git tree state — binds claims to a commit, and exports a self-contained HTML report. Its README is precise about scope: "didrun is a recording, not a proof." There is no LLM in the trust path. - **verity** (`github.com/pietro-falco/verity`) — checks declared claims in a manifest against the filesystem, git HEAD, and command exit codes, stamping each true or false deterministically and offline. It explicitly "checks truth, not quality." ## Where they overlap with CodeVetter All of these projects, CodeVetter included, reject the same premise: that a second model reading the diff counts as verification. They all favor deterministic, inspectable evidence and local or CI execution over a hosted reviewer. ## Where they differ The CLIs are deliberately narrow. Each answers one question well — did the diff do what it claimed (Verdict), does the required proof exist (PatchDrill), are there structural merge blockers (ProofGate), did the session's claims match its events (evigate), what did the agent actually run (didrun), is this declared claim true (verity). None of them tries to answer the broader task question. CodeVetter aims at that broader question: did the agent complete the requested software task correctly? Its verification loop binds the task and exact change to executable checks — repository tests plus focused browser or API checks — and separates pass, fail, and unverified rather than emitting a single gate verdict. The output is an evidence bundle (JSON, Markdown, or offline HTML) that a reviewer can inspect or replay locally. ## When a verification CLI is the better choice Plainly: often, today. - **You want a merge gate in CI.** These tools were designed to block merges; CodeVetter is a local workbench, not a branch-protection check. - **You need one narrow guarantee.** If the question is exactly "did the agent's report match what it executed," evigate or didrun answers it with less machinery. - **You are not on a Mac.** CodeVetter publishes an Apple-silicon macOS build only; every tool on this page runs anywhere. - **You want zero dependencies and auditable code.** Several of these CLIs are small enough to read in an afternoon. ## When CodeVetter is the better choice - **The question is task completion, not claim fidelity.** A diff can faithfully implement its PR description and still break the requested behavior; a claims-checker will not see that. - **You need evidence for a human reviewer.** CodeVetter emits a portable bundle with checks, artifacts, and explicit uncertainty rather than a single exit code. - **You want the whole review loop local.** Findings, re-checks, and history stay on your machine; your model key is used only for review findings. ## How this comparison was built **Sources.** Each tool's public README at `github.com/verdict-ci/verdict`, `github.com/seungdori/patchdrill`, `github.com/aevryone/proofgate`, `github.com/shiki-yusuke/evigate`, `github.com/nelsonwerd/didrun`, and `github.com/pietro-falco/verity` — all checked 2026-09-18. CodeVetter claims come from this repository and `/privacy`. **What we did not test.** We did not run any of these tools against a shared case set. No catch rate, latency, or cost figure appears on this page because none was measured. These projects are young and change quickly; re-check their READMEs before relying on a specific capability. **What a real head-to-head needs.** The same frozen agent tasks, disclosed configuration, repeated runs, and one scorer applied to every tool — the same standard CodeVetter applies to itself at `/benchmark`. ## Where to go next - [Review vs. verification](https://codevetter.com/ai-code-review-vs-verification) — the concepts, no vendor - [CodeVetter vs. CodeRabbit](https://codevetter.com/codevetter-vs-coderabbit) and [vs. Greptile](https://codevetter.com/codevetter-vs-greptile) - [How verification works](https://codevetter.com/coding-agent-verification) - [The public benchmark](https://codevetter.com/benchmark) - [Download CodeVetter](https://codevetter.com/download) ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Contact CodeVetter > How to reach the CodeVetter team: GitHub issues, email, and social. Execution-backed verification for coding-agent changes, with local checks, portable… Canonical: https://codevetter.com/contact CodeVetter is open-source and local-first. The fastest way to get help or report a bug is GitHub Issues. ## GitHub - github.com/Codevetter/codevetter/issues — bug reports, feature requests, and verification case contributions - github.com/Codevetter/codevetter — source code and releases ## Email sarthakagrawal@agentmail.to ## Social - @sarthakcodes on X - sarthakagrawal.dev — personal site ## Agent email For agent-to-agent or directory verification, use sarthakagrawal@agentmail.to. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Aider — tools that shaped CodeVetter > Aider: A small, ranked repository map gives an agent useful context without dumping the whole tree. Read what CodeVetter learned and where our own… Canonical: https://codevetter.com/inspiration/aider A small, ranked repository map gives an agent useful context without dumping the whole tree. ## What we admire Aider makes repository context a deliberate budget. Its map uses syntax and relationships to select definitions that are likely to help with the current task, then leaves room for the agent to read the real files. ## The principle that stayed The valuable unit is a navigable lead with an explicit source, rather than a large prompt that merely looks comprehensive. Ranking matters because every extra token competes with evidence from the actual change. ## Where CodeVetter stands Our repository graph and Repo Unpack packets pursue bounded, source-linked navigation. We did not port Aider’s PageRank implementation or make a repository map the verdict. The verifier still has to execute checks and attach run evidence. ## Sources - [Aider repository map](https://aider.chat/docs/repomap.html) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Arbor — tools that shaped CodeVetter > Arbor: Deterministic graph traversal is a useful answer to structural questions. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/arbor Deterministic graph traversal is a useful answer to structural questions. ## What we admire Arbor’s graph-native approach values exact symbol relationships over a vague semantic match. That is a principled way to answer questions such as “what calls this function?” and “what might this change affect?” ## The principle that stayed A graph edge should carry a source location and a defined extraction rule. Determinism makes a result repeatable, although a parser graph still cannot prove the program’s runtime behavior. ## Where CodeVetter stands CodeVetter owns a bundled Tree-sitter structural graph and marks extracted links as evidence leads. Arbor’s philosophy informed our evaluation, but its engine is not bundled. ## Sources - [Arbor source repository](https://github.com/getArbor-dev/arbor) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # ast-grep — tools that shaped CodeVetter > ast-grep: Structural search can express focused review rules more clearly than text matching. Read what CodeVetter learned and where our own implementation… Canonical: https://codevetter.com/inspiration/ast-grep Structural search can express focused review rules more clearly than text matching. ## What we admire ast-grep makes syntax-aware search available as a straightforward CLI. It is well suited to small, inspectable rules over changed code rather than an opaque, sweeping analyzer. ## The principle that stayed A rule match is a lead with a source location. An absent match does not mean the patch is safe; the rule may not cover the language, pattern, or behavior that matters. ## Where CodeVetter stands CodeVetter can use an installed `sg` as an optional collector for narrow changed-file evidence. It falls back cleanly when absent and does not require ast-grep for the canonical graph. ## Sources - [ast-grep CLI reference](https://ast-grep.github.io/reference/cli) — the creator's own source - [CodeVetter OSS integration decision](https://github.com/Codevetter/codevetter/blob/main/docs/architecture/decisions/oss-integration.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Augment Code — tools that shaped CodeVetter > Augment Code: Fresh repository context should follow the developer across tools. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/augment-code Fresh repository context should follow the developer across tools. ## What we admire Augment’s Context Engine emphasizes a continuously updated understanding of code, dependencies, and history. The ambition is compelling because stale context can make an otherwise capable agent confidently wrong. ## The principle that stayed Freshness is part of correctness. A context packet should identify the revision it describes and say when indexing is incomplete or out of date. ## Where CodeVetter stands Our graph and history receipts report indexed revisions and stale states. Augment was a context-engine research reference; CodeVetter does not use its hosted index. ## Sources - [Augment Context Engine](https://www.augmentcode.com/context-engine) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Bito — tools that shaped CodeVetter > Bito: Repository architecture and external project context can change the meaning of a diff. Read what CodeVetter learned and where our own implementation… Canonical: https://codevetter.com/inspiration/bito Repository architecture and external project context can change the meaning of a diff. ## What we admire Bito’s AI Architect and review tooling treat the codebase as more than a set of changed files. The connection between architecture, history, and review was a useful research reference for our own context work. ## The principle that stayed Context sources should remain attributable. A ticket, a graph edge, and a test failure carry different kinds of authority, and mixing them without provenance makes a verdict harder to trust. ## Where CodeVetter stands CodeVetter records source-linked graph and history leads and keeps runtime checks distinct. Bito is part of our landscape research; there is no Bito dependency or service connection. ## Sources - [Bito review overview](https://docs.bito.ai/ai-code-reviews-in-git/overview) — the creator's own source - [CodeVetter competitive landscape](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/competitive-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Claude Code — tools that shaped CodeVetter > Claude Code: An agent can discover context through targeted search and reads. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/claude-code An agent can discover context through targeted search and reads. ## What we admire Claude Code’s tool-driven workflow helped normalize a coding agent that explores the repository, edits files, and runs checks in one loop. Its review workflow also provides a clear comparison for AI-first review. ## The principle that stayed The path from task to change is observable, but an agent’s own confidence is not a verdict. Independent, executable checks and explicit coverage are needed before calling the task complete. ## Where CodeVetter stands CodeVetter can work with an installed Claude CLI and can evaluate its changes. It stores per-run evidence and does not treat a Claude review as independent proof of correctness. ## Sources - [Claude Code overview](https://code.claude.com/docs/en/overview) — the creator's own source - [CodeVetter project log](https://github.com/Codevetter/codevetter/blob/main/docs/archive/project-log.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Cline — tools that shaped CodeVetter > Cline: Planning and acting benefit from a visible boundary. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/cline Planning and acting benefit from a visible boundary. ## What we admire Cline’s Plan and Act workflow makes a consequential transition clear: first understand the task, then change code and run tools. That is a helpful reference for communicating what an agent is doing. ## The principle that stayed A plan is useful evidence of intent, but it does not verify the implementation. The final check still has to inspect what changed and what actually ran. ## Where CodeVetter stands CodeVetter records the task and exact agent change before executable verification. Cline is one of the coding-agent workflows we studied, not a bundled runtime. ## Sources - [Cline Plan and Act](https://github.com/cline/cline/blob/main/docs/core-workflows/plan-and-act.mdx) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeQL — tools that shaped CodeVetter > CodeQL: Deep code queries can reveal data flow and security relationships. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/codeql Deep code queries can reveal data flow and security relationships. ## What we admire CodeQL is a strong example of asking precise questions over a structured representation of code. Its query libraries show how much depth a specialized security analyzer can provide. ## The principle that stayed Extraction and query scope are part of the result. A deep analyzer is valuable when its language support, database state, and findings can be connected to the exact change under review. ## Where CodeVetter stands We evaluated CodeQL and parked a direct integration for a security-focused need. CodeVetter’s local graph is narrower and its verdict remains grounded in the checks actually run. ## Sources - [CodeQL documentation](https://codeql.github.com/docs/) — the creator's own source - [CodeVetter OSS integration decision](https://github.com/Codevetter/codevetter/blob/main/docs/architecture/decisions/oss-integration.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # CodeRabbit — tools that shaped CodeVetter > CodeRabbit: Review is more useful when it arrives in the developer’s normal change workflow. Read what CodeVetter learned and where our own implementation… Canonical: https://codevetter.com/inspiration/coderabbit Review is more useful when it arrives in the developer’s normal change workflow. ## What we admire CodeRabbit helped make automated, context-aware PR feedback a familiar part of the review cycle. Its summaries, inline comments, and developer-facing explanations show the value of giving findings a clear place to be acted on. ## The principle that stayed Delivery and presentation matter, but a review comment is still a hypothesis until someone can reproduce the underlying problem. Review volume alone is not a quality metric. ## Where CodeVetter stands We studied CodeRabbit as a baseline for agent-written PRs. CodeVetter keeps its verification local and records reproduced, unchecked, and failed states explicitly; the public synthetic corpus does not prove superiority on production PRs. ## Sources - [CodeRabbit documentation](https://docs.coderabbit.ai/) — the creator's own source - [CodeVetter competitive landscape](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/competitive-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # codesize — tools that shaped CodeVetter > codesize: A tiny heuristic can still point a reviewer toward risky code. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/codesize A tiny heuristic can still point a reviewer toward risky code. ## What we admire codesize was interesting because it suggests function size as one inexpensive maintainability and review signal. Small, transparent heuristics can be useful when they are clearly labeled as leads. ## The principle that stayed Function size cannot predict a defect by itself. A signal should be calibrated against real outcomes and should never be presented as a proof of failure. ## Where CodeVetter stands Our integration review decided not to depend on codesize. Repo Unpack’s health inventory remains explicitly heuristic and keeps defect, maintainability, and performance leads separate. ## Sources - [codesize source repository](https://github.com/ChrisGVE/codesize) — the creator's own source - [CodeVetter OSS integration decision](https://github.com/Codevetter/codevetter/blob/main/docs/architecture/decisions/oss-integration.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # OpenAI Codex — tools that shaped CodeVetter > OpenAI Codex: An agent’s patch and its execution trace should be available for inspection. Read what CodeVetter learned and where our own implementation… Canonical: https://codevetter.com/inspiration/codex An agent’s patch and its execution trace should be available for inspection. ## What we admire Codex’s task-oriented coding workflow makes it natural to delegate a change and inspect the resulting diff, commands, and tests. That creates the exact handoff CodeVetter was built to verify. ## The principle that stayed Even when an agent reports successful tests, the verifier should bind those observations to the precise revision and test scope. A changed branch or missing environment changes what the claim means. ## Where CodeVetter stands CodeVetter supports local Codex sessions as one input to verification. It is independent software and does not use Codex output itself as the final judge. ## Sources - [OpenAI Codex](https://developers.openai.com/learn/codex) — the creator's own source - [CodeVetter project log](https://github.com/Codevetter/codevetter/blob/main/docs/archive/project-log.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Continue — tools that shaped CodeVetter > Continue: The developer should be able to choose models and context sources. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/continue The developer should be able to choose models and context sources. ## What we admire Continue’s context-provider model makes the source of an AI assistant’s context more explicit. The openness to different models and local setups is a useful reference for a product that should not depend on one provider. ## The principle that stayed Provider choice is strongest when the evidence contract stays stable. Swapping a model should not change the meaning of a verification receipt or quietly change the rules for a verdict. ## Where CodeVetter stands CodeVetter supports user-configured providers while Rust-owned schemas keep CLI, MCP, and viewer aligned. Continue was a design study, not an integration. ## Sources - [Continue context providers](https://docs.continue.dev/customize/deep-dives/custom-providers) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Cursor — tools that shaped CodeVetter > Cursor: Codebase navigation can feel immediate inside a coding workflow. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/cursor Codebase navigation can feel immediate inside a coding workflow. ## What we admire Cursor made repository-aware editing feel accessible to many developers. Its ability to move from a question to relevant code is a useful interaction reference even for a separate verification product. ## The principle that stayed Fast retrieval is valuable when it leads to inspectable files and a small next action. It should never let an inferred summary quietly become a claim about runtime behavior. ## Where CodeVetter stands CodeVetter’s local graph and repo-query tools favor attributable source leads. It is a verifier alongside coding tools, not a Cursor extension or IDE replacement. ## Sources - [Cursor documentation](https://cursor.com/docs) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # DeepWiki — tools that shaped CodeVetter > DeepWiki: A repository can be made legible quickly through structured explanation. Read what CodeVetter learned and where our own implementation begins and… Canonical: https://codevetter.com/inspiration/deepwiki A repository can be made legible quickly through structured explanation. ## What we admire DeepWiki turned public repository exploration into a low-friction experience: enter a repo and get navigable explanations and diagrams. That feeling of a codebase becoming understandable was a strong reference for Repo Unpack. ## The principle that stayed Generated explanations are useful entry points, especially for unfamiliar repos. They need direct links back to files and a clear freshness boundary before someone relies on them for a technical decision. ## Where CodeVetter stands Repo Unpack publishes source-linked, pinned-commit artifacts and labels inferences. CodeVetter does not claim that generated prose proves behavior, and it does not depend on DeepWiki. ## Sources - [Cognition introduces DeepWiki](https://cognition.com/blog/deepwiki) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # DeepWork — tools that shaped CodeVetter > DeepWork: Long agent tasks need explicit steps, artifacts, and quality gates. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/deepwork Long agent tasks need explicit steps, artifacts, and quality gates. ## What we admire DeepWork turns a loosely worded task into a procedure with required steps and checks. That is a useful model for work that cannot be trusted merely because an agent produced a polished final answer. ## The principle that stayed A procedure should say what each step consumes, what it produces, and what makes it pass. When a gate fails, the handoff should preserve the failure and the remaining questions. ## Where CodeVetter stands CodeVetter’s Evidence Pattern Search design names DeepWork-style procedures for review, fix, QA, and handoff. That reference is about workflow structure; DeepWork is not a required runtime or dependency. ## Sources - [DeepWork source repository](https://github.com/Unsupervisedcom/deepwork) — the creator's own source - [CodeVetter evidence-pattern design](https://github.com/Codevetter/codevetter/blob/main/docs/archive/PRD-EVIDENCE-PATTERN-SEARCH.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Ellipsis — tools that shaped CodeVetter > Ellipsis: The useful review loop continues from finding to attempted fix. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/ellipsis The useful review loop continues from finding to attempted fix. ## What we admire Ellipsis connects review feedback with an agent that can make a change and run checks. That closes an important gap between a comment that identifies a problem and an actual repair that a developer can inspect. ## The principle that stayed A generated fix needs its own verification. The fact that an agent acted on a comment does not establish that the behavior is repaired or that the patch stayed within scope. ## Where CodeVetter stands CodeVetter’s agent-fix packets and review-proof states make the proposed repair reviewable and distinguish fixed, reproduced, and unchecked outcomes. Ellipsis was a workflow reference, not an integration. ## Sources - [Ellipsis code review documentation](https://www.ellipsis.dev/docs/code-review) — the creator's own source - [CodeVetter competitive landscape](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/competitive-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Gemini CLI — tools that shaped CodeVetter > Gemini CLI: A terminal agent can keep planning, tools, and code changes in one inspectable loop. Read what CodeVetter learned and where our own… Canonical: https://codevetter.com/inspiration/gemini-cli A terminal agent can keep planning, tools, and code changes in one inspectable loop. ## What we admire Gemini CLI’s open terminal workflow was part of our survey of agent tools. It illustrates how a developer can combine a large-context model with local search and controlled tool use. ## The principle that stayed A context window is not a substitute for proving the changed behavior. The verifier needs explicit tests, exact source state, and the limitations of the run. ## Where CodeVetter stands CodeVetter can verify changes made by terminal agents without adopting their internal planning model. Gemini CLI was studied as an agent workflow, not made a required companion. ## Sources - [Gemini CLI source repository](https://github.com/google-gemini/gemini-cli) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # GitHub Copilot code review — tools that shaped CodeVetter > GitHub Copilot code review: A code host can make automated review a normal request on a pull request. Read what CodeVetter learned and where our own… Canonical: https://codevetter.com/inspiration/github-copilot A code host can make automated review a normal request on a pull request. ## What we admire Copilot code review is a useful mainstream baseline: a developer can ask for review in the same place they inspect a PR and act on suggestions without learning a separate system. ## The principle that stayed Convenience makes review more likely to happen. It does not establish whether a suggestion describes a reproducible failure, so benchmark comparisons must count real findings and false positives. ## Where CodeVetter stands CodeVetter’s benchmark and verification bundle make evidence and scoring inspectable outside a code host. We have not claimed a production catch-rate win against Copilot code review. ## Sources - [GitHub Copilot code review](https://docs.github.com/en/copilot/concepts/agents/code-review) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # GitNexus — tools that shaped CodeVetter > GitNexus: A code graph can be exposed as a useful interface for agents. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/gitnexus A code graph can be exposed as a useful interface for agents. ## What we admire GitNexus explores code knowledge graphs that an agent can query for relationships and flows. It was a useful comparison while we decided how much of our graph should be an owned local engine. ## The principle that stayed The graph’s runtime and parser versions affect reproducibility. An optional external adapter can be useful, but the canonical evidence path should remain available offline. ## Where CodeVetter stands The graph decision keeps CodeVetter’s Tree-sitter index in Rust and treats a GitNexus subprocess as optional secondary research. It is not required by the product. ## Sources - [GitNexus source repository](https://github.com/abhigyanpatwari/GitNexus) — the creator's own source - [CodeVetter OSS integration decision](https://github.com/Codevetter/codevetter/blob/main/docs/architecture/decisions/oss-integration.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # gitoxide — tools that shaped CodeVetter > gitoxide: Git history can be explored through an owned Rust interface. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/gitoxide Git history can be explored through an owned Rust interface. ## What we admire gitoxide is an ambitious Rust implementation of Git. Its library approach was worth considering for local history traversal and diff processing, where process boundaries and correctness both matter. ## The principle that stayed An integration has to improve a measured bottleneck or capability. A cleaner dependency graph on paper does not justify replacing a working Git CLI path without parity evidence. ## Where CodeVetter stands Our OSS review kept gitoxide on a watchlist. CodeVetter continues to use its current Git path for history evidence; gitoxide is not a production dependency. ## Sources - [gitoxide source repository](https://github.com/GitoxideLabs/gitoxide) — the creator's own source - [CodeVetter OSS integration decision](https://github.com/Codevetter/codevetter/blob/main/docs/architecture/decisions/oss-integration.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) --- # Google Code Wiki — tools that shaped CodeVetter > Google Code Wiki: Repository explanations should be refreshed as code changes. Read what CodeVetter learned and where our own implementation begins and ends. Canonical: https://codevetter.com/inspiration/google-code-wiki Repository explanations should be refreshed as code changes. ## What we admire Google Code Wiki’s continuously updated wiki is a useful counterpoint to one-time documentation generation. It treats freshness as part of the reader’s experience, not an implementation detail. ## The principle that stayed A wiki should say which version of the repository it describes. Regeneration alone is not enough if readers cannot trace an important statement to source or distinguish an inferred relationship. ## Where CodeVetter stands Repo Unpack artifacts carry a scanned commit and evidence links. We researched Code Wiki as a documentation pattern; CodeVetter does not generate a continuously hosted wiki. ## Sources - [Google introduces Code Wiki](https://developers.googleblog.com/introducing-code-wiki-accelerating-your-code-understanding/) — the creator's own source - [CodeVetter codebase-context research](https://github.com/Codevetter/codevetter/blob/main/docs/knowledge/codebase-context-tools-landscape.md) — the CodeVetter project record This is independent appreciation, not endorsement, partnership, code reuse, or feature parity. ## Public product links - [CodeVetter](https://codevetter.com/) - [Download](https://codevetter.com/download) - [Documentation](https://codevetter.com/docs/) - [Source](https://github.com/Codevetter/codevetter) Included 50 of 1834 registered public pages (limit 50 pages / 500000 characters). Full catalog: https://codevetter.com/api/md/_catalog