CodeVetter · Repo Unpack · Testing signals
earendil-works/pi
Vitest is the primary framework (packages/coding-agent/test ~297 files, agent+ai ~240 combined), plus node:test for packages/tui and scripts/*.test.mjs. A faux provider (pi-ai providers/faux.ts) enables deterministic agent-loop testing without API keys, and test/suite/harness.ts builds full AgentSession fixtures. ./test.sh runs the whole suite in a scrubbed environment (temp HOME, no API keys) because e2e tests activate when real endpoint/auth env vars exist. CI runs build → check → npm test on ubuntu; supply-chain checks (pinned deps, shrinkwrap, lockfile gate) run in npm run check and the husky pre-commit hook.
- ./test.sh runs `env -i` npm test with a mktemp HOME, empty git config, disabled git prompts, PI_NO_LOCAL_LLM=1, and AWS_EC2_METADATA_DISABLED — deliberately stripping all credentials so tests never hit real providers; cleanup refuses to rm -rf unverified directories.
- AGENTS.md forbids running the raw vitest suite because it includes e2e tests that activate when endpoint/auth env vars are present; prescribed: ./test.sh, per-file vitest via node_modules/vitest CLI, or node --test for packages/tui.
- The coding-agent suite harness (test/suite/harness.ts) creates a real AgentSession + SessionManager + SettingsManager + AuthStorage in a temp dir and registers the faux provider (registerFauxProvider from pi-ai/compat) with scripted FauxResponseStep lists, so session behavior is tested end-to-end without network.
- packages/coding-agent/test contains ~297 test files covering session branching, compaction, retry, dynamic tools/providers, extensions, args parsing, auth storage, clipboard, keybindings, Windows quirks, plus a regressions/ subdir.
- packages/agent has three vitest configs: default unit tests, vitest.harness.config.ts for harness/session conformance tests (test/harness/memory-conformance.test.ts runs a conformance suite over session storage implementations), and vitest.benchmark.config.ts for session benchmarks.
- packages/tui tests use node:test (editor, keybindings, overlay, markdown, fuzzy, layout, plus regression files for CJK width, SIGWINCH, regional indicators) alongside bench scripts like alt-screen-large-transcript-bench.ts.
- scripts/*.test.mjs run via `node --test` under test:scripts and cover repo tooling: check-runtime-deps, coding-agent-consumer, publish-release-announcement, sync-versions.
- packages/evals is a separate vitest-evals harness: *.eval.ts host evals and *.docs.eval.ts documentation-lift comparisons run in docker containers, requiring PI_PROVIDER/PI_MODEL — it is opt-in and not part of npm test.
- CI (.github/workflows/ci.yml) is a single job: checkout → setup-node 22 → apt deps (cairo/pango/jpeg/gif/rsvg for native builds, fd, ripgrep) → npm ci --ignore-scripts → npm run build → npm run check → npm test, on ubuntu-latest only.
- npm run check is the gate: biome check --write --error-on-warnings, pinned-dep verification, runtime-dep check, TS relative-import check, entry-graph check, shrinkwrap --check, install-lock --check, tsgo --noEmit, and a browser smoke check.
- The husky pre-commit hook runs scripts/check-lockfile-commit.mjs (blocks lockfile commits unless PI_ALLOW_LOCKFILE_CHANGE=1), then full npm run check, then a browser smoke test when staged files touch packages/ai, packages/web-ui, or the lockfile.
- Coverage gaps: real provider streaming paths, OAuth flows, and end-to-end interactive TUI behavior are only exercised when env keys exist or via manual tmux testing documented in .pi/skills/interactive-testing.md — most CI runs exercise mocks. (inferred)
- A scheduled npm-audit workflow runs npm audit --omit=dev plus npm audit signatures, extending supply-chain checking beyond commit-time checks.