CodeVetter · Repo Unpack · Testing signals
google-gemini/gemini-cli
Testing is heavy and layered: Vitest 3.2 with colocated *.test.ts(x) unit tests across every package (a large share of scanned TS files are tests), an integration-tests/ workspace running the real built binary (optionally under docker/podman sandbox) with recorded .responses fixtures, evals/ behavioral evals gated behind RUN_EVALS with an llm-judge, plus perf-tests/ and memory-tests/ with baseline updating. CI (ci.yml) enforces lint (eslint+actionlint+shellcheck+yamllint+prettier+sensitive-keywords+action pinning), build, typecheck, per-workspace test:ci, bundle smoke (`node ./bundle/gemini.js --version`), generated-settings-doc freshness, and lockfile checks. Husky+lint-staged run prettier/eslint on commit.
- Unit tests are colocated Vitest files next to sources (e.g. client.test.ts, scheduler.test.ts, extension-manager*.test.ts, App.test.tsx); vitest.config.ts per package; `npm run test` runs --workspaces plus a sea/sea-launch.test.js binary-launch check, and posttest runs a full build.
- integration-tests/ is a separate vitest root (52+ *.test.ts files) with testTimeout 300000, retry 2, 8-16 threads, globalSetup.ts, GEMINI_TEST_TYPE=integration, and *.responses replay fixtures for browser-agent/hooks/context-compression flows; sandbox variants selected via GEMINI_SANDBOX=none|docker|podman.
- Fake/recorded model backends are first-class test hooks: --fake-responses/--fake-responses-non-strict load FakeContentGenerator from a file and --record-responses wraps the real generator in RecordingContentGenerator — this is how the .responses fixtures are produced and replayed.
- evals/ contains ~30 *.eval.ts behavioral evals (answer-vs-act, plan_mode, prompt_injection_mcp, memory_persistence, model_steering, sandbox_recovery, ...) driven by an llm-judge.ts; only 'always_passing' run by default, all evals need RUN_EVALS=1.
- perf-tests/ and memory-tests/ are dedicated vitest roots with baseline files updated via UPDATE_PERF_BASELINES/UPDATE_MEMORY_BASELINES env flags, plus memory-nightly.yml and perf-nightly.yml scheduled workflows.
- CI runs a lint battery beyond eslint: scripts/lint.js --setup/--eslint/--actionlint/--shellcheck/--yamllint/--prettier/--sensitive-keywords/--check-github-actions-pinning, then build, typecheck, settings-doc regeneration check (docs:settings --check), and lockfile integrity.
- CI test matrix splits jobs by workspace: `npm run test:ci --workspace @google/gemini-cli` on one runner and core+a2a-server+vscode-ide-companion+test-utils (with coverage disabled) on another, followed by bundle verification `node ./bundle/gemini.js --version`.
- A deflake harness exists (scripts/deflake.js + deflake.yml workflow + RUN_FLAKY_INTEGRATION flag) specifically to rerun integration suites, signalling acknowledged flaky e2e coverage.
- Test infrastructure deps include msw (HTTP mocking), mock-fs/memfs (fs virtualization), react-devtools-core, and a dedicated @google/gemini-cli-test-utils workspace; ink is replaced with a fork (@jrichman/ink) via npm overrides to make TUI testable.
- eslint-plugin-headers enforces the Google Apache license header on source files, and lint-staged runs prettier+eslint --fix on commit via husky hooks.
- Coverage gaps visible from the file tree: packages/devtools client has few colocated tests, scripts/ has a separate vitest config, and several heavy paths (sandbox.ts 1431 lines, AppContainer.tsx 2869 lines) have limited direct unit coverage relative to size — integration tests carry much of that weight. (inferred)