CodeVetter · Repo Unpack · Testing signals
github/spec-kit
Testing is pytest-based (~253 files under `tests/`) with coverage configured in pyproject. The suite mirrors the CLI command surface under `tests/specify_cli/` per `design/cli.md`, has one file per integration under `tests/integrations/`, parity tests proving bash↔python↔powershell script equivalence, contract tests for wheel bundling and catalog/manifest schemas, and workflow/overlay/engine suites. CI runs a 3-OS × 2-Python pytest matrix plus ruff, markdownlint, a dependency security audit, and CodeQL.
- pytest is configured in pyproject with `testpaths=["tests"]`, strict markers, and short tracebacks; the `test` extra adds `pytest` and `pytest-cov`, and `[tool.coverage]` scopes coverage to `src/`.
- Command tests mirror the source layout exactly — `src/specify_cli/extensions/command_add.py` ↔ `tests/specify_cli/extensions/test_command_add.py`, `catalog/command_add.py` ↔ `tests/specify_cli/extensions/catalog/test_command_add.py` — a convention mandated by `design/cli.md`.
- Every registered integration has a dedicated test file `tests/integrations/test_integration_<key>.py` (hyphens→underscores), plus base-class suites (`test_integration_base_markdown/toml/yaml/skills`) and registry-level tests (`test_base`, `test_cli`, `test_events`).
- tests/conftest.py provides the shared harness: `_has_working_bash()` probes for a real bash (rejecting the WSL launcher on Windows, override `SPECKIT_TEST_BASH=1`), autouse fixtures isolate `~/.specify/auth.json` and strip inherited `SPECIFY_*`/`SPECKIT_INTEGRATION_*` env vars, and argv0 fixtures fake uv-tool/pipx/uvx install locations for self-upgrade tests.
- Script parity is enforced by dedicated tests — `test_setup_plan_python_parity.py`, `test_create_new_feature_python_parity.py`, `test_resolve_template_python_parity.py`, `test_check_prerequisites_python_parity.py`, `test_setup_tasks_python_parity.py` — matching the AGENTS.md rule that all `scripts:` frontmatter entries must be behaviorally equivalent across sh/ps/py.
- Contract tests under `tests/contract/` pin the wheel's bundled payload and schemas: `test_wheel_core_pack_scripts.py`, `test_wheel_bundled_presets.py`, `test_wheel_bundled_workflows.py`, `test_catalog_schema.py`, `test_manifest_schema.py`, `test_firstparty_bundle_catalog_consistency.py`.
- Workflow coverage lives in `tests/workflows/` (bundled speckit/bugfix/assess workflow validation, overlay composer/merge/schema/security, slot step, resolver integration) plus top-level `test_workflows.py` and `test_workflow_run_without_project.py`.
- Security-focused suites exist for the risky paths: `test_download_security.py` (bounded reads/extraction), `test_registrar_path_traversal.py`, `test_command_add_path_traversal.py`, `test_security_workflow.py`, and `test_authentication.py`.
- CI: `.github/workflows/test.yml` runs ruff (`uvx ruff@0.15.0 check src tests`) then pytest on ubuntu/windows/macos × Python 3.13/3.14; `lint.yml` adds `git diff --check` and markdownlint-cli2; `security.yml` runs a dependency audit script; `codeql.yml` runs CodeQL.
- The suite is heavily fixture/monkeypatch-driven (tmp_path projects, faked argv0, stubbed catalogs) rather than spawning real agent CLIs; `_has_working_bash` auto-skips shell-dependent tests when bash is absent, so script-level coverage can silently thin out on under-provisioned runners. (inferred)
- Extension behavior has both legacy consolidated suites (`tests/test_extensions.py`, `tests/test_presets.py`) and newer per-command mirrored suites; `design/cli.md` explicitly allows the consolidated domain suites to remain until separately reorganized, so coverage is duplicated across both styles rather than missing.