# Testing signals — github/spec-kit

> Canonical page: https://codevetter.com/unpack/spec-kit/testing-signals

# Testing signals — github/spec-kit

Scanned commit 175adbcd826befdccce8c60faa07ae3d8bbc8d87 (2026-09-22). Generated by codevetter codevetter 1.12.0 on 2026-09-22.

Testing is pytest-based (~253 files under `tests/`) with coverage configured in pyproject. The suite mirrors the CLI command surface under `tests/specify_cli/` per `design/cli.md`, has one file per integration under `tests/integrations/`, parity tests proving bash↔python↔powershell script equivalence, contract tests for wheel bundling and catalog/manifest schemas, and workflow/overlay/engine suites. CI runs a 3-OS × 2-Python pytest matrix plus ruff, markdownlint, a dependency security audit, and CodeQL.
- pytest is configured in pyproject with `testpaths=["tests"]`, strict markers, and short tracebacks; the `test` extra adds `pytest` and `pytest-cov`, and `[tool.coverage]` scopes coverage to `src/`. — [pyproject.toml#L59-83](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/pyproject.toml#L59-L83)
- Command tests mirror the source layout exactly — `src/specify_cli/extensions/command_add.py` ↔ `tests/specify_cli/extensions/test_command_add.py`, `catalog/command_add.py` ↔ `tests/specify_cli/extensions/catalog/test_command_add.py` — a convention mandated by `design/cli.md`. — [design/cli.md#L186-232](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/design/cli.md#L186-L232) [tests/specify_cli/extensions/test_command_add.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/specify_cli/extensions/test_command_add.py) [tests/specify_cli/extensions/catalog/test_command_add.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/specify_cli/extensions/catalog/test_command_add.py)
- Every registered integration has a dedicated test file `tests/integrations/test_integration_<key>.py` (hyphens→underscores), plus base-class suites (`test_integration_base_markdown/toml/yaml/skills`) and registry-level tests (`test_base`, `test_cli`, `test_events`). — [tests/integrations/test_integration_claude.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/integrations/test_integration_claude.py) [tests/integrations/test_integration_base_markdown.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/integrations/test_integration_base_markdown.py) [AGENTS.md](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/AGENTS.md)
- tests/conftest.py provides the shared harness: `_has_working_bash()` probes for a real bash (rejecting the WSL launcher on Windows, override `SPECKIT_TEST_BASH=1`), autouse fixtures isolate `~/.specify/auth.json` and strip inherited `SPECIFY_*`/`SPECKIT_INTEGRATION_*` env vars, and argv0 fixtures fake uv-tool/pipx/uvx install locations for self-upgrade tests. — [tests/conftest.py#L18-238](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/conftest.py#L18-L238)
- Script parity is enforced by dedicated tests — `test_setup_plan_python_parity.py`, `test_create_new_feature_python_parity.py`, `test_resolve_template_python_parity.py`, `test_check_prerequisites_python_parity.py`, `test_setup_tasks_python_parity.py` — matching the AGENTS.md rule that all `scripts:` frontmatter entries must be behaviorally equivalent across sh/ps/py. — [tests/test_setup_plan_python_parity.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/test_setup_plan_python_parity.py) [tests/parity_helpers.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/parity_helpers.py) [AGENTS.md](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/AGENTS.md)
- Contract tests under `tests/contract/` pin the wheel's bundled payload and schemas: `test_wheel_core_pack_scripts.py`, `test_wheel_bundled_presets.py`, `test_wheel_bundled_workflows.py`, `test_catalog_schema.py`, `test_manifest_schema.py`, `test_firstparty_bundle_catalog_consistency.py`. — [tests/contract/test_wheel_core_pack_scripts.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/contract/test_wheel_core_pack_scripts.py) [tests/contract/test_catalog_schema.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/contract/test_catalog_schema.py)
- Workflow coverage lives in `tests/workflows/` (bundled speckit/bugfix/assess workflow validation, overlay composer/merge/schema/security, slot step, resolver integration) plus top-level `test_workflows.py` and `test_workflow_run_without_project.py`. — [tests/workflows/test_bundled_speckit_workflow.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/workflows/test_bundled_speckit_workflow.py) [tests/workflows/test_overlay_security.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/workflows/test_overlay_security.py) [tests/test_workflows.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/test_workflows.py)
- Security-focused suites exist for the risky paths: `test_download_security.py` (bounded reads/extraction), `test_registrar_path_traversal.py`, `test_command_add_path_traversal.py`, `test_security_workflow.py`, and `test_authentication.py`. — [tests/test_download_security.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/test_download_security.py) [tests/specify_cli/extensions/test_command_add_path_traversal.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/specify_cli/extensions/test_command_add_path_traversal.py) [tests/test_authentication.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/test_authentication.py)
- CI: `.github/workflows/test.yml` runs ruff (`uvx ruff@0.15.0 check src tests`) then pytest on ubuntu/windows/macos × Python 3.13/3.14; `lint.yml` adds `git diff --check` and markdownlint-cli2; `security.yml` runs a dependency audit script; `codeql.yml` runs CodeQL. — [.github/workflows/test.yml](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/.github/workflows/test.yml) [.github/workflows/lint.yml](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/.github/workflows/lint.yml) [.github/workflows/security.yml](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/.github/workflows/security.yml) [.github/workflows/codeql.yml](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/.github/workflows/codeql.yml)
- The suite is heavily fixture/monkeypatch-driven (tmp_path projects, faked argv0, stubbed catalogs) rather than spawning real agent CLIs; `_has_working_bash` auto-skips shell-dependent tests when bash is absent, so script-level coverage can silently thin out on under-provisioned runners. (inferred) — [tests/conftest.py#L18-67,179-238](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/conftest.py#L18-67,179-L238)
- Extension behavior has both legacy consolidated suites (`tests/test_extensions.py`, `tests/test_presets.py`) and newer per-command mirrored suites; `design/cli.md` explicitly allows the consolidated domain suites to remain until separately reorganized, so coverage is duplicated across both styles rather than missing. — [design/cli.md#L234-248](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/design/cli.md#L234-L248) [tests/test_extensions.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/test_extensions.py) [tests/specify_cli/extensions/test_command_update.py](https://github.com/github/spec-kit/blob/175adbcd826befdccce8c60faa07ae3d8bbc8d87/tests/specify_cli/extensions/test_command_update.py)

Every claim links to its cited files at the pinned commit. Citations identify source evidence; they do not establish runtime correctness. Independent CodeVetter scan — not reviewed by upstream maintainers.

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)