CodeVetter · Repo Unpack · Testing signals
Graphify-Labs/graphify
Testing is pytest-based with ~400 test files under tests/, one file per module plus many regression tests named after GitHub issues (test_issue_3405_*, etc.). Fixtures cover cross-language corpora (crate_a/crate_b Cargo workspaces, cpp/objc dirs, SQL samples). conftest.py sandboxes HOME and strips backend env vars so installers and detect_backend never touch the developer's real config. CI runs the suite on Python 3.10-3.14 plus a skillgen drift-guard job and non-blocking bandit/pip-audit scans.
- pytest is the test framework: pyproject sets testpaths=['tests'], norecursedirs excludes worked/ and corpus dirs, and the dev dependency group adds hypothesis, pytest-cov, ruff, pyright, bandit, pip-audit.
- conftest.py autouse fixtures give every test a throwaway HOME (protecting ~/.claude, ~/.agents, etc. from installer tests) and delete every env var detect_backend() consults so backend-detection tests are hermetic.
- ARCHITECTURE.md's module table is load-bearing documentation: tests/test_architecture_doc.py imports every symbol the doc names, so the architecture doc cannot drift from the code without a red test.
- The extractor migration is guarded by tests/test_extractors_registry.py — each ported language gets a module-import + facade-identity + registry-identity test before the move lands.
- tests/fixtures/ contains real mini-corpora: crate_a/crate_b Cargo workspaces for cross-crate call tests, cpp_paired/cpp_samedir dirs, objc_mixed, pascal_cross_file, robot_keywords.resource, sample_*.sql, and a committed extraction.json.
- CI test job runs `uv sync --all-extras --frozen` then `pytest tests/ -q --tb=short` on Python 3.10, 3.12, 3.13, 3.14, plus an install smoke test (`graphify --help`, `graphify install`).
- A separate skillgen-check CI job enforces that generated skill artifacts match their fragments: --check byte-diff, --audit-coverage per host, --schema-singleton file_type enum, --monolith-roundtrip and --always-on-roundtrip against the pinned v8 baseline commit.
- Security scanning exists but is advisory: bandit -r graphify -ll and pip-audit --strict both run with continue-on-error: true, so a new HIGH finding does not fail the build.
- Coverage breadth: dedicated test files exist for the CLI surface (test_cli_help, test_explain_cli, test_query_cli, test_path_cli, test_extract_cli), security (test_security, test_injection_sentinel_coverage), caching/incremental (test_cache, test_incremental, test_partial_cache), serve/MCP (test_serve, test_serve_http, test_query_mcp_direction), and ~40 per-language extraction tests (test_csharp_*, test_js_*, test_pascal_*, test_terraform, ...).
- Performance benchmarks live in-repo as tests/bench_extract.py and tests/bench_query_scoring.py, separate from the unit suite.
- Gap: the skill.md orchestration path (host-agent subagents writing chunk JSONs) is validated only indirectly via install-string and skillgen tests; no test exercises a real end-to-end /graphify run against a live assistant. The LLM client code is tested at the parsing/retry layer (test_llm_parser, test_llm_backends) but no live-API tests exist — consistent with the 'pure unit tests, no network calls' claim in ARCHITECTURE.md. (inferred)
- Regression-heavy style: dozens of tests are named for fixed issues (test_issue_3405_python_resolution, test_issue_3472_source_file_collision) and source comments cite issue numbers (#1392, #1666, #1941...) pervasively, meaning the test suite is the primary record of past failure modes.