CodeVetter · Repo Unpack · Testing signals
astral-sh/ruff
Tests are Rust-native: cargo-nextest in CI, insta snapshots, and ty Markdown fixtures executed by `datatest-stable`. Parser, linter, and formatter crates carry large snapshot and fixture trees; the Ruff CLI has integration tests under `crates/ruff/tests`. CI path-filters which jobs run, and additionally dogfoods `ty check` on in-repo Python. Fuzz targets cover parse, fix, and format idempotency. Playground Worker code and file-watching tests are the obvious gaps.
- Linux CI job `cargo test (linux)` runs `uv run --locked --only-dev cargo insta test --all-features --unreferenced reject --test-runner nextest --disable-nextest-doctest`, then `cargo test --doc --all-features`. It is skipped when the PR has label `no-test` or `determine_changes` says no code changed (except on main).
- The same job dogfoods ty: `cargo run -p ty check` on `python/py-fuzzer`, `scripts/ty_benchmark`, and `scripts/*.py` (the last with `TY_UV=scripts` and `UV_LOCKED=true`). A prior step runs `cargo test -p ty_python_semantic --test mdtest` with `MDTEST_GITHUB_ANNOTATIONS_FORMAT=1` and ignores its exit code so annotations cannot fail the job.
- CI also runs `cargo fmt`, `shellcheck`, `cargo clippy` (including a wasm clippy job), `cargo hawk` at HAWK_VERSION 0.1.13, and a `cargo publish` dry-run. `determine_changes` sets outputs `parser`, `linter`, `formatter`, `ty`, `playground`, `fuzz`, `py-fuzzer`, `benchmarks`, and `release` from path filters. Workflow permissions are `contents: read`.
- ty semantic tests are Markdown files. `crates/ty_python_semantic/tests/mdtest.rs` registers `mdtest` and `lint_doc` via datatest-stable; each fixture runs on a one-thread Rayon pool. `lint_doc` looks the filename stem up in `default_lint_registry` and passes it as `default_error_rule`. Fixtures live under `./resources/mdtest` and `./resources/lint_docs`.
- Ruff CLI coverage includes `crates/ruff/tests` (integration and CLI snapshot tests) and a unit test `unreadable_files` in `commands/check.rs` that creates mode-0o000 files and snapshots E902 / invalid-pyproject diagnostics with insta. Rule crates keep insta snapshots beside rules (for example `rules/pyflakes` and `rules/pylint`).
- Parser tests are fixture-driven: `crates/ruff_python_parser/resources` holds hundreds of `.py` fixtures and `crates/ruff_python_parser/tests` holds matching snapshots. Formatter tests similarly pair `resources/test` fixtures with `tests` snapshots. The parser fuzz target `ruff_parse_simple` and idempotency target `ruff_parse_idempotency` sit outside the workspace.
- Fuzz package `ruff-fuzz` (separate `[workspace]`, cargo-fuzz, libfuzzer-sys) builds bins `ty_check_invalid_syntax`, `ruff_parse_simple`, `ruff_fix_validity`, `ruff_formatter_validity`, `ruff_parse_idempotency`, and `ruff_formatter_idempotency`. It depends on the linter, parser, formatter, and ty semantic crates with the semantic `testing` feature.
- AGENTS.md requires `INSTA_FORCE_PASS=1 INSTA_UPDATE=always MDTEST_UPDATE_SNAPSHOTS=1 cargo nextest run` (or `cargo test` if nextest is missing), forbids hand-editing snapshot bodies, and says file-watcher tests do not work in the sandbox. Clippy gate is `cargo clippy --workspace --all-targets --all-features -- -D warnings`.
- `pyproject.toml` defines a `ruff-lsp-test` dependency group (pytest, pygls 1.x, lsprotocol) separate from the Rust LSP crates, and a `ty-ecosystem` group on `ecosystem-analyzer`. Those are integration/ecosystem harnesses, not the in-crate nextest suites.
- The playground Worker (`playground/api/src/index.ts`) has no test module beside it, and `playground/package.json` scripts are `dev:wasm`, oxlint, `tsc`, and prettier rather than a behavioral test runner. WASM API tests exist as `crates/ruff_wasm/tests/api.rs` and `crates/ty_wasm/tests/api.rs`, which is a different layer.
- An ignored `linter_quick_test` in `ruff_linter` is an ad-hoc debug entry point (`#[ignore]`) that runs one rule through `test_contents`. It is not part of normal CI coverage.