# Testing signals — astral-sh/ruff

> Canonical page: https://codevetter.com/unpack/ruff/testing-signals

# Testing signals — astral-sh/ruff

Scanned commit c131cbad5a0f858ef2ba207e0260d76bb33069b1 (2026-09-22). Generated by codevetter codevetter 1.12.0 on 2026-09-22.

Tests are Rust-native: cargo-nextest in CI, insta snapshots, and ty Markdown fixtures executed by `datatest-stable`. Parser, linter, and formatter crates carry large snapshot and fixture trees; the Ruff CLI has integration tests under `crates/ruff/tests`. CI path-filters which jobs run, and additionally dogfoods `ty check` on in-repo Python. Fuzz targets cover parse, fix, and format idempotency. Playground Worker code and file-watching tests are the obvious gaps.
- Linux CI job `cargo test (linux)` runs `uv run --locked --only-dev cargo insta test --all-features --unreferenced reject --test-runner nextest --disable-nextest-doctest`, then `cargo test --doc --all-features`. It is skipped when the PR has label `no-test` or `determine_changes` says no code changed (except on main). — [.github/workflows/ci.yaml#L379-L417](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/.github/workflows/ci.yaml#L379-L417)
- The same job dogfoods ty: `cargo run -p ty check` on `python/py-fuzzer`, `scripts/ty_benchmark`, and `scripts/*.py` (the last with `TY_UV=scripts` and `UV_LOCKED=true`). A prior step runs `cargo test -p ty_python_semantic --test mdtest` with `MDTEST_GITHUB_ANNOTATIONS_FORMAT=1` and ignores its exit code so annotations cannot fail the job. — [.github/workflows/ci.yaml#L406-L427](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/.github/workflows/ci.yaml#L406-L427)
- CI also runs `cargo fmt`, `shellcheck`, `cargo clippy` (including a wasm clippy job), `cargo hawk` at HAWK_VERSION 0.1.13, and a `cargo publish` dry-run. `determine_changes` sets outputs `parser`, `linter`, `formatter`, `ty`, `playground`, `fuzz`, `py-fuzzer`, `benchmarks`, and `release` from path filters. Workflow permissions are `contents: read`. — [.github/workflows/ci.yaml#L1-L64](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/.github/workflows/ci.yaml#L1-L64) [.github/workflows/ci.yaml#L279-L375](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/.github/workflows/ci.yaml#L279-L375)
- ty semantic tests are Markdown files. `crates/ty_python_semantic/tests/mdtest.rs` registers `mdtest` and `lint_doc` via datatest-stable; each fixture runs on a one-thread Rayon pool. `lint_doc` looks the filename stem up in `default_lint_registry` and passes it as `default_error_rule`. Fixtures live under `./resources/mdtest` and `./resources/lint_docs`. — [crates/ty_python_semantic/tests/mdtest.rs#L1-L40](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/crates/ty_python_semantic/tests/mdtest.rs#L1-L40)
- Ruff CLI coverage includes `crates/ruff/tests` (integration and CLI snapshot tests) and a unit test `unreadable_files` in `commands/check.rs` that creates mode-0o000 files and snapshots E902 / invalid-pyproject diagnostics with insta. Rule crates keep insta snapshots beside rules (for example `rules/pyflakes` and `rules/pylint`). — [crates/ruff/src/commands/check.rs#L217-L302](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/crates/ruff/src/commands/check.rs#L217-L302)
- Parser tests are fixture-driven: `crates/ruff_python_parser/resources` holds hundreds of `.py` fixtures and `crates/ruff_python_parser/tests` holds matching snapshots. Formatter tests similarly pair `resources/test` fixtures with `tests` snapshots. The parser fuzz target `ruff_parse_simple` and idempotency target `ruff_parse_idempotency` sit outside the workspace. — [fuzz/Cargo.toml#L47-L69](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/fuzz/Cargo.toml#L47-L69)
- Fuzz package `ruff-fuzz` (separate `[workspace]`, cargo-fuzz, libfuzzer-sys) builds bins `ty_check_invalid_syntax`, `ruff_parse_simple`, `ruff_fix_validity`, `ruff_formatter_validity`, `ruff_parse_idempotency`, and `ruff_formatter_idempotency`. It depends on the linter, parser, formatter, and ty semantic crates with the semantic `testing` feature. — [fuzz/Cargo.toml#L1-L69](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/fuzz/Cargo.toml#L1-L69)
- AGENTS.md requires `INSTA_FORCE_PASS=1 INSTA_UPDATE=always MDTEST_UPDATE_SNAPSHOTS=1 cargo nextest run` (or `cargo test` if nextest is missing), forbids hand-editing snapshot bodies, and says file-watcher tests do not work in the sandbox. Clippy gate is `cargo clippy --workspace --all-targets --all-features -- -D warnings`. — [AGENTS.md](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/AGENTS.md)
- `pyproject.toml` defines a `ruff-lsp-test` dependency group (pytest, pygls 1.x, lsprotocol) separate from the Rust LSP crates, and a `ty-ecosystem` group on `ecosystem-analyzer`. Those are integration/ecosystem harnesses, not the in-crate nextest suites. — [pyproject.toml#L80-L93](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/pyproject.toml#L80-L93) [pyproject.toml#L151-L158](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/pyproject.toml#L151-L158)
- The playground Worker (`playground/api/src/index.ts`) has no test module beside it, and `playground/package.json` scripts are `dev:wasm`, oxlint, `tsc`, and prettier rather than a behavioral test runner. WASM API tests exist as `crates/ruff_wasm/tests/api.rs` and `crates/ty_wasm/tests/api.rs`, which is a different layer. — [playground/package.json#L9-L16](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/playground/package.json#L9-L16) [playground/api/src/index.ts](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/playground/api/src/index.ts)
- An ignored `linter_quick_test` in `ruff_linter` is an ad-hoc debug entry point (`#[ignore]`) that runs one rule through `test_contents`. It is not part of normal CI coverage. — [crates/ruff_linter/src/lib.rs#L59-L95](https://github.com/astral-sh/ruff/blob/c131cbad5a0f858ef2ba207e0260d76bb33069b1/crates/ruff_linter/src/lib.rs#L59-L95)

Every claim links to its cited files at the pinned commit. Citations identify source evidence; they do not establish runtime correctness. Independent CodeVetter scan — not reviewed by upstream maintainers.

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)