CodeVetter · Repo Unpack · Testing signals
pytorch/pytorch
Tests live in test/ (~411 files) and run under pytest driven by the custom harness test/run_test.py, with unittest-style TestCase classes from torch.testing._internal.common_utils. Device-generic coverage comes from instantiate_device_type_tests, and snapshot assertions from expecttest (assertExpectedInline). CI runs pull.yml on PRs and trunk.yml on main, with a job-filter and target-determination pipeline that shards tests and can run only ~25% via TD. Env vars (PYTORCH_TEST_WITH_*) gate ASAN/ROCM/slow/dynamo/inductor configurations; disabled and slow test lists are consumed from JSON files. C++ tests are built as gtest binaries and discovered by run_test.py.
- pytest.ini configures the suite: testpaths=test, --tb=native, --capture=sys, no warnings plugin, plain asserts, xfail_strict=True, and custom markers serial/multigpu/periodic (sharding via a custom pytest_shard_custom.py, not pytest-shard).
- test/run_test.py partitions TESTS into groups (INDUCTOR_TESTS, DISTRIBUTED_TESTS, DYNAMO_CORE_TESTS, CPYTHON_TESTS, ONNX_TESTS, QUANTIZATION_TESTS, FUNCTORCH_TESTS, CPP_TESTS) and supports TESTS_TO_INCLUDE env, --include/--exclude, sharding (do_sharding) using per-file/class historical times, and TD mode that runs ~25% of tests.
- The shared base is TestCase(expecttest.TestCase) in common_utils (line ~3720) with assertExpectedInline for golden-string snapshot tests; run_tests() lints that every test class subclasses TestCase, supports --discover, --save-xml, subprocess-per-test-case mode, slow/disabled test JSON import, and 15-minute timeouts with one retry.
- Test gating is env-var driven: PYTORCH_TEST_WITH_ASAN/TSAN/UBSAN/ROCM/MTIA/PERIODIC/SLOW/DYNAMO/INDUCTOR/AOT_EAGER, PYTORCH_TEST_SKIP_FAST, PYTORCH_TESTING_DEVICE_EXCEPT_FOR, PYTORCH_TEST_CUDA_MEM_LEAK_CHECK etc. select configurations at module import.
- AGENTS.md mandates the test idiom: subclass torch.testing._internal.common_utils.TestCase, call run_tests() in __main__, use assertEqual for tensor equality, @parametrize for input matrices, and instantiate_device_type_tests for device-generic numeric tests.
- Dynamo tests live in test/dynamo/ using torch._dynamo.test_case.TestCase (auto-resets dynamo and patches strict config); debugging workflows center on TORCH_LOGS=guards,graph_breaks,+dynamo and CompileCounter/EagerAndRecordGraphs helpers.
- CI orchestration: pull.yml (PRs) and trunk.yml (main/release pushes + cron) both call job-filter.yml then target_determination.yml (fed by llm_td_retrieval.yml) before the reusable _binary-build-* and _binary-test-* per-platform workflows.
- Result plumbing: run_test.py uploads artifacts mid-run in CI (parse_xml_and_upload_json + zip_and_upload_artifacts), writes into test/test-reports/, and maps negative exit codes to signal names for crash reporting.
- C++ tests are gtest binaries discovered via _is_cpp_test; googletest is a third_party submodule, and conftest.py/pytest_shard_custom.py are copied into CPP_TESTS_DIR so cpp tests share the pytest sharding path.
- Lint/type gates are configured in-tree: .lintrunner.toml (lintrunner -a before commits), ruff/isort/usort config in pyproject.toml (line-length 88), mypy.ini + mypy-strict.ini and pyrefly.toml.
- Disabled/flaky tests are data-driven: PYTORCH_TEST_RERUN_DISABLED_TESTS re-runs entries from DISABLED_TESTS_FILE, and CI downloads test-case configs and slow-test classifications (SLOW_TESTS_FILE) before running.