CodeVetter · Repo Unpack · Testing signals
yt-dlp/yt-dlp
Tests live in `test/` (pytest + unittest hybrid). Two tiers: 'core' tests (utils, networking, jsinterp, postprocessors, options) run offline under `-m 'not download'`, and 'download' tests that hit live sites, generated dynamically from each extractor's `_TESTS`/`_WEBPAGE_TESTS`. CI runs core tests across CPython 3.10-3.15 and PyPy on Linux+Windows with reruns; lint is ruff + autopep8. Coverage is explicitly unimplemented. The extractor download tests depend on live third-party sites and are therefore inherently flaky and mostly excluded from core CI.
- pytest is the test runner (`pytest~=9.0`, `pytest-rerunfailures`), configured with `-ra`, `--tb=short`, `strict_markers`, and a custom `download` marker.
- `test_download.py` dynamically generates `TestDownload` unittest methods from `gettestcases()`/`getwebpagetestcases()`, which iterate every extractor's `_TESTS`/`_WEBPAGE_TESTS`; tests download real URLs and verify info dicts plus md5 checksums.
- `devscripts/run_tests.py` is the canonical entrypoint: `core` => `pytest -m 'not download'`, `download` => `-m download`, and bare names map to `test/test_download.py::TestDownload::test_<IE>`; it falls back to `python -m unittest` when pytest is absent.
- Core CI (`core.yml`) runs on ubuntu+windows across py3.11-3.15 and pypy-3.11 (py3.10 on Windows + quick-test.yml on Linux), skipping `yt_dlp/extractor/**` changes except youtube/common/extractors, and adds `--flaky` only when networking files changed.
- `test_all_urls.py` asserts URL->extractor matching precedence by calling `ie.suitable(url)` across all extractors, guarding against `_VALID_URL` regressions like youtu.be and playlist/tab disambiguation.
- Networking tests are parametrized per request handler via a `handler` pytest fixture that looks up classes in `_REQUEST_HANDLERS` and supports `skip_handler`/`skip_handler_if` markers for missing optional deps.
- Test parameters come from `test/parameters.json` (`test: true`, `writeinfojson`, `format: 'b/bv'`) mergeable with a git-ignored `local_parameters.json` for secrets/overrides.
- Dedicated suites exist for the YouTube PO-token provider framework (`test_pot_framework.py`, director, builtin providers) and the JS-challenge/ejs integration (`test_ejs_integration.py`).
- `make test` = `pytest -Werror` then `make codetest` (ruff check + autopep8 --diff); `make offlinetest` runs `-m 'not download'` plus lint; hatch env `hatch-test` adds pytest-randomly and pytest-xdist.
- Code coverage is not wired up: the hatch `run-cov` script is a stub that echoes 'Code coverage not implemented' and exits 1.
- Download tests self-skip for extractors marked `_WORKING = False` and for `expected_exception` cases, acknowledging the live-site flakiness; warnings are escalated to errors via a `report_warning` override.
- A `FakeYDL` helper subclasses `YoutubeDL` with `auto_init=False`, `trouble`->raise, and `download`->record, so unit tests exercise the real orchestrator without network.