CodeVetter · Repo Unpack · Testing signals
openai/whisper
Tests use pytest in tests/ (5 modules + conftest registering a `requires_cuda` marker and a `random` seed fixture). Coverage is good for the numeric utilities (dtw, median_filter vs scipy), normalizers (extensive assertions), and tokenizer edge cases, plus one end-to-end transcription test parametrized over all 13 model checkpoints against tests/jfk.flac. Major gaps: no coverage of the CLI, output writers, beam search, logit filters, model download, or the transcribe seek-loop heuristics. CI runs pre-commit (black/isort/flake8) and a 9-cell Python×PyTorch matrix on CPU-only torch wheels.
- Test framework is pytest; conftest.py registers the `requires_cuda` marker and provides a `random` fixture seeding both `random` and `numpy.random` to 42.
- test_transcribe parametrizes over every name in whisper.available_models() (13 checkpoints, downloaded on demand), transcribes tests/jfk.flac with word_timestamps=True, asserts 'my fellow americans' etc., tokenizer decode round-trip, and that the word 'Americans' brackets t≈1.8 s.
- test_audio verifies load_audio returns a 1-D waveform of ~10-12 s with std in (0,1) and that log_mel_spectrogram from array equals from-file, with mel dynamic range ≤ 2.0.
- test_timing checks dtw_cpu against a manually constructed optimal path, median_filter against scipy.ndimage.median_filter on 4 shapes × widths {3,5,7,13}, and CUDA equivalence for both kernels gated behind requires_cuda.
- test_normalizer exercises EnglishNumberNormalizer and EnglishTextNormalizer with ~70 assertions (ordinals, currency, 'double oh seven', 'and a half', contractions, titles) and EnglishSpellingNormalizer mappings.
- test_tokenizer covers sot_sequence membership, language-token/code parity, multilingual vs gpt2 encode/decode round-trip on Korean text, and split_tokens_on_unicode on a partial-UTF8 French sequence; note the `multilingual` parametrize value is ignored — the test always calls get_tokenizer(multilingual=False).
- CI (test.yml) runs a pre-commit job (black, isort, flake8, whitespace hooks) then a matrix over Python 3.8–3.13 × torch 1.10.1–2.5.1 with CPU wheels, conda-installed ffmpeg, invoking `pytest --durations=0 -vv -k 'not test_transcribe or test_transcribe[tiny] or test_transcribe[tiny.en]' -m 'not requires_cuda'`.
- Untested surface: cli() argument parsing and validation, all six ResultWriter formats, BeamSearchDecoder, every LogitFilter, _download SHA256 logic, and the hallucination-silence heuristics have no tests. (inferred)
- Dev extras (black, flake8, isort, pytest, scipy) are declared in pyproject [project.optional-dependencies].dev; scipy is only needed by test_timing's reference comparison.