# Testing signals — openai/whisper

> Canonical page: https://codevetter.com/unpack/whisper/testing-signals

# Testing signals — openai/whisper

Scanned commit 86098128c0b4f24f0e2aa2994de830614b474227 (2026-08-31). Generated by codevetter codevetter 1.12.0 on 2026-09-22.

Tests use pytest in tests/ (5 modules + conftest registering a `requires_cuda` marker and a `random` seed fixture). Coverage is good for the numeric utilities (dtw, median_filter vs scipy), normalizers (extensive assertions), and tokenizer edge cases, plus one end-to-end transcription test parametrized over all 13 model checkpoints against tests/jfk.flac. Major gaps: no coverage of the CLI, output writers, beam search, logit filters, model download, or the transcribe seek-loop heuristics. CI runs pre-commit (black/isort/flake8) and a 9-cell Python×PyTorch matrix on CPU-only torch wheels.
- Test framework is pytest; conftest.py registers the `requires_cuda` marker and provides a `random` fixture seeding both `random` and `numpy.random` to 42. — [tests/conftest.py](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/tests/conftest.py)
- test_transcribe parametrizes over every name in whisper.available_models() (13 checkpoints, downloaded on demand), transcribes tests/jfk.flac with word_timestamps=True, asserts 'my fellow americans' etc., tokenizer decode round-trip, and that the word 'Americans' brackets t≈1.8 s. — [tests/test_transcribe.py](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/tests/test_transcribe.py) [tests/jfk.flac](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/tests/jfk.flac)
- test_audio verifies load_audio returns a 1-D waveform of ~10-12 s with std in (0,1) and that log_mel_spectrogram from array equals from-file, with mel dynamic range ≤ 2.0. — [tests/test_audio.py](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/tests/test_audio.py)
- test_timing checks dtw_cpu against a manually constructed optimal path, median_filter against scipy.ndimage.median_filter on 4 shapes × widths {3,5,7,13}, and CUDA equivalence for both kernels gated behind requires_cuda. — [tests/test_timing.py](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/tests/test_timing.py)
- test_normalizer exercises EnglishNumberNormalizer and EnglishTextNormalizer with ~70 assertions (ordinals, currency, 'double oh seven', 'and a half', contractions, titles) and EnglishSpellingNormalizer mappings. — [tests/test_normalizer.py](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/tests/test_normalizer.py)
- test_tokenizer covers sot_sequence membership, language-token/code parity, multilingual vs gpt2 encode/decode round-trip on Korean text, and split_tokens_on_unicode on a partial-UTF8 French sequence; note the `multilingual` parametrize value is ignored — the test always calls get_tokenizer(multilingual=False). — [tests/test_tokenizer.py](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/tests/test_tokenizer.py)
- CI (test.yml) runs a pre-commit job (black, isort, flake8, whitespace hooks) then a matrix over Python 3.8–3.13 × torch 1.10.1–2.5.1 with CPU wheels, conda-installed ffmpeg, invoking `pytest --durations=0 -vv -k 'not test_transcribe or test_transcribe[tiny] or test_transcribe[tiny.en]' -m 'not requires_cuda'`. — [.github/workflows/test.yml](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/.github/workflows/test.yml) [.pre-commit-config.yaml](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/.pre-commit-config.yaml)
- Untested surface: cli() argument parsing and validation, all six ResultWriter formats, BeamSearchDecoder, every LogitFilter, _download SHA256 logic, and the hallucination-silence heuristics have no tests. (inferred) — [tests/](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/tests/) [whisper/transcribe.py#L517-619](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/transcribe.py#L517-L619) [whisper/utils.py#L306-329](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/utils.py#L306-L329) [whisper/decoding.py#L301-505](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/decoding.py#L301-L505)
- Dev extras (black, flake8, isort, pytest, scipy) are declared in pyproject [project.optional-dependencies].dev; scipy is only needed by test_timing's reference comparison. — [pyproject.toml#L33](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/pyproject.toml#L33) [tests/test_timing.py#L3](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/tests/test_timing.py#L3)

Every claim links to its cited files at the pinned commit. Citations identify source evidence; they do not establish runtime correctness. Independent CodeVetter scan — not reviewed by upstream maintainers.

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)