openai/whisper
github.com/openai/whisper ↗ · 42 files · 2.2 MB
Whisper [[Blog]](https://openai.com/blog/whisper) [[Paper]](https://arxiv.org/abs/2212.04356) [[Model card]](https://github.com/openai/whisper/blob/main/model-card.md) [[Colab example]](https://colab.research.google.com/github/openai/whisper/blob/master/notebo
Overview
OpenAI Whisper is a general-purpose speech-recognition library (Python + PyTorch) that ships both a `whisper` CLI and an importable `whisper` package. It downloads pretrained encoder-decoder Transformer checkpoints (tiny through large-v3-turbo) from OpenAI's CDN with SHA256 verification, decodes arbitrary audio to 16 kHz mono via an ffmpeg subprocess, converts it to log-Mel spectrograms, and runs an autoregressive seq2seq decoder over 30-second sliding windows. A single multitask model performs multilingual transcription, X→English translation, language identification, and (via cross-attention DTW) word-level timestamp alignment, emitting results as txt/vtt/srt/tsv/json/jsonl.
System map
The repo is a single flat Python package, `whisper/`, plus a `tests/` pytest suite, GitHub Actions CI, and evaluation docs/notebooks. The CLI entrypoint `whisper.transcribe:cli` (also reachable via `python -m whisper`) parses ~30 argparse options, loads a checkpoint through `load_model` in `whisper/__init__.py`, and runs `transcribe()`. Checkpoints are runtime downloads from openaipublic.azureedge.net, verified by a SHA256 embedded in the URL and cached under ~/.cache/whisper. The model itself is a standard Transformer encoder/decoder defined in model.py; decoding.py implements the sampling engine (greedy/beam, logit filters, KV-cache hooks); audio.py, tokenizer.py, timing.py, utils.py, and triton_ops.py cover the audio pipeline, BPE vocab, DTW alignment, output writers, and optional CUDA kernels respectively.
- The console script `whisper` maps to `whisper.transcribe:cli` in pyproject `[project.scripts]`, and `whisper/__main__.py` calls the same `cli()` function, so `whisper` and `python -m whisper` are equivalent entrypoints.
- The package is organized as flat modules under whisper/: audio.py (mel pipeline), model.py (nn.Modules), decoding.py (sampling engine), tokenizer.py (tiktoken wrapper), timing.py (DTW alignment), transcribe.py (orchestration + CLI), utils.py (writers/helpers), normalizers/ (evaluation text normalization), triton_ops.py (CUDA kernels).
- whisper/__init__.py is the public API surface: it re-exports load_audio, log_mel_spectrogram, pad_or_trim, decode, detect_language, DecodingOptions, DecodingResult, Whisper, ModelDimensions, transcribe, and __version__, and holds the _MODELS download-URL table plus load_model().
- Model weights are downloaded at runtime from openaipublic.azureedge.net URLs in _MODELS; the expected SHA256 is the second-to-last URL path segment, checked both before reuse of a cached file and after download, defaulting to $XDG_CACHE_HOME/whisper or ~/.cache/whisper.
- Whisper is a conventional encoder-decoder Transformer: AudioEncoder applies two Conv1d layers (second stride 2), adds sinusoidal positional embeddings, then ResidualAttentionBlocks; TextDecoder uses learned positional embeddings, cross-attention blocks, a causal mask buffer, and tied output logits computed against token_embedding.weight.
- ffmpeg is a hard external dependency: load_audio() spawns `ffmpeg -nostdin -threads 0 -i <file> -f s16le -ac 1 -acodec pcm_s16le -ar 16000 -` and parses stdout as int16 scaled to float32.
- Declared dependencies are more-itertools, numba, numpy, tiktoken, torch, tqdm, and triton>=2 (Linux x86_64 only); Python >=3.8 is required.
- Binary assets ship inside the package: whisper/assets/gpt2.tiktoken and multilingual.tiktoken BPE vocab files, mel_filters.npz (80- and 128-band filterbanks), and normalizers/english.json (UK→US spelling map), all declared in MANIFEST.in.
- The Whisper class binds module-level functions as class attributes (`transcribe = transcribe_function`, `decode = decode_function`, `detect_language = detect_language_function`), so `model.transcribe(...)`, `model.decode(...)`, and `model.detect_language(...)` work without separate methods.
- Optional GPU acceleration lives in triton_ops.py (Triton JIT kernels for DTW and median filter); callers in timing.py wrap kernel launches in try/except and fall back to numba-JIT CPU implementations with a warning.
- Distribution is the PyPI package `openai-whisper` with dynamic version read from whisper/version.py (__version__ = "20250625"); python-publish.yml triggers on commits to main whose message matches `^Release <v>` and uploads via twine.
- Build/test: `pip install .[dev]` installs setuptools-built package plus pytest/black/flake8/isort/scipy; CI runs pre-commit then `pytest -k 'not test_transcribe or test_transcribe[tiny] or test_transcribe[tiny.en]' -m 'not requires_cuda'` on a 9-cell Python×PyTorch matrix.
Feature catalog
User-facing surface is (a) the `whisper` CLI, (b) the Python API (load_model, transcribe, decode, detect_language), and (c) six output formats. Core capabilities: multilingual transcription across 99 languages, X→English translation, automatic language detection, word-level timestamps via attention-based DTW, clip selection, prompt conditioning, and two search strategies (greedy/best-of-N vs beam search). Auxiliary features include text normalizers for evaluation and a hallucination-silence skip heuristic.
- The CLI accepts one or more audio paths plus ~30 flags including --model (default `turbo`), --task, --language, --output_dir/-o, --output_format/-f (txt|vtt|srt|tsv|json|jsonl|all), --verbose, --fp16, and --threads.
- Python API: `whisper.load_model(name)` returns a Whisper on cuda-if-available; `model.transcribe(path)` returns {text, segments, language}; lower-level `whisper.decode(model, mel, options)` and `whisper.detect_language(model, mel)` are exported.
- Thirteen model names are available — tiny/base/small/medium each with a `.en` English-only variant, large-v1/v2/v3, `large` (alias of large-v3), and large-v3-turbo/`turbo` — all mapped to CDN URLs in _MODELS.
- Tasks: `transcribe` (same-language ASR) and `translate` (X→English) are selected via --task and encoded as <|transcribe|>/<|translate|> special tokens in the SOT sequence; `lang_id` is a third internal task that short-circuits after language detection.
- Automatic language detection: when --language is omitted on a multilingual model, transcribe() runs model.detect_language() on the first 30 s mel segment and picks argmax over per-language probabilities.
- Six output writers exist — WriteTXT, WriteVTT, WriteSRT, WriteTSV, WriteJSON, WriteJSONL — selected by name through the get_writer() registry; `all` instantiates every writer.
- Word-level timestamps (--word_timestamps) align tokens to audio via cross-attention weights + dynamic time warping; supporting flags --highlight_words (underline in srt/vtt), --max_line_width, --max_line_count, --max_words_per_line control subtitle reflow in SubtitlesWriter.iterate_result.
- Search strategies: temperature==0 uses BeamSearchDecoder (--beam_size, --patience, --length_penalty); temperature>0 uses GreedyDecoder with sampling and --best_of candidates; MaximumLikelihoodRanker picks the final hypothesis.
- Fallback ladder: transcribe() accepts a temperature tuple (CLI builds it via --temperature_increment_on_fallback) and retries each 30 s window at higher temperatures when compression_ratio_threshold or logprob_threshold fails; no_speech_threshold can instead mark the window silent.
- --clip_timestamps accepts comma-separated start,end pairs (seconds) restricting transcription to those ranges; --hallucination_silence_threshold skips long silences around anomalous word segments when word_timestamps is on.
- Prompt conditioning: --initial_prompt seeds the first window, --carry_initial_prompt prepends it to every window, and --condition_on_previous_text feeds prior output tokens as prompt for the next window.
- Language support covers 99 languages enumerated in LANGUAGES (tokenized as <|en|>…<|yue|> specials) plus alias resolution in TO_LANGUAGE_CODE (e.g. burmese→my, mandarin→zh).
Data flow
Audio enters as a file path (decoded by an ffmpeg subprocess to 16 kHz mono float32) or a NumPy/Torch waveform, then becomes a log-Mel spectrogram (80 or 128 bins, STFT n_fft=400 hop=160) padded with 30 s of silence. The mel tensor feeds AudioEncoder once per 30 s window; DecodingTask autoregressively produces tokens under logit filters and a KV cache, with timestamp tokens (>=tokenizer.timestamp_begin, 0.02 s each) delimiting segments. Segments accumulate in all_segments/all_tokens; optional word-level alignment runs a second decoder pass capturing cross-attention weights through forward hooks. The final dict {text, segments, language} returns to the caller and is serialized by ResultWriter subclasses into output_dir.
- Input boundary: transcribe() accepts str path | np.ndarray | torch.Tensor; only the str path hits load_audio(), which shells out to ffmpeg and returns float32 mono 16 kHz scaled by /32768.
- Waveform→mel: log_mel_spectrogram computes torch.stft (N_FFT=400, HOP_LENGTH=160), squares magnitudes, applies a cached mel filterbank from assets/mel_filters.npz, then log10-clamps to [max-8, max] and rescales to [-1,1]-ish; transcribe() pads N_SAMPLES (480k) so window slicing stays in bounds.
- Mel→features: AudioEncoder conv1 (n_mels→n_state, k=3) + conv2 (stride 2, halving 3000 frames to n_audio_ctx=1500), adds sinusoidal positional_embedding, runs n_audio_layer residual attention blocks; DecodingTask._get_audio_features skips the encoder if the input is already (n_audio_ctx, n_audio_state).
- Tokens: get_encoding() builds a tiktoken.Encoding from assets/{gpt2,multilingual}.tiktoken plus specials — eot, sot, one token per language, translate/transcribe, sot_prev, sot_lm, nospeech, notimestamps, and 1501 timestamp tokens <|0.00|>…<|30.00|>.
- The SOT prefix per decode is sot_sequence (+ optional <|notimestamps|>) plus optional prompt tokens behind <|startofprev|> and optional prefix tokens, assembled in DecodingTask._get_initial_tokens.
- Per-step decode: _main_loop calls inference.logits() (feeding only the last token after the first pass thanks to KV-cache hooks), captures no_speech_prob at the sot index on step 0, applies each LogitFilter in place, then decoder.update() picks the next token until EOT or n_ctx.
- KV caching: Whisper.install_kv_cache_hooks registers forward hooks on every decoder MultiHeadAttention key/value module, appending outputs to a dict cache; BeamSearchDecoder calls inference.rearrange_kv_cache(source_indices) to keep cache aligned with surviving beams.
- Segment boundaries come from timestamp tokens: consecutive timestamp pairs slice tokens into segments with start/end = (token - timestamp_begin) * 0.02 s + window time_offset; a single trailing timestamp means silence for the remainder and fast-forwards the seek.
- Word alignment is a second decoder pass: find_alignment hooks each decoder block's cross_attn to capture QK matrices, runs model() under disable_sdpa, stacks weights at model.alignment_heads indices, median-filters, and runs DTW against token positions to produce WordTiming(word, tokens, start, end, probability).
- Result shape: each segment dict carries seek, start, end, text, tokens, temperature, avg_logprob, compression_ratio, no_speech_prob (+words when enabled); transcribe() returns {text: tokenizer.decode(all_tokens minus initial prompt), segments, language}.
- Output boundary: ResultWriter.__call__ writes `<audio basename>.<extension>` under output_dir in utf-8; subtitle writers (VTT/SRT) reflow words via iterate_result using get_start/get_end, long-pause (>3 s) breaks, and <u> highlighting.
- Prompt state: all_tokens grows with emitted tokens; decode_options['prompt'] is all_tokens[prompt_reset_since:] (or initial_prompt_tokens + tail when carry_initial_prompt), and prompt_reset_since is reset to len(all_tokens) when condition_on_previous_text is off or temperature exceeded 0.5.
Behavior traces
Three end-to-end flows were traced. (1) CLI startup: cli() → argparse → load_model → per-file transcribe → writer. (2) Model loading: name → _MODELS URL or local path → _download with SHA256 → torch.load → Whisper(dims) → set_alignment_heads. (3) Transcription: mel → language detect → per-window decode_with_fallback → DecodingTask.run (encoder → language → _main_loop → finalize → rank) → timestamp segmentation → optional word-level DTW alignment → result dict → file writers.
- CLI flow: whisper/__main__.py → transcribe.cli() → argparse (~30 args, valid_model_name validates against available_models() or file existence) → os.makedirs(output_dir) → .en-model language guard → temperature ladder built from --temperature_increment_on_fallback → load_model(model_name, device, download_root=model_dir) → get_writer(output_format, output_dir) → for each audio: transcribe(model, path, temperature=..., **args) then writer(result, audio_path, **writer_args).
- CLI resilience: each file's transcribe+write is wrapped in try/except that prints the traceback and `Skipping <path> due to ...`, so one bad file doesn't abort the batch.
- Model-load flow: load_model picks device (cuda if available) and download_root (~/.cache/whisper or $XDG_CACHE_HOME/whisper) → _download(url, root, in_memory) creates root, checks existing file's SHA256 (re-downloads on mismatch with a warning), streams via urllib with tqdm progress, verifies hash again post-download → torch.load(weights_only=True on torch>=1.13) → ModelDimensions(**checkpoint['dims']) → Whisper(dims) → load_state_dict → set_alignment_heads for official models.
- Transcribe flow (in order): dtype/fp16 resolution with CPU-fp16 downgrade warning → log_mel_spectrogram(audio, n_mels, padding=N_SAMPLES) → language detection via model.detect_language on first 30 s when unset → get_tokenizer → clip_timestamps → seek_clips pairs → while clip_idx < len(seek_clips): slice mel_segment, pad_or_trim, build prompt, decode_with_fallback → split tokens on consecutive timestamps into new_segment dicts → optional add_word_timestamps → append to all_segments/all_tokens → return dict.
- decode_with_fallback retries the window at each temperature in the ladder, dropping beam_size/patience when t>0 and best_of when t==0, and marks failure when compression_ratio > 2.4 or avg_logprob < -1.0 — unless no_speech_prob > 0.6 together with low logprob, which instead classifies the segment as silence.
- DecodingTask.run order: decoder.reset() → _get_audio_features (encoder forward, fp16 cast) → tokens = initial_tokens repeated per audio → _detect_language (optionally writes language token into sot_index+1, short-circuits for task='lang_id') → repeat_interleave by n_group (beam_size or best_of) → _main_loop → reshape (n_audio, n_group) → decoder.finalize → slice tokens between sample_begin and first EOT → sequence_ranker.rank → build DecodingResult list with avg_logprob, no_speech_prob, compression_ratio.
- Word-timestamp flow: add_word_timestamps → find_alignment (hooks on cross_attn capture QK, model() under disable_sdpa, weights over alignment_heads → median_filter → mean → dtw(-matrix) → split_to_word_tokens → WordTiming list) → sentence-boundary duration clamp → merge_punctuations → distribute words into segment['words'] with seek-derived time_offset and segment start/end reconciliation heuristics.
- Persistence flow: writer(result, audio_path) → ResultWriter.__call__ → output_path = output_dir/<basename>.<ext> → open utf-8 → write_result; format 'all' returns a closure invoking all six writers; subtitle writers produce `start --> end\ntext` blocks via iterate_result.
- Word-anomaly / hallucination-skip flow inside the seek loop: is_segment_anomaly scores words (prob<0.15, duration<0.133 s or >2 s) on the first 8 non-punctuation words; when hallucination_silence_threshold is set and an anomalous segment is surrounded by silence, the seek jumps past it and trailing current_segments are dropped.
Testing signals
Tests use pytest in tests/ (5 modules + conftest registering a `requires_cuda` marker and a `random` seed fixture). Coverage is good for the numeric utilities (dtw, median_filter vs scipy), normalizers (extensive assertions), and tokenizer edge cases, plus one end-to-end transcription test parametrized over all 13 model checkpoints against tests/jfk.flac. Major gaps: no coverage of the CLI, output writers, beam search, logit filters, model download, or the transcribe seek-loop heuristics. CI runs pre-commit (black/isort/flake8) and a 9-cell Python×PyTorch matrix on CPU-only torch wheels.
- Test framework is pytest; conftest.py registers the `requires_cuda` marker and provides a `random` fixture seeding both `random` and `numpy.random` to 42.
- test_transcribe parametrizes over every name in whisper.available_models() (13 checkpoints, downloaded on demand), transcribes tests/jfk.flac with word_timestamps=True, asserts 'my fellow americans' etc., tokenizer decode round-trip, and that the word 'Americans' brackets t≈1.8 s.
- test_audio verifies load_audio returns a 1-D waveform of ~10-12 s with std in (0,1) and that log_mel_spectrogram from array equals from-file, with mel dynamic range ≤ 2.0.
- test_timing checks dtw_cpu against a manually constructed optimal path, median_filter against scipy.ndimage.median_filter on 4 shapes × widths {3,5,7,13}, and CUDA equivalence for both kernels gated behind requires_cuda.
- test_normalizer exercises EnglishNumberNormalizer and EnglishTextNormalizer with ~70 assertions (ordinals, currency, 'double oh seven', 'and a half', contractions, titles) and EnglishSpellingNormalizer mappings.
- test_tokenizer covers sot_sequence membership, language-token/code parity, multilingual vs gpt2 encode/decode round-trip on Korean text, and split_tokens_on_unicode on a partial-UTF8 French sequence; note the `multilingual` parametrize value is ignored — the test always calls get_tokenizer(multilingual=False).
- CI (test.yml) runs a pre-commit job (black, isort, flake8, whitespace hooks) then a matrix over Python 3.8–3.13 × torch 1.10.1–2.5.1 with CPU wheels, conda-installed ffmpeg, invoking `pytest --durations=0 -vv -k 'not test_transcribe or test_transcribe[tiny] or test_transcribe[tiny.en]' -m 'not requires_cuda'`.
- Untested surface: cli() argument parsing and validation, all six ResultWriter formats, BeamSearchDecoder, every LogitFilter, _download SHA256 logic, and the hallucination-silence heuristics have no tests. (inferred)
- Dev extras (black, flake8, isort, pytest, scipy) are declared in pyproject [project.optional-dependencies].dev; scipy is only needed by test_timing's reference comparison.
Risk map
Highest-risk areas are (1) the `transcribe()` seek loop — a deliberately flattened, heuristic-heavy control flow the authors flag as obscure; (2) runtime code paths that execute or load external artifacts (ffmpeg subprocess, torch.load of checkpoint files, Triton JIT source patching); and (3) fragile fallbacks — numba JIT at import, the missing `regex` dependency, and silent error swallowing in the CLI. There is also a real latent bug in `_get_audio_features` returning a TypeError object instead of raising it.
- load_audio passes the user-supplied file path to an ffmpeg subprocess (list-form argv, no shell injection, but any local path incl. URLs/devices reaches ffmpeg); a CalledProcessError surfaces as RuntimeError with stderr.
- Checkpoint loading calls torch.load(weights_only=True) only when torch>=1.13; on older versions any local checkpoint path passed to load_model gets full pickle deserialization — arbitrary code execution risk for untrusted .pt files.
- Model download integrity relies on the SHA256 embedded in the URL path segment, verified before reuse and after download — but the URL itself is plain CDN state; a bad cached file triggers re-download, not a hard failure.
- The transcribe() main loop is acknowledged-fragile: an in-code comment says 'This loop is obscurely flattened to make the diff readable', and the seek is mutated by ~6 different mechanisms (segment_size fast-forward, last_timestamp seek, word-end seek, anomaly skips), making regressions easy and coverage nonexistent.
- DecodingTask._get_audio_features returns `TypeError(...)` as a value instead of raising it — a dead-branch bug where a dtype mismatch would propagate a TypeError object as audio_features.
- triton_ops generates the median-filter kernel by string-replacing LOAD_ALL_ROWS_HERE/BUBBLESORT_HERE placeholders inside a @triton.jit function's source, then mutating kernel.src/_unsafe_update_src — brittle across Triton versions; timing.py wraps calls in try/except (RuntimeError, CalledProcessError) with slow-path fallback.
- normalizers/basic.py imports the third-party `regex` package, which is absent from both pyproject dependencies and requirements.txt — importing whisper.normalizers.basic fails on a clean install unless regex arrives transitively.
- numba is a hard dependency imported at module top of timing.py and JIT-compiles dtw_cpu/backtrace at first call — first-call latency plus platform fragility for a function used only when word_timestamps is on.
- The CLI catches all exceptions per audio file and merely prints 'Skipping ...' — a fully-failed batch still exits 0, which can silently produce empty output directories in scripted pipelines.
- Default suppress_tokens='-1' expands to tokenizer.non_speech_tokens (music notes, brackets, symbols) plus all special/task tokens — a hard-coded suppression list that can mask legitimately-decoded characters, and any bug there degrades output quality rather than erroring.
- ApplyTimestampRules enforces timestamp/text alternation and monotonicity by in-place -inf masking per batch row; combined with the 'timestamp probability beats best text token' rule, subtle logic errors would produce shifted or looping timestamps rather than failures. (inferred)
- get_tokenizer/get_encoding/median_kernel are lru_cache'd singletons, so Tokenizer instances are shared process-wide; they are effectively immutable in practice, but any future mutability would leak state across calls. (inferred)
Extension points
Extension is mostly class-substitution rather than plugin registration: the get_writer() dict is a mini-registry for output formats; TokenDecoder, LogitFilter, SequenceRanker, and Inference are abstract seams inside DecodingTask; new checkpoints plug in via _MODELS/_ALIGNMENT_HEADS or by path; DecodingOptions is the frozen config dataclass every knob flows through. There is no dynamic plugin discovery — all seams require editing source.
- Output formats: subclass ResultWriter (set `extension`, implement write_result) and add it to the `writers` dict in get_writer() — the CLI's --output_format choices list in transcribe.py must also be updated.
- Search strategies: implement TokenDecoder (reset/update/finalize) — GreedyDecoder and BeamSearchDecoder are the built-ins — and select it in DecodingTask.__init__ where `options.beam_size is not None` chooses between them.
- Logit-level policies: implement LogitFilter.apply(logits, tokens) and append to DecodingTask.logit_filters, currently populated with SuppressBlank, SuppressTokens, and ApplyTimestampRules.
- Reranking: SequenceRanker.rank(tokens, sum_logprobs) → indices; MaximumLikelihoodRanker (length-normalized or Google-NMT penalty) is the only implementation, assigned in DecodingTask.__init__.
- Inference backend: Inference defines logits() / rearrange_kv_cache() / cleanup_caching(); PyTorchInference implements it via KV-cache forward hooks — an alternative (e.g. compiled or batched backend) would subclass Inference and be wired in DecodingTask.__init__.
- New official models: add a CDN URL to _MODELS and a base85 gzip'd (n_layers × n_heads) mask to _ALIGNMENT_HEADS in __init__.py; unofficial/fine-tuned checkpoints work without changes by passing a file path to load_model.
- DecodingOptions is the single frozen config dataclass for a decode (task, language, temperature, beam/best_of, prompt/prefix, suppress_tokens, timestamps, fp16); transcribe() forwards **decode_options straight into DecodingOptions(**kwargs), so new CLI/API knobs route through it.
- Languages are table-driven via LANGUAGES and TO_LANGUAGE_CODE, but special tokens are baked into the .tiktoken vocab — new languages can't be added without retraining; only aliases are cheap to extend.
- Accelerator backends: timing.py dispatches CUDA vs CPU implementations by trying the triton path inside try/except — the established pattern for adding a new kernel backend to median_filter and dtw.
- Normalizers are composable: EnglishTextNormalizer wires an ignore_patterns regex, a replacers dict, EnglishNumberNormalizer, and EnglishSpellingNormalizer(json map) — a new locale would follow the same class shape and export through normalizers/__init__.py.
Contributor handoff
Small, flat, readable codebase (~3.7k LOC of Python). Before editing, read whisper/transcribe.py (orchestrator+CLI), whisper/decoding.py (engine), and whisper/model.py (nn defs). Style is enforced by pre-commit (black/isort/flake8) in CI. Tests need ffmpeg and network for model downloads; the fastest local check is `pytest -k 'not test_transcribe'`. Known traps: a return-instead-of-raise bug in _get_audio_features, an undeclared `regex` dependency, Whisper binding free functions as class attributes, and the deliberately convoluted seek loop in transcribe().
- Read-first files: transcribe.py (623 lines, orchestration + cli()), decoding.py (826 lines, DecodingTask/DecodingOptions/decoders/filters), model.py (352 lines, Whisper/AudioEncoder/TextDecoder), __init__.py (load_model + _MODELS), tokenizer.py (vocab + specials).
- Formatting is enforced: black + isort (profile=black, line_length=88, trailing comma, multi_line_output=3) + flake8 (--max-line-length 88, ignore E203,E501,W503,W504) via .pre-commit-config.yaml, run as the gating CI job.
- Test invocation: `pip install .[dev]` (pulls pytest+scipy) then `pytest`; ffmpeg must be on PATH; test_transcribe downloads checkpoints (~GBs) so `pytest -k 'not test_transcribe'` mirrors most of CI's fast path; CUDA tests are behind the requires_cuda marker.
- Releases: bump whisper/version.py __version__ (date-based, e.g. "20250625") and push a commit titled 'Release <version>' to main — python-publish.yml regex-matches the message, tags v<version>, builds sdist, and uploads to PyPI with TWINE secrets.
- Trap: Whisper binds transcribe/decode/detect_language as class attributes pointing at module-level functions — renaming those functions or changing their signatures breaks the model.method() API used in README examples.
- Trap: DecodingTask._verify_options rejects incompatible combos (beam_size+best_of, best_of at T=0, patience without beam_size, length_penalty outside [0,1]) — new CLI flags must respect these invariants.
- Trap: tokenizer.decode() silently drops all tokens >= timestamp_begin; use decode_with_timestamps() (renders <|1.08|>) when debugging timestamp/token issues.
- Trap: fp16 is default-on but unsupported on CPU — transcribe() downgrades to fp32 with a warning, and DecodingOptions.fp16 controls the mel cast in _get_audio_features; keep both consistent when changing dtype logic.
- Trap: binary assets (assets/*.tiktoken, assets/mel_filters.npz, normalizers/english.json) load via os.path.dirname(__file__) — restructuring the package requires updating MANIFEST.in and these loaders together.
- Safe-edit boundaries: utils.py writers and normalizers/ are low-risk leaf modules; audio.py constants (SAMPLE_RATE=16000, N_FRAMES=3000, hop=160) are load-bearing for timestamp math everywhere; the transcribe() seek loop and timing.py heuristics are the blast-radius hotspots. (inferred)
Languages
| Language | Files | Share of source |
|---|---|---|
| Python | 20 | 95% |
| Markdown | 4 | docs/config |
| YAML | 4 | docs/config |
| JSON | 2 | docs/config |
| TOML | 1 | 5% |
Start reading here
Probable source entrypoint: pyproject.toml — inferred from the directory tree and manifest, not the upstream docs.
Source roots
whisper/ (18 files)
Source layout — whisper/
| Path | Files |
|---|---|
whisper/assets/ | 3 |
whisper/normalizers/ | 4 |
whisper/__init__.py | 1 |
whisper/__main__.py | 1 |
whisper/audio.py | 1 |
whisper/decoding.py | 1 |
whisper/model.py | 1 |
whisper/timing.py | 1 |
whisper/tokenizer.py | 1 |
whisper/transcribe.py | 1 |
whisper/triton_ops.py | 1 |
whisper/utils.py | 1 |
whisper/version.py | 1 |
Repository structure
| Top-level path | Files |
|---|---|
whisper/ | 18 |
tests/ | 6 |
.github/ | 3 |
data/ | 2 |
notebooks/ | 1 |
Docs found: README.md
Package
pyproject.toml — false v{ attr = "whisper.version.__version__" }, 0 declared dependencies.
CI & config entrypoints
README.md— Repository readme.github/workflows/python-publish.yml— GitHub Actions workflow.github/workflows/test.yml— GitHub Actions workflow
Structural health signals
Average score 7.5/10 across 14 analyzed source files; 3 hotspots surfaced.
whisper/decoding.py— 826 lines, score 6.2 (Large file, Deep nesting)whisper/transcribe.py— 623 lines, score 6.2 (Large file, Deep nesting)whisper/normalizers/english.py— 552 lines, score 6.2 (Large file, Deep nesting)
These are deterministic review leads — size, churn, and structural signals — not proof of a bug in this project.
Recent history
86098128c0b42026-08-31 — Only spell out a standalone 1 in the English number normalizer (#2836)5f86d1d863632026-07-28 — Fix SDPA cross-attention falling back to the math kernel during beam search (#2812)90fdc51112dc2026-07-29 — Add JSONL output format (one JSON object per line per segment) (#2815)04f449b8a4372026-04-15 — Pin pre-commit hook revisions to immutable commits (#2760)cba3768142a22026-03-27 — Pin GitHub Actions workflow references (#2751)