CodeVetter · Repo Unpack · Feature catalog
openai/whisper
User-facing surface is (a) the `whisper` CLI, (b) the Python API (load_model, transcribe, decode, detect_language), and (c) six output formats. Core capabilities: multilingual transcription across 99 languages, X→English translation, automatic language detection, word-level timestamps via attention-based DTW, clip selection, prompt conditioning, and two search strategies (greedy/best-of-N vs beam search). Auxiliary features include text normalizers for evaluation and a hallucination-silence skip heuristic.
- The CLI accepts one or more audio paths plus ~30 flags including --model (default `turbo`), --task, --language, --output_dir/-o, --output_format/-f (txt|vtt|srt|tsv|json|jsonl|all), --verbose, --fp16, and --threads.
- Python API: `whisper.load_model(name)` returns a Whisper on cuda-if-available; `model.transcribe(path)` returns {text, segments, language}; lower-level `whisper.decode(model, mel, options)` and `whisper.detect_language(model, mel)` are exported.
- Thirteen model names are available — tiny/base/small/medium each with a `.en` English-only variant, large-v1/v2/v3, `large` (alias of large-v3), and large-v3-turbo/`turbo` — all mapped to CDN URLs in _MODELS.
- Tasks: `transcribe` (same-language ASR) and `translate` (X→English) are selected via --task and encoded as <|transcribe|>/<|translate|> special tokens in the SOT sequence; `lang_id` is a third internal task that short-circuits after language detection.
- Automatic language detection: when --language is omitted on a multilingual model, transcribe() runs model.detect_language() on the first 30 s mel segment and picks argmax over per-language probabilities.
- Six output writers exist — WriteTXT, WriteVTT, WriteSRT, WriteTSV, WriteJSON, WriteJSONL — selected by name through the get_writer() registry; `all` instantiates every writer.
- Word-level timestamps (--word_timestamps) align tokens to audio via cross-attention weights + dynamic time warping; supporting flags --highlight_words (underline in srt/vtt), --max_line_width, --max_line_count, --max_words_per_line control subtitle reflow in SubtitlesWriter.iterate_result.
- Search strategies: temperature==0 uses BeamSearchDecoder (--beam_size, --patience, --length_penalty); temperature>0 uses GreedyDecoder with sampling and --best_of candidates; MaximumLikelihoodRanker picks the final hypothesis.
- Fallback ladder: transcribe() accepts a temperature tuple (CLI builds it via --temperature_increment_on_fallback) and retries each 30 s window at higher temperatures when compression_ratio_threshold or logprob_threshold fails; no_speech_threshold can instead mark the window silent.
- --clip_timestamps accepts comma-separated start,end pairs (seconds) restricting transcription to those ranges; --hallucination_silence_threshold skips long silences around anomalous word segments when word_timestamps is on.
- Prompt conditioning: --initial_prompt seeds the first window, --carry_initial_prompt prepends it to every window, and --condition_on_previous_text feeds prior output tokens as prompt for the next window.
- Language support covers 99 languages enumerated in LANGUAGES (tokenized as <|en|>…<|yue|> specials) plus alias resolution in TO_LANGUAGE_CODE (e.g. burmese→my, mandarin→zh).
- Evaluation normalizers: BasicTextNormalizer (lowercase, strip bracketed content, NFKD symbol/diacritic removal) and EnglishTextNormalizer (contractions, number words→digits via EnglishNumberNormalizer, UK→US spelling via EnglishSpellingNormalizer) for WER scoring.