CodeVetter · Repo Unpack · Data flow
openai/whisper
Audio enters as a file path (decoded by an ffmpeg subprocess to 16 kHz mono float32) or a NumPy/Torch waveform, then becomes a log-Mel spectrogram (80 or 128 bins, STFT n_fft=400 hop=160) padded with 30 s of silence. The mel tensor feeds AudioEncoder once per 30 s window; DecodingTask autoregressively produces tokens under logit filters and a KV cache, with timestamp tokens (>=tokenizer.timestamp_begin, 0.02 s each) delimiting segments. Segments accumulate in all_segments/all_tokens; optional word-level alignment runs a second decoder pass capturing cross-attention weights through forward hooks. The final dict {text, segments, language} returns to the caller and is serialized by ResultWriter subclasses into output_dir.
- Input boundary: transcribe() accepts str path | np.ndarray | torch.Tensor; only the str path hits load_audio(), which shells out to ffmpeg and returns float32 mono 16 kHz scaled by /32768.
- Waveform→mel: log_mel_spectrogram computes torch.stft (N_FFT=400, HOP_LENGTH=160), squares magnitudes, applies a cached mel filterbank from assets/mel_filters.npz, then log10-clamps to [max-8, max] and rescales to [-1,1]-ish; transcribe() pads N_SAMPLES (480k) so window slicing stays in bounds.
- Mel→features: AudioEncoder conv1 (n_mels→n_state, k=3) + conv2 (stride 2, halving 3000 frames to n_audio_ctx=1500), adds sinusoidal positional_embedding, runs n_audio_layer residual attention blocks; DecodingTask._get_audio_features skips the encoder if the input is already (n_audio_ctx, n_audio_state).
- Tokens: get_encoding() builds a tiktoken.Encoding from assets/{gpt2,multilingual}.tiktoken plus specials — eot, sot, one token per language, translate/transcribe, sot_prev, sot_lm, nospeech, notimestamps, and 1501 timestamp tokens <|0.00|>…<|30.00|>.
- The SOT prefix per decode is sot_sequence (+ optional <|notimestamps|>) plus optional prompt tokens behind <|startofprev|> and optional prefix tokens, assembled in DecodingTask._get_initial_tokens.
- Per-step decode: _main_loop calls inference.logits() (feeding only the last token after the first pass thanks to KV-cache hooks), captures no_speech_prob at the sot index on step 0, applies each LogitFilter in place, then decoder.update() picks the next token until EOT or n_ctx.
- KV caching: Whisper.install_kv_cache_hooks registers forward hooks on every decoder MultiHeadAttention key/value module, appending outputs to a dict cache; BeamSearchDecoder calls inference.rearrange_kv_cache(source_indices) to keep cache aligned with surviving beams.
- Segment boundaries come from timestamp tokens: consecutive timestamp pairs slice tokens into segments with start/end = (token - timestamp_begin) * 0.02 s + window time_offset; a single trailing timestamp means silence for the remainder and fast-forwards the seek.
- Word alignment is a second decoder pass: find_alignment hooks each decoder block's cross_attn to capture QK matrices, runs model() under disable_sdpa, stacks weights at model.alignment_heads indices, median-filters, and runs DTW against token positions to produce WordTiming(word, tokens, start, end, probability).
- Result shape: each segment dict carries seek, start, end, text, tokens, temperature, avg_logprob, compression_ratio, no_speech_prob (+words when enabled); transcribe() returns {text: tokenizer.decode(all_tokens minus initial prompt), segments, language}.
- Output boundary: ResultWriter.__call__ writes `<audio basename>.<extension>` under output_dir in utf-8; subtitle writers (VTT/SRT) reflow words via iterate_result using get_start/get_end, long-pause (>3 s) breaks, and <u> highlighting.
- Prompt state: all_tokens grows with emitted tokens; decode_options['prompt'] is all_tokens[prompt_reset_since:] (or initial_prompt_tokens + tail when carry_initial_prompt), and prompt_reset_since is reset to len(all_tokens) when condition_on_previous_text is off or temperature exceeded 0.5.