# Data flow — openai/whisper

> Canonical page: https://codevetter.com/unpack/whisper/data-flow

# Data flow — openai/whisper

Scanned commit 86098128c0b4f24f0e2aa2994de830614b474227 (2026-08-31). Generated by codevetter codevetter 1.12.0 on 2026-09-22.

Audio enters as a file path (decoded by an ffmpeg subprocess to 16 kHz mono float32) or a NumPy/Torch waveform, then becomes a log-Mel spectrogram (80 or 128 bins, STFT n_fft=400 hop=160) padded with 30 s of silence. The mel tensor feeds AudioEncoder once per 30 s window; DecodingTask autoregressively produces tokens under logit filters and a KV cache, with timestamp tokens (>=tokenizer.timestamp_begin, 0.02 s each) delimiting segments. Segments accumulate in all_segments/all_tokens; optional word-level alignment runs a second decoder pass capturing cross-attention weights through forward hooks. The final dict {text, segments, language} returns to the caller and is serialized by ResultWriter subclasses into output_dir.
- Input boundary: transcribe() accepts str path | np.ndarray | torch.Tensor; only the str path hits load_audio(), which shells out to ffmpeg and returns float32 mono 16 kHz scaled by /32768. — [whisper/transcribe.py#L38-40](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/transcribe.py#L38-L40) [whisper/audio.py#L25-62](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/audio.py#L25-L62) [whisper/audio.py#L138-141](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/audio.py#L138-L141)
- Waveform→mel: log_mel_spectrogram computes torch.stft (N_FFT=400, HOP_LENGTH=160), squares magnitudes, applies a cached mel filterbank from assets/mel_filters.npz, then log10-clamps to [max-8, max] and rescales to [-1,1]-ish; transcribe() pads N_SAMPLES (480k) so window slicing stays in bounds. — [whisper/audio.py#L91-157](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/audio.py#L91-L157) [whisper/transcribe.py#L138-141](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/transcribe.py#L138-L141)
- Mel→features: AudioEncoder conv1 (n_mels→n_state, k=3) + conv2 (stride 2, halving 3000 frames to n_audio_ctx=1500), adds sinusoidal positional_embedding, runs n_audio_layer residual attention blocks; DecodingTask._get_audio_features skips the encoder if the input is already (n_audio_ctx, n_audio_state). — [whisper/model.py#L181-211](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/model.py#L181-L211) [whisper/decoding.py#L644-664](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/decoding.py#L644-L664)
- Tokens: get_encoding() builds a tiktoken.Encoding from assets/{gpt2,multilingual}.tiktoken plus specials — eot, sot, one token per language, translate/transcribe, sot_prev, sot_lm, nospeech, notimestamps, and 1501 timestamp tokens <|0.00|>…<|30.00|>. — [whisper/tokenizer.py#L330-363](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/tokenizer.py#L330-L363)
- The SOT prefix per decode is sot_sequence (+ optional <|notimestamps|>) plus optional prompt tokens behind <|startofprev|> and optional prefix tokens, assembled in DecodingTask._get_initial_tokens. — [whisper/decoding.py#L587-613](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/decoding.py#L587-L613) [whisper/tokenizer.py#L151-159](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/tokenizer.py#L151-L159)
- Per-step decode: _main_loop calls inference.logits() (feeding only the last token after the first pass thanks to KV-cache hooks), captures no_speech_prob at the sot index on step 0, applies each LogitFilter in place, then decoder.update() picks the next token until EOT or n_ctx. — [whisper/decoding.py#L680-710](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/decoding.py#L680-L710) [whisper/decoding.py#L155-163](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/decoding.py#L155-L163)
- KV caching: Whisper.install_kv_cache_hooks registers forward hooks on every decoder MultiHeadAttention key/value module, appending outputs to a dict cache; BeamSearchDecoder calls inference.rearrange_kv_cache(source_indices) to keep cache aligned with surviving beams. — [whisper/model.py#L317-348](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/model.py#L317-L348) [whisper/decoding.py#L144-177](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/decoding.py#L144-L177) [whisper/decoding.py#L364-365](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/decoding.py#L364-L365)
- Segment boundaries come from timestamp tokens: consecutive timestamp pairs slice tokens into segments with start/end = (token - timestamp_begin) * 0.02 s + window time_offset; a single trailing timestamp means silence for the remainder and fast-forwards the seek. — [whisper/transcribe.py#L339-399](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/transcribe.py#L339-L399) [whisper/audio.py#L20-22](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/audio.py#L20-L22)
- Word alignment is a second decoder pass: find_alignment hooks each decoder block's cross_attn to capture QK matrices, runs model() under disable_sdpa, stacks weights at model.alignment_heads indices, median-filters, and runs DTW against token positions to produce WordTiming(word, tokens, start, end, probability). — [whisper/timing.py#L163-242](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/timing.py#L163-L242) [whisper/model.py#L285-292](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/model.py#L285-L292)
- Result shape: each segment dict carries seek, start, end, text, tokens, temperature, avg_logprob, compression_ratio, no_speech_prob (+words when enabled); transcribe() returns {text: tokenizer.decode(all_tokens minus initial prompt), segments, language}. — [whisper/transcribe.py#L246-261](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/transcribe.py#L246-L261) [whisper/transcribe.py#L510-514](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/transcribe.py#L510-L514)
- Output boundary: ResultWriter.__call__ writes `<audio basename>.<extension>` under output_dir in utf-8; subtitle writers (VTT/SRT) reflow words via iterate_result using get_start/get_end, long-pause (>3 s) breaks, and <u> highlighting. — [whisper/utils.py#L85-107](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/utils.py#L85-L107) [whisper/utils.py#L119-235](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/utils.py#L119-L235)
- Prompt state: all_tokens grows with emitted tokens; decode_options['prompt'] is all_tokens[prompt_reset_since:] (or initial_prompt_tokens + tail when carry_initial_prompt), and prompt_reset_since is reset to len(all_tokens) when condition_on_previous_text is off or temperature exceeded 0.5. — [whisper/transcribe.py#L288-293](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/transcribe.py#L288-L293) [whisper/transcribe.py#L499-505](https://github.com/openai/whisper/blob/86098128c0b4f24f0e2aa2994de830614b474227/whisper/transcribe.py#L499-L505)

Every claim links to its cited files at the pinned commit. Citations identify source evidence; they do not establish runtime correctness. Independent CodeVetter scan — not reviewed by upstream maintainers.

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)