CodeVetter · Repo Unpack · Behavior traces
openai/whisper
Three end-to-end flows were traced. (1) CLI startup: cli() → argparse → load_model → per-file transcribe → writer. (2) Model loading: name → _MODELS URL or local path → _download with SHA256 → torch.load → Whisper(dims) → set_alignment_heads. (3) Transcription: mel → language detect → per-window decode_with_fallback → DecodingTask.run (encoder → language → _main_loop → finalize → rank) → timestamp segmentation → optional word-level DTW alignment → result dict → file writers.
- CLI flow: whisper/__main__.py → transcribe.cli() → argparse (~30 args, valid_model_name validates against available_models() or file existence) → os.makedirs(output_dir) → .en-model language guard → temperature ladder built from --temperature_increment_on_fallback → load_model(model_name, device, download_root=model_dir) → get_writer(output_format, output_dir) → for each audio: transcribe(model, path, temperature=..., **args) then writer(result, audio_path, **writer_args).
- CLI resilience: each file's transcribe+write is wrapped in try/except that prints the traceback and `Skipping <path> due to ...`, so one bad file doesn't abort the batch.
- Model-load flow: load_model picks device (cuda if available) and download_root (~/.cache/whisper or $XDG_CACHE_HOME/whisper) → _download(url, root, in_memory) creates root, checks existing file's SHA256 (re-downloads on mismatch with a warning), streams via urllib with tqdm progress, verifies hash again post-download → torch.load(weights_only=True on torch>=1.13) → ModelDimensions(**checkpoint['dims']) → Whisper(dims) → load_state_dict → set_alignment_heads for official models.
- Transcribe flow (in order): dtype/fp16 resolution with CPU-fp16 downgrade warning → log_mel_spectrogram(audio, n_mels, padding=N_SAMPLES) → language detection via model.detect_language on first 30 s when unset → get_tokenizer → clip_timestamps → seek_clips pairs → while clip_idx < len(seek_clips): slice mel_segment, pad_or_trim, build prompt, decode_with_fallback → split tokens on consecutive timestamps into new_segment dicts → optional add_word_timestamps → append to all_segments/all_tokens → return dict.
- decode_with_fallback retries the window at each temperature in the ladder, dropping beam_size/patience when t>0 and best_of when t==0, and marks failure when compression_ratio > 2.4 or avg_logprob < -1.0 — unless no_speech_prob > 0.6 together with low logprob, which instead classifies the segment as silence.
- DecodingTask.run order: decoder.reset() → _get_audio_features (encoder forward, fp16 cast) → tokens = initial_tokens repeated per audio → _detect_language (optionally writes language token into sot_index+1, short-circuits for task='lang_id') → repeat_interleave by n_group (beam_size or best_of) → _main_loop → reshape (n_audio, n_group) → decoder.finalize → slice tokens between sample_begin and first EOT → sequence_ranker.rank → build DecodingResult list with avg_logprob, no_speech_prob, compression_ratio.
- Word-timestamp flow: add_word_timestamps → find_alignment (hooks on cross_attn capture QK, model() under disable_sdpa, weights over alignment_heads → median_filter → mean → dtw(-matrix) → split_to_word_tokens → WordTiming list) → sentence-boundary duration clamp → merge_punctuations → distribute words into segment['words'] with seek-derived time_offset and segment start/end reconciliation heuristics.
- Persistence flow: writer(result, audio_path) → ResultWriter.__call__ → output_path = output_dir/<basename>.<ext> → open utf-8 → write_result; format 'all' returns a closure invoking all six writers; subtitle writers produce `start --> end\ntext` blocks via iterate_result.
- Word-anomaly / hallucination-skip flow inside the seek loop: is_segment_anomaly scores words (prob<0.15, duration<0.133 s or >2 s) on the first 8 non-punctuation words; when hallucination_silence_threshold is set and an anomalous segment is surrounded by silence, the seek jumps past it and trailing current_segments are dropped.