CodeVetter · Repo Unpack · Behavior traces
deepseek-ai/DeepSeek-V3
Three flows were traced end-to-end. Startup: torchrun env -> process group -> config -> model construction -> warmup -> weight load. Interactive turn: input -> broadcast -> chat template -> generate loop -> decode -> history append. Per-decode-step forward: embed -> freqs_cis slice -> per-Block (attn_norm -> MLA -> residual; ffn_norm -> MLP/MoE -> residual) -> final norm -> head -> all-gather -> sample. Offline conversion runs glob -> safe_open -> rename -> shard/expert-partition -> save_file -> tokenizer copy.
- Startup trace: torchrun sets WORLD_SIZE/RANK/LOCAL_RANK -> main() reads them via os.getenv -> dist.init_process_group('nccl') if world_size>1 -> non-rank-0 rebinds print to a no-op -> torch.cuda.set_device(local_rank) -> set_default_dtype(bfloat16) -> ModelArgs loaded from --config JSON.
- Startup continues: Transformer(args) constructor sets globals world_size/rank from torch.distributed and Linear.dtype (float8_e4m3fn if args.dtype=='fp8' else bfloat16), then builds embed/layers/norm/head and precomputes freqs_cis.
- Startup finishes with a warmup: tokenizer loaded via AutoTokenizer, generate() called on the tokenized string 'DeepSeek' for 2 tokens, then safetensors load_model applies the rank's shard into the already-allocated module tree.
- Interactive turn trace: input('>>> ') on rank 0 -> dist.broadcast_object_list propagates to other ranks -> '/exit' breaks, '/clear' empties messages -> messages.append(user) -> apply_chat_template -> generate() -> tokenizer.decode(skip_special_tokens=True) -> print -> messages.append(assistant).
- Decode-step trace inside generate(): for each cur_pos, model.forward(tokens[:, prev_pos:cur_pos], prev_pos) -> Transformer.forward embeds tokens, slices freqs_cis[start_pos:start_pos+seqlen], builds causal mask if seqlen>1, runs every Block, applies final norm on the last position only (h[:, -1]), projects to logits, all_gathers across ranks.
- MLA 'absorb' trace: wq_a -> q_norm -> wq_b -> view to heads -> split q_nope/q_pe -> apply_rotary_emb(q_pe) -> wkv_a -> split kv/k_pe -> apply_rotary_emb(k_pe) -> kv_norm(kv) written to kv_cache, k_pe to pe_cache -> scores = einsum(q_nope,kv_cache)+einsum(q_pe,pe_cache) -> softmax fp32 -> einsum with kv_cache then wkv_b value slice -> wo.
- Gate trace: linear(x, gate.weight) -> softmax-or-sigmoid scores -> add bias if present -> if n_groups>1, compute group scores (amax without bias, topk(2) sum with bias), keep top topk_groups groups via masked_fill(-inf) -> topk(n_activated_experts) indices -> gather weights from pre-bias scores -> sigmoid weights normalized -> *= route_scale.
- MoE trace: gate(x) -> bincount per-expert counts -> for each local expert index with count>0, torch.where(indices==i) selects tokens and y[idx] += expert(x[idx]) * weight -> shared_experts MLP on all tokens -> dist.all_reduce(y) -> return (y+z).view(shape).
- Conversion trace: glob(hf_ckpt_path/*.safetensors) -> safe_open each file -> per tensor: skip 'model.layers.61', strip 'model.' prefix, apply name replacements, look up (new_key, shard_dim) in mapping -> for each rank i: filter experts to that rank's contiguous range or narrow the tensor along shard_dim -> save_file per rank -> copy *token* files.
- fp8->bf16 trace: load index.json weight_map -> sort *.safetensors -> per file: load to cuda, drop *_scale_inv entries, weight_dequant each 1-byte tensor with its scale_inv (warning + passthrough on KeyError) -> save_file -> prune scale_inv keys from weight_map -> write new index.json.