CodeVetter · Repo Unpack · Feature catalog
deepseek-ai/DeepSeek-V3
User-facing surface is deliberately small: a torchrun-launched CLI supporting interactive multi-turn chat and batch file inference, two checkpoint-conversion utilities, and a set of compile-time implementation switches (attention impl, GEMM impl, gate score function, quantization scale format). Config presets cover 16B/236B/671B model sizes plus the V3.1 variant with ue8m0 scale format. The README documents the model itself (benchmarks, weights layout, deployment ecosystem) more than the code, which is positioned as a demo.
- Interactive chat mode (--interactive) presents a '>>> ' prompt loop with two built-in slash commands: /exit to quit and /clear to reset message history.
- Batch mode (--input-file FILE) reads one prompt per line, applies the chat template to each as a single user message, and prints Prompt/Completion pairs; prompt count is asserted <= args.max_batch_size.
- Sampling supports temperature-scaled multinomial sampling (implemented as argmax of probs/exponential(1), i.e. the Gumbel-max trick) and greedy decoding when temperature <= 0 via logits.argmax.
- Multi-turn context is maintained in a messages list and rendered each turn by tokenizer.apply_chat_template(messages, add_generation_prompt=True).
- convert.py converts a Hugging Face checkpoint directory into per-rank demo shards, with required CLI args --hf-ckpt-path, --save-path, --n-experts, --model-parallel.
- fp8_cast_bf16.py converts an FP8 HF checkpoint to BF16 by dequantizing every 1-byte weight with its *_scale_inv tensor and rewriting model.safetensors.index.json.
- Two attention implementations are selectable via the module-level attn_impl global: 'naive' (materializes per-head k/v caches) and 'absorb' (default; caches latent kv + rope pe vectors).
- Two GEMM paths are selectable via the gemm_impl global: 'bf16' dequantizes FP8 weights then calls F.linear; 'fp8' quantizes activations with act_quant and runs fp8_gemm.
- MoE gating supports grouped top-k expert routing with either softmax or sigmoid scoring (score_func), an optional learned bias for auxiliary-loss-free balancing, and route_scale weight scaling.
- YaRN-style RoPE extension is implemented in precompute_freqs_cis using original_seq_len, rope_factor, beta_fast/beta_slow correction ranges, and an mscale-adjusted softmax_scale for long context.
- Four shipped config presets parameterize the same code for different models: config_16B (DeepSeek-V2-Lite-like, q_lora_rank 0, softmax gate), config_236B, config_671B (61 layers, 256 experts, sigmoid gate, fp8), and config_v3.1 (adds scale_fmt ue8m0).
- model.py doubles as a smoke-test script: its __main__ block builds a default ModelArgs Transformer on CUDA and prints the output logits shape.
- The Multi-Token Prediction (MTP) module weights documented in README_WEIGHTS.md (layer 61: enorm, hnorm, eh_proj, shared_head) are not supported by the demo — convert.py explicitly skips any tensor whose name contains 'model.layers.61'.
- Non-rank-0 ranks participate in interactive chat by receiving the prompt via dist.broadcast_object_list, so all ranks stay synchronized through the input loop.