CodeVetter · Repo Unpack · Contributor handoff
deepseek-ai/DeepSeek-V3
Small codebase, high blast radius: five Python files where every edit is near the numerics. Read README.md section 6 first for the intended workflow, then generate.py (orchestration), model.py (all architecture), kernel.py (Triton numerics), and convert.py (weight naming contract). Validation is manual — there are no tests — so the smoke path is `python model.py` (needs CUDA) or a short torchrun generate. Known traps: module globals set only in Transformer.__init__, the dim==7168 gate-bias hack, silent layer-61 drops in convert.py, assert-based validation, and the requirement to run everything from inside inference/ with per-rank shard naming.
- Read order for onboarding: README.md (workflow + ecosystem) -> inference/generate.py (CLI + distributed setup) -> inference/model.py (ModelArgs + all modules) -> inference/kernel.py (FP8 kernels) -> inference/convert.py (weight-name contract) -> README_WEIGHTS.md (checkpoint layout).
- Environment setup is exactly: cd inference && pip install -r requirements.txt (torch 2.4.1, triton 3.0.0, transformers 4.46.3, safetensors 0.4.5); Linux + Python 3.10 + CUDA only — Triton kernels and torch.set_default_device('cuda') mean no CPU/macOS path.
- Canonical run commands: convert with 'python convert.py --hf-ckpt-path <hf> --save-path <out> --n-experts 256 --model-parallel 16', then generate with 'torchrun --nnodes N --nproc-per-node 8 generate.py --ckpt-path <out> --config configs/config_671B.json --interactive'.
- Checkpoint shard naming is a strict contract between convert.py and generate.py: files must be named model{rank}-mp{world_size}.safetensors in --ckpt-path; a mismatch in world_size between conversion and torchrun silently loads nothing-compatible.
- Always run scripts from inside inference/ — imports are bare ('from kernel import ...', 'from model import ...') with no package structure.
- Trap: Gate.bias exists iff args.dim == 7168. When adding a config or changing dim, also check whether the checkpoint has e_score_correction_bias tensors, or load_model will fail on unexpected keys.
- Trap: world_size/rank globals and Linear.dtype are only assigned inside Transformer.__init__; never instantiate Block/MLA/Linear standalone expecting distributed or fp8 behavior.
- Trap: convert.py drops all 'model.layers.61' tensors (MTP module). If asked to support MTP/speculative decoding, this skip plus the missing eh_proj/enorm/hnorm mapping entries are the first things to change.
- Smoke-test procedure after edits: 'python model.py' should print a logits shape (requires CUDA); for a real check, convert a small config (config_16B.json) and run a 2-token generate — there is no automated test to run instead.
- Code conventions: every public class/function carries a docstring with Args/Returns; type hints use typing.Literal for mode switches; distributed code checks 'if world_size > 1' before every collective; keep new code consistent.
- Safe edit boundaries: kernel.py Triton code and Gate routing math are numerics-critical and untested — changes there need manual numerical comparison; generate.py CLI plumbing and convert.py key renames are lower-risk. (inferred)
- Do not commit model weights or generated shards — nothing in .gitignore excludes *.safetensors or checkpoint dirs, so a careless 'git add -A' can stage hundreds of GB.
- Do not change the pinned repo state expecting upstream compatibility: HF Transformers is explicitly unsupported, and the demo expects weights already converted to its shard format.