deepseek-ai/DeepSeek-V3
The repo is a small, flat reference codebase: all executable code lives under inference/ (five Python files plus a configs/ directory of JSON model presets). The primary entrypoint is generate.py, launched via torchrun for multi-node, tensor-parallel inference; convert.py and fp8_cast_bf16.py are standalone CLI utilities for checkpoint preparation. model.py contains the entire model definition (Transformer, Block, MLA, MoE, Gate, parallel linears) and kernel.py contains the Triton kernels for FP8 quantization and GEMM. There is no build system, package manifest, or test suite — the only CI is a GitHub Actions stale-issue bot. Weights are safetensors files on disk, sharded per rank by convert.py, and runtime state is GPU-resident KV-cache buffers.
- All runnable code lives in inference/: model.py (model definition, 808 lines), kernel.py (Triton kernels), generate.py (CLI entrypoint), convert.py (HF->demo checkpoint converter), fp8_cast_bf16.py (FP8->BF16 weight converter), plus configs/*.json presets.
- generate.py is the primary entrypoint, designed to be launched by torchrun; main() reads WORLD_SIZE, RANK, and LOCAL_RANK environment variables and calls dist.init_process_group('nccl') when world_size > 1.
- model.py defines the full architecture in one file: Transformer wraps ParallelEmbedding, a ModuleList of Block (MLA attention + MLP or MoE FFN), final RMSNorm, and a ColumnParallelLinear head.
- kernel.py provides three Triton kernels — act_quant_kernel (per-128-block activation quantization to float8_e4m3fn), weight_dequant_kernel (blockwise FP8->BF16 dequant), and an autotuned fp8_gemm_kernel accumulating in fp32.
- Checkpoints are stored as per-rank safetensors shards named model{rank}-mp{world_size}.safetensors, produced by convert.py and consumed by generate.py via safetensors.torch.load_model.
- Model hyperparameters are supplied as JSON files under inference/configs/ (config_16B.json, config_236B.json, config_671B.json, config_v3.1.json) deserialized directly into the ModelArgs dataclass in main().
- Dependencies are pinned in inference/requirements.txt: torch==2.4.1, triton==3.0.0, transformers==4.46.3, safetensors==0.4.5; the README states Linux + Python 3.10 only, with Mac and Windows unsupported.
- There is no package manifest (no setup.py/pyproject.toml) and imports are bare sibling imports ('from kernel import ...', 'from model import ...'), so scripts must be run from inside the inference/ directory.
- The only CI/CD automation is .github/workflows/stale.yml, a daily actions/stale@v9 job that marks issues stale after 30 days and closes them after 14 more; there is no build, lint, or test workflow.
- Runtime state is GPU-resident: MLA registers non-persistent buffers kv_cache/pe_cache (absorb impl) or k_cache/v_cache (naive impl) sized [max_batch_size, max_seq_len, ...] at construction.
- Tensor parallelism is implemented via ColumnParallelLinear/RowParallelLinear, ParallelEmbedding, and expert-sharded MoE, synchronized with dist.all_reduce / dist.all_gather at layer boundaries.
- The tokenizer is loaded from the checkpoint directory with transformers AutoTokenizer.from_pretrained(ckpt_path), so the converted output dir must contain HF tokenizer files (convert.py copies files matching '*token*').
- Production deployment is delegated to external frameworks; the README recommends SGLang, LMDeploy, TensorRT-LLM, vLLM, and LightLLM, and notes Hugging Face Transformers is not directly supported.
- Licensing is split: code under MIT (LICENSE-CODE) and model weights under a separate Model License (LICENSE-MODEL), with commercial use permitted.