# Data flow — deepseek-ai/DeepSeek-V3

> Canonical page: https://codevetter.com/unpack/deepseek-v3/data-flow

# Data flow — deepseek-ai/DeepSeek-V3

Scanned commit 9b4e9788e4a3a731f7567338ed15d3ec549ce03b (2025-08-28). Generated by codevetter codevetter 1.12.0 on 2026-09-22.

Data moves along three pipelines: (1) offline checkpoint conversion — HF safetensors are renamed via the convert.py mapping table, column/row-sharded across mp ranks, and experts are partitioned contiguously; (2) startup — JSON config -> ModelArgs -> Transformer construction -> safetensors load_model into each rank's shard; (3) inference — token IDs flow through vocab-parallel embedding, N Blocks of RMSNorm->MLA->residual and RMSNorm->(MLP|MoE)->residual, a final norm + head producing logits that are all-gathered across ranks, then sampled and appended back into the token buffer. FP8 support threads through as blockwise scale tensors attached to Linear weights.
- Config data flows: --config JSON -> json.load -> ModelArgs(**dict); every architecture knob (dim, n_layers, n_routed_experts, dtype, scale_fmt, rope params) is a ModelArgs field with defaults matching the 16B model. — [inference/generate.py#L112-113](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/generate.py#L112-L113) [inference/model.py#L19-86](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L19-L86) [inference/configs/config_16B.json](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/configs/config_16B.json)
- Weight-name conversion is driven by the convert.py `mapping` dict, which renames HF keys (e.g. self_attn.q_a_proj -> attn.wq_a, mlp.gate_proj -> ffn.w1, weight_scale_inv -> scale, e_score_correction_bias -> bias) and records the shard axis (0 or 1) per tensor type. — [inference/convert.py#L11-30](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/convert.py#L11-L30) [inference/convert.py#L56-65](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/convert.py#L56-L65)
- Expert sharding in convert.py assigns expert i to rank floor(i / n_local_experts); non-expert tensors with a shard axis are narrowed into mp contiguous slices, asserted divisible by mp. — [inference/convert.py#L66-76](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/convert.py#L66-L76)
- At startup each rank loads only its shard file model{rank}-mp{world_size}.safetensors after a 2-token warmup generate call that exercises the model before weights are loaded. — [inference/generate.py#L115-119](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/generate.py#L115-L119)
- Token flow: input IDs -> ParallelEmbedding (each rank embeds only its vocab slice, masks others to zero, then all_reduce) -> h [bsz, seqlen, dim]. — [inference/model.py#L107-128](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L107-L128) [inference/model.py#L784-785](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L784-L785)
- MLA forward computes q via wq or wq_a->q_norm->wq_b, splits q into q_nope/q_pe, applies RoPE to q_pe and to the k_pe slice of wkv_a output, then either writes k/v caches (naive) or caches normalized kv latent + pe (absorb). — [inference/model.py#L446-487](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L446-L487)
- Attention scores are softmaxed in float32 then cast back to the input dtype; the causal mask is an additive -inf upper-triangular tensor built only when seqlen > 1. — [inference/model.py#L488-490](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L488-L490) [inference/model.py#L787-789](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L787-L789)
- MoE forward flattens tokens to [tokens, dim], routes via Gate (scores -> optional group masking -> top-k indices -> gather original scores -> normalize -> route_scale), then loops over local experts scattering weighted outputs, adds shared_experts output, and all_reduces across ranks. — [inference/model.py#L669-693](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L669-L693) [inference/model.py#L566-598](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L566-L598)
- Final logits are produced by ColumnParallelLinear head and reassembled across ranks with dist.all_gather + concat along the vocab dimension. — [inference/model.py#L792-797](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L792-L797)
- The decode loop writes prompt tokens into a -1-padded buffer, iterates cur_pos from min(prompt_lens) to total_len calling model.forward on the slice [prev_pos:cur_pos], and forces prompt positions via prompt_mask so prefilled tokens override sampled ones. — [inference/generate.py#L51-71](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/generate.py#L51-L71)
- FP8 quantization data flow: act_quant produces (float8_e4m3fn tensor, fp32 scales) per 128-element block; fp8_gemm multiplies scaled blocks accumulating in fp32; weight scales live as a .scale Parameter on each Linear whose weight element_size() == 1. — [inference/kernel.py#L38-57](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/kernel.py#L38-L57) [inference/kernel.py#L118-172](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/kernel.py#L118-L172) [inference/model.py#L183-189](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L183-L189)
- fp8_cast_bf16 reads model.safetensors.index.json weight_map to locate tensors across shard files, keeps a 2-file LRU cache of loaded safetensors on cuda, dequantizes each FP8 weight with weight_dequant, and strips *_scale_inv entries from the output index. — [inference/fp8_cast_bf16.py#L34-103](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/fp8_cast_bf16.py#L34-L103)
- EOS handling: finished flags update only for generated positions (not prompt), generation breaks early when all sequences finish, and each completion is truncated at its first eos_id. — [inference/generate.py#L68-78](https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/generate.py#L68-L78)

Every claim links to its cited files at the pinned commit. Citations identify source evidence; they do not establish runtime correctness. Independent CodeVetter scan — not reviewed by upstream maintainers.

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)