CodeVetter · Repo Unpack · Risk map
deepseek-ai/DeepSeek-V3
The codebase is a demo, and its risks match: correctness depends on runtime asserts (stripped under -O), module-level globals that must be set before model construction, a hardcoded dim==7168 special case in Gate, and silent behaviors like dropping all layer-61 (MTP) weights during conversion. The distributed path uses pickle-based broadcast_object_list and suppresses stdout on non-zero ranks. Memory is pre-allocated aggressively (full-length KV caches). There is no input validation on paths and no test coverage anywhere.
- Gate creates its correction bias only when self.dim == 7168 — a magic-number coupling to the 671B config; any other model dim that needs e_score_correction_bias (e.g., V3.1-style variants on different dims) would silently load-mismatch.
- world_size, rank, gemm_impl, attn_impl, and block_size are module-level globals in model.py; world_size/rank are only (re)assigned inside Transformer.__init__, so constructing modules before the Transformer or changing impl flags post-hoc gives stale values.
- gemm_impl and attn_impl are not exposed as CLI flags in generate.py — changing them requires editing model.py, and nothing validates the combination against the loaded checkpoint dtype.
- main() rebinds the builtin name 'print' to a no-op lambda on non-zero ranks via 'global print' — any library code or later edits relying on print on those ranks are silently swallowed.
- convert.py unconditionally skips every tensor containing 'model.layers.61', silently discarding the MTP module weights without warning.
- Interactive mode synchronizes ranks with dist.broadcast_object_list, which pickles Python objects over the wire — acceptable for a trusted torchrun cluster but unsafe if any peer is untrusted. (inferred)
- User-supplied paths (--ckpt-path, --config, --input-file, --hf-ckpt-path, --save-path) are used directly in open()/glob()/os.makedirs with no validation; this is a CLI tool so impact is limited to the invoking user's privileges. (inferred)
- Validation is assert-based throughout (CLI arg checks, batch-size limit, divisibility, contiguity); running under python -O strips all of it, turning config errors into shape mismatches deep in the forward pass.
- KV caches are allocated at construction as zeros(max_batch_size, max_seq_len, ...) — with defaults max_seq_len=16384 that's a large always-on GPU allocation per layer regardless of actual prompt lengths.
- The MoE expert dispatch is a Python loop over experts with torch.where gather per expert — O(n_routed_experts) kernel launches per layer per step; correct but far from production performance (README itself steers users to SGLang/vLLM).
- weight_dequant inside the absorb attention path runs every forward step when wkv_b is FP8 (wkv_b.scale is not None), dequantizing the full kv_b weight per decode step — a hidden per-token cost.
- sample() divides probs in-place by an exponential_ tensor (probs.div_); logits are also divided by max(temperature,1e-5) — correct Gumbel-max sampling, but in-place mutation of freshly computed probs is fragile if reused. (inferred)
- fp8_cast_bf16 loads entire safetensor files to cuda and caches up to 2 files; on machines with less VRAM than 2 shards this OOMs, and the 'missing scale_inv' path only prints a warning while still writing the FP8 weight into a 'BF16' checkpoint.
- generate() pads the token buffer with -1 and relies on prompt_mask; if a batch mixes prompt lengths, positions before min(prompt_lens) are forced from the buffer — correct, but the loop can never start before min(prompt_lens), so shorter prompts waste no steps while all sequences share total_len.