DeepSeek-V3 is a 61-layer Transformer with a sparse mixture-of-experts block: the first three layers keep dense FFNs while the remaining 58 replace them with DeepSeekMoE layers of one shared expert and 256 fine-grained routed experts (per-expert width 2048), eight of which fire per token — 671B total, 37B active parameters. Attention is Multi-head Latent Attention (MLA) carried over from DeepSeek-V2: keys and values are jointly compressed into a 512-dimensional latent plus a 64-dimensional decoupled RoPE key per head, so the KV cache stores only the latent and the rotary key per token, and queries are low-rank compressed (1536-dim) to cut training activations. Load balancing is auxiliary-loss-free: per-expert biases steer sigmoid-gated top-8 routing and are nudged by a bias-update speed of 0.001, with a tiny sequence-wise auxiliary loss (alpha=0.0001) guarding against within-sequence imbalance; node-limited routing sends each token to at most four nodes, and no tokens are dropped in training or inference. A depth-1 Multi-Token Prediction module — shared embedding and output head with RMSNorm-gated fusion — adds a second-token loss and doubles as a speculative-decoding draft (85-90% acceptance, 1.8x tokens per second). Training on 2048 H800 GPUs combines FP8 mixed precision (E4M3, fine-grained tile/block quantization, CUDA-core accumulation), DualPipe 16-way pipeline parallelism, 64-way expert parallelism, and no tensor parallelism, over 14.8T tokens, then YaRN-based two-stage extension to 128K, then SFT and GRPO RL with reasoning distilled from DeepSeek-R1. The design's significance is cost: about 2.788M GPU hours (roughly $5.6M) for a frontier-competitive open-weight model.
What the report claims is novel
Pioneers an auxiliary-loss-free load balancing strategy for DeepSeekMoE: per-expert bias terms adjusted each step (bias update speed gamma=0.001) steer top-k routing without an auxiliary loss, avoiding the performance penalty of auxiliary-loss-based balancing; a small sequence-wise auxiliary loss (alpha=0.0001) only prevents extreme within-sequence imbalance, and no tokens are dropped [§2.1.2 / §4.2]
Investigates a Multi-Token Prediction (MTP) training objective using sequential, causality-preserving modules with a shared embedding layer and output head; it improves evaluation performance, and the MTP modules can be repurposed for speculative decoding (second-token acceptance 85-90%, ~1.8x TPS) [§2.2 / §5.4.3]
Introduces an FP8 mixed-precision training framework and, for the first time, validates FP8 training on an extremely large-scale model (fine-grained 1x128/128x128 quantization with online scaling, E4M3 everywhere, CUDA-core promotion for FP32 accumulation; relative loss error below 0.25% vs BF16) [§3.3 / Appendix B]
Designs DualPipe, a bidirectional pipeline-parallel algorithm with fewer bubbles and full computation-communication overlap, plus custom cross-node all-to-all kernels over IB and NVLink, achieving near-zero communication overhead for fine-grained cross-node experts and enabling training without tensor parallelism [§3.2]
Completes pre-training of DeepSeek-V3 on 14.8T tokens at an economical cost of 2.664M H800 GPU hours (2.788M including context extension and post-training, ~$5.576M total), producing the strongest open-source base model at the time of writing, with no irrecoverable loss spikes or rollbacks [§1 / Table 1]
Introduces a post-training methodology that distills reasoning capability from a long-CoT DeepSeek-R1 series model into a standard LLM, incorporating R1's verification and reflection patterns while maintaining control over output style and length [§5 / §5.4.1]
Atlas assessment: The core block is inherited from DeepSeek-V2 — MLA and DeepSeekMoE are adopted wholesale, so the genuinely new mechanisms are the auxiliary-loss-free bias-based load balancing (previously published by DeepSeek's own group, Wang et al. 2024a, per the report) and the sequential MTP objective (a causality-preserving variant of Gloeckle et al.'s parallel multi-token prediction). Against other atlas entries, DeepSeek-V3 is distinctive for low-rank latent KV compression versus GQA (Llama 3.1) or plain top-k MoE (Mixtral), for fine-grained 256-expert routing with a shared expert, and for a training scheme that avoids tensor parallelism; the FP8 framework and DualPipe are engineering contributions whose novelty lies mainly in the 671B-scale validation. The report's 'pioneers' and best-in-class open-source claims rest on its own internal evaluation harness, and the closed-source comparisons (GPT-4o, Claude-3.5-Sonnet) are self-reported, so headline numbers should be read with the same caution as any vendor report.
Relation to the atlas
Lineage in
MLA (multi-head latent attention) from DeepSeek-V2
DeepSeekMoE (fine-grained routed experts plus isolated shared experts) from DeepSeek-V2 (Dai et al. 2024)
Auxiliary-loss-free bias-based load balancing from Wang et al. 2024a (DeepSeek auxiliary-loss-free MoE work)
Multi-token prediction (MTP) training objective from Gloeckle et al. 2024 (parallel MTP; V3 makes it sequential)
Causal-chain speculative draft (EAGLE-style) from EAGLE (Li et al. 2024b)
YaRN context extension (applied to the decoupled MLA key) from Peng et al. 2023a (YaRN)
GRPO (group relative policy optimization) from DeepSeekMath (Shao et al. 2024)
Influence out
moonshot-kimi-k3: MLA lineage: K3 retains MLA in its periodic Gated MLA layers (report cites DeepSeek-V2)
zyphra-zaya1-8b: Auxiliary-loss-free bias-based load balancing — PID-style update of the DeepSeek bias scheme
poolside-laguna-xs-2-33b: Shared + routed experts with routed-expert modulation ('similar to DeepSeek-V3')
deepseek-v3-2: Base architecture with 128K context extension — DeepSeek-V3.2 starts from the DeepSeek-V3.1-Terminus base checkpoint and shares its architecture except for DSA
moonshot-kimi-k2: Multi-head Latent Attention (MLA) with partial rotary encoding
auxiliary-loss-free bias-based balancing (per-expert bias b_i, bias update speed γ=0.001) plus a complementary sequence-wise auxiliary loss (α=0.0001); no token dropping[12]
gradient clipping norm 1.0; FP8 training keeps embedding, output head, MoE gating, normalization and attention operators in BF16/FP32, with master weights and gradients in FP32; EMA of model parameters kept on CPU for early loss-decay estimates[18]
Extended vocabulary of 128K tokens; pretokenizer and tokenizer training data modified for multilingual compression efficiency; pretokenizer adds combined punctuation-and-linebreak tokens, mitigated by randomly splitting a proportion of such tokens during training; Fill-in-the-Middle applied at rate 0.1 via the Prefix-Suffix-Middle (PSM) framework[19]
14.8T tokens; versus DeepSeek-V2 the corpus raises the ratio of mathematical and programming samples and expands multilingual coverage beyond English and Chinese; processing pipeline refined to minimise redundancy while preserving diversity; document packing without cross-sample attention masking; FIM at rate 0.1 (PSM). Exact mixture percentages are not disclosed[19]
Curriculum / staging
Single pre-training run at 4K sequence length with a phased LR schedule (linear warmup, constant, cosine decay, final constant); then two-stage context extension (32K, 128K); then post-training (SFT, RL)[20]
linear warmup 0 to 2.2e-4 over the first 2K steps; constant 2.2e-4 until 10T tokens; cosine decay to 2.2e-5 over 4.3T tokens; final 500B tokens: constant 2.2e-5 for the first 333B then 7.3e-6 for the remaining 167B[7]
Batch schedule
batch size gradually increased from 3072 to 15360 over the first 469B tokens, then held at 15360[7]
Precision
FP8 mixed precision: GEMMs in FP8 (E4M3 format on all tensors); embedding, output head, MoE gating, normalization and attention kept in BF16/FP32; master weights and gradients in FP32; optimizer first/second moments in BF16; activations cached FP8 (custom E5M6 for inputs of the Linear after attention); fine-grained quantization (1x128 activation tiles, 128x128 weight blocks) with online scaling; promotion to CUDA cores every Nc=128 elements for FP32 accumulation[21]
Parallelism
16-way pipeline parallelism (DualPipe, bidirectional), 64-way expert parallelism spanning 8 nodes, ZeRO-1 data parallelism; no tensor parallelism in training[22]
Hardware
2048 NVIDIA H800 GPUs; nodes of 8 GPUs with NVLink/NVSwitch (NVLink 160 GB/s); InfiniBand (50 GB/s) across nodes[23]
Disclosed compute
2.788M H800 GPU hours total (2664K pre-training + 119K context extension + 5K post-training); ~$5.576M at $2 per GPU hour; 180K GPU hours per trillion tokens[24]
Long-CoT reasoning data distilled from the DeepSeek-R1 series (reflection/verification patterns); SFT data curated from RL-trained expert model generators via rejection sampling; GRPO RL with rule-based and model-based reward models; self-rewarding via constitutional-AI-style voting[27]
Distillation
DeepSeek-R1 series: reasoning capability distilled into DeepSeek-V3 via R1-generated SFT data and expert-model rejection sampling[28]
MLA low-rank latent KV cache: only the compressed KV latent (dc=512) and the decoupled RoPE key (64 dims per head) are cached per token, sharply reducing KV cache versus MHA[9]
MTP modules repurposable for speculative decoding: second-token acceptance rate 85-90%, ~1.8x decoding speed (tokens per second)[31]
Serving optimisations
Separate prefill/decode deployment: prefill TP4+SP with DP8 and EP32 (4 nodes/32 GPUs, 32 redundant experts); decode TP4+SP with DP80 and EP320 (40 nodes/320 GPUs, one expert per GPU, shared expert treated as always-selected routed expert); redundant high-load expert duplication with periodic rebalancing; dynamic redundancy explored; IBGDA; dual micro-batch overlap of attention and MoE dispatch/combine[30]
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
MMLU (EM)
88.5
Table 6
MMLU-Redux (EM)
89.1
Table 6
MMLU-Pro (EM)
75.9
Table 6
GPQA-Diamond (Pass@1)
59.1
Table 6
SimpleQA (Correct)
24.9
Table 6
C-SimpleQA (Correct)
64.8
Table 6
DROP (3-shot F1)
91.6
Table 6
LongBench v2 (Acc.)
48.7
Table 6
HumanEval-Mul (Pass@1)
82.6
Table 6
LiveCodeBench (Pass@1-COT)
40.5
Table 6
Codeforces (Percentile)
51.6
Table 6
SWE Verified (Resolved)
42.0
Table 6
AIME 2024 (Pass@1)
39.2
Table 6
MATH-500 (EM)
90.2
Table 6
CNMO 2024 (Pass@1)
43.2
Table 6
Provenance references
stated · Title / Abstract
stated · Title page author line
stated · pipeline
stated · Abstract ('The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3')
stated · Abstract / §4.2 / Table 3
stated · Abstract / §4.2
stated · §4.2
n/a · §2.1.1
stated · §2.1.1
stated · §3.3.3 ('Inputs of the SwiGLU operator in MoE')