DeepSeek-V4

DeepSeek-AI · 2026-04-26 · analysed 2026-08-10 · skill v0.4.0

sparse-MoE efficient-attention text-only open-weights frontier long-context

Technical report ↗

Design overview

DeepSeek V4 is the line's efficiency overhaul: it removes dense attention entirely. The first two layers are pure sliding-window (window 128); the rest interleave CSA — which compresses KV every 4 tokens into two overlapping streams and then applies the lightning-indexer sparse attention from V3.2 — with HCA, a heavy-compression (m'=128) dense attention. MLA and QK-Clip are gone, replaced by RWKV-style RMSNorm on queries and compressed KV entries (qk_norm true). The residual stream uses mHC, Hyper-Connections whose connection matrix is constrained to the doubly-stochastic manifold via Sinkhorn-Knopp — a new depth-mixing mechanism for the atlas (v0.4.0). MoE keeps DeepSeekMoE's shared-expert structure (1 shared + 384 routed, top-6, Sqrt(Softplus) affinity) with FP4 QAT training. V4-Pro is 1.6T/49B active over 61 layers; V4-Flash is 284B/13B over 43. NOTE: the gallery's HF-hosted DeepSeek_V4.pdf URL is dead (404); the report lives on arXiv 2606.19348.

What the report claims is novel

Atlas assessment: The most aggressive efficiency-first evolution of the DeepSeek line in the atlas: MLA (V3) and DSA (V3.2) are superseded by a compressed sparse-attention stack, and the residual stream gains the first Hyper-Connections mechanism in the atlas (triggered the v0.4.0 depth_mixing schema extension). Novelty is high relative to atlas peers; the architecture section is dense and specific.

Relation to the atlas

Lineage in

Influence out

No descendants recorded yet.

Notable omissions

Closest relatives in the atlas

Spec sheet

Identity

FamilyDeepSeek-V4[1]
OrganisationDeepSeek-AI[2]
Release date2026-04-26
Report URLhttps://arxiv.org/abs/2606.19348[3]
Open weightsyes[4]

Variants

Scale

Total parameters1.6T[5]
Active parameters / token49B[6]
Layers61[7]
Hidden dimension7168[7]
FFN inner dimension3072[7]
FFN ratio0.43
Query heads128[8]
Head dimension512[7]
Vocabulary size128000[9]

Per-variant configs

VariantTotalLayersHiddenFFNQ headsKV headsContext
V4-Flash (284B)284B434096204864not disclosed1M
V4-Pro (1.6T)1.6T6171683072128not disclosed1M

Core block

Block typesparse-MoE
Attention varianthybrid[10]
Attention layer patternFirst 2 layers pure sliding-window (window 128); remaining layers interleaved CSA (KV compressed every m=4 tokens, overlapping dual compressed streams, with DeepSeek Sparse Attention lightning-indexer top-k) and HCA (heavy compression m'=128, dense attention)[10]
Depth mixinghyper-connections[11]
ActivationSwiGLU[12]
Expert count384[13]
Experts per token6[7]
Shared experts1[7]
RoutingSqrt(Softplus) affinity; hash routing in first 3 MoE layers[10]
Load balancingaux-loss-free load balancing + sequence balance loss[10]
MethodRoPE[15]
Partial RoPEyes[16]
TypeRMSNorm[10]
QK-normyes[17]
Attention sinksyes[18]
OtherSwiGLU clamping; Anticipatory Routing; mHC doubly-stochastic residual constraint[10]

Context

Trained context length1M[19]
Deployed context length1M[20]
Extension methodStaged sequence-length curriculum 4K→16K→64K→1M[10]

Tokenizer

Training

Training tokens33T[21]
Curriculum / stagingSequence-length staging 4K→16K→64K→1M[10]
OptimizerMuon with hybrid Newton-Schulz[22]
LR scheduleCosine-style decay 2.0e-4 → 2.0e-5 (V4-Pro)[22]
Batch scheduleBatch 94.4M tokens (V4-Pro) / 75.5M (V4-Flash)[22]

Post-training

Preference optimisationother
Reasoning trainingnone disclosed
Distillationnone disclosed

Modality

Typetext-only[23]

Inference efficiency

KV-cache designCSA compresses KV every m=4 tokens (overlapping dual streams) + HCA m'=128 + shared-KV MQA; FP4 (MXFP4) QAT on expert weights and indexer QK path
Quantisation shippedFP4 (MXFP4) quantization-aware training on expert weights and indexer QK path[24]
Speculative decodingMTP (depth 1)[10]

Evaluation

Benchmarks (report's own numbers only)

BenchmarkValueRef
MMLU90.1eval tables
MMLU-Pro73.5eval tables
SuperGPQA53.9eval tables
SimpleQA-verified55.2eval tables
HumanEval76.8eval tables
MATH64.5eval tables
LongBench-V251.5eval tables
GPQA-Diamond90.1eval tables
SWE-Verified80.6eval tables
MRCR-1M83.5eval tables
CorpusQA-1M62.0eval tables

Provenance references

  1. stated · Title (report: 'Towards Highly Efficient Million-Token Context Intelligence')
  2. stated · Title page byline
  3. stated · pipeline
  4. stated · Abstract (checkpoints released on Hugging Face)
  5. stated · config tables (V4-Pro 1.6T)
  6. stated · config tables (V4-Pro 49B active)
  7. stated · config tables
  8. stated · config tables (128 Q heads, head_dim 512)
  9. stated · config tables (128K)
  10. stated · architecture section
  11. stated · architecture section (mHC: Manifold-Constrained Hyper-Connections; residual matrix constrained to doubly stochastic manifold via Sinkhorn-Knopp, n_hc=4)
  12. stated · architecture section (with clamping)
  13. stated · config tables (routed; plus 1 shared)
  14. stated · Abstract
  15. stated · architecture section (partial)
  16. stated · architecture section (RoPE on last 64 dims, with -i RoPE on outputs)
  17. stated · architecture section (RWKV-style RMSNorm on queries and compressed KV entries, explicitly replacing QK-Clip)
  18. stated · architecture section (designed attention sinks)
  19. stated · Abstract (final curriculum stage trains at 1M)
  20. stated · Abstract (million-token context)
  21. stated · config tables (V4-Pro 33T; V4-Flash 32T)
  22. stated · training section
  23. stated · §6 (multimodal capabilities 'also working on')
  24. stated · quantization section