DeepSeek-V3.2
DeepSeek-AI · 2025-12-02 · analysed 2026-08-10 · skill v0.3.5
sparse-MoE efficient-attention text-only open-weights reasoning-specialisedDesign overview
DeepSeek-V3.2 is a continued-trained member of the DeepSeek-V3 line: it starts from the DeepSeek-V3.1-Terminus base checkpoint (context already extended to 128K, architecture otherwise unchanged) and its only architectural modification is DeepSeek Sparse Attention (DSA). DSA replaces vanilla attention with a lightning indexer — a small number of heads computing a weighted ReLU score between each query and every preceding token, implementable in FP8 — and a fine-grained top-k selection that retrieves only the k=2048 best-scoring key-value entries per query, cutting the main attention cost from O(L²) to O(Lk). DSA is instantiated under MLA in its MQA mode (each latent KV vector shared across all query heads) for kernel efficiency; short-sequence prefilling uses a masked MHA mode that simulates DSA. The indexer is initialized in a frozen dense warm-up (1000 steps, 2.1B tokens) by KL-aligning its softmax scores to the summed head-wise attention distribution, then trained jointly in a 943.7B-token sparse stage. Post-training keeps the V3.2-Exp pipeline: six domain specialists plus writing and QA (thinking and non-thinking modes), distilled into the final model, then a single mixed GRPO stage fusing reasoning, agent, and alignment rewards, stabilized by an unbiased KL estimate, off-policy sequence masking, keep-routing, and keep-sampling masks; RL compute exceeds 10% of pre-training cost. Agentic capability comes from thinking context management (reasoning retained across tool messages) and a synthesis pipeline of 1,827 environments / 85,000 prompts. The relaxed-length DeepSeek-V3.2-Speciale variant, adding DeepSeekMath-V2 techniques, reaches gold-medal level at IMO/CMO/IOI 2025 and ranks 2nd at ICPC WF 2025.
What the report claims is novel
- DeepSeek Sparse Attention (DSA): a lightning indexer (few heads, ReLU-scored, FP8-implementable) plus fine-grained top-k token selection, cutting the main attention complexity from O(L²) to O(Lk) with k=2048 selected key-value entries per query, instantiated under MLA in MQA mode [§2.1]
- A stable, scalable RL protocol: GRPO augmented with an unbiased KL estimate, off-policy sequence masking, keep routing, and keep sampling mask, supporting a post-training computational budget exceeding 10% of the pre-training cost [§1, §3.1]
- Large-scale agentic task synthesis pipeline: over 1,800 synthesized environments and 85,000 complex prompts (24,667 code-agent, 50,275 search-agent, 4,417 general-agent, 5,908 code-interpreter tasks) for scalable agentic post-training [§1, §3.2.3]
- Thinking context management for tool-use: historical reasoning content discarded only when a new user message arrives, with tool-call history preserved, avoiding redundant re-reasoning per tool call [§3.2.1]
- DeepSeek-V3.2-Speciale: a relaxed-length, high-compute variant reaching gold-medal performance in IMO 2025, CMO 2025, IOI 2025 and ICPC World Finals 2025 (ranked 2nd), 'without targeted training' [§4.2, Table 4]
- Parity claim: the DSA base model matches DeepSeek-V3.1-Terminus on standard benchmarks and human preference (closely matched ChatbotArena Elo, +4 points on AA-LCR) while substantially improving long-sequence computational efficiency [§2.2]
Atlas assessment: DSA is the first sparse-attention mechanism in the atlas built on MLA: an auxiliary lightning indexer computes cheap ReLU scores, a frozen dense warm-up (1000 steps) KL-aligns indexer softmax scores to the summed head-wise attention distribution, and top-k=2048 selection then sparsifies every layer's attention to O(Lk). It is a training-stabilized retrieval-style sparsification, not a new mixing family — contrast Kimi K3's KDA/MLA layerwise hybrid and the GLM-4.5 attention variants. The report's own parity evidence (base model matches V3.1-Terminus) is the key architectural claim and is internally consistent. The GRPO stability recipe (unbiased KL, off-policy masking, keep-routing, keep-sampling) is a substantive engineering contribution beyond deepseek-r1's report. Everything else — MLA, DeepSeekMoE, 128K context — is inherited unchanged from the V3 line (stated explicitly), so the atlas deepseek-v3 record remains the family's disclosure source for parameters. The >10%-of-pre-training RL compute and competition gold medals are internal claims, not independently verifiable.
Relation to the atlas
Lineage in
- Multi-head Latent Attention (MLA) — DSA is instantiated under MLA, with the MQA mode used for kernel efficiency from DeepSeek-V2
- Base architecture with 128K context extension — DeepSeek-V3.2 starts from the DeepSeek-V3.1-Terminus base checkpoint and shares its architecture except for DSA from DeepSeek-V3.1-Terminus
- Cold-start phase unifying reasoning and tool-use within single trajectories from DeepSeek-V3
- Group Relative Policy Optimization (GRPO) as the RL training algorithm from DeepSeek-R1 / DeepSeekMath
- DeepSeekMath-V2 dataset and reward method for mathematical proofs (used in DeepSeek-V3.2-Speciale) from DeepSeekMath-V2
Influence out
- deepseek-v4: Lightning-indexer sparse attention (DSA)
- zai-glm-5: DeepSeek Sparse Attention (DSA) — content-based top-k sparse attention via a lightning indexer
Notable omissions
- No parameter counts, active-parameter count, layer count, or hidden dims restated — the report refers to the DeepSeek-V3.1-Terminus base without numbers
- No MoE expert counts, router description, or load-balancing details (MoE acknowledged only in passing, §3.1)
- No positional-encoding, normalization, activation, or head-dimension details (deferred to the V3.1 architecture)
- No tokenizer or vocabulary description
- No training hardware, optimizer, precision, or absolute compute (FLOPs/GPU-hours); only the relative '>10% of pre-training cost' RL budget
- No license named for the release
- No explicit statement that model weights are released (only the open-source inference implementation link, §2.1)
- Number of indexer heads (H^I) and indexer dimension (d^I) not given numerically
- Top-k=2048 given for the sparse training stage only; the deployed k for inference is not stated
- The masked MHA mode for short-context prefill is named but not specified in detail
- No statement on DeepSeek-V3.2-Speciale availability/weights
Closest relatives in the atlas
- GLM-5 — 5 shared comparable fields
- DeepSeek-V3 — 4 shared comparable fields
- Kimi K2 — 4 shared comparable fields
Spec sheet
Identity
| Family | DeepSeek-V3.2[1] |
|---|---|
| Organisation | DeepSeek-AI[2] |
| Release date | 2025-12-02[3] |
| Report URL | https://arxiv.org/abs/2512.02556[4] |
| License | not disclosed |
| Open weights | yes |
Scale
| Total parameters | not disclosed |
|---|---|
| Active parameters / token | not disclosed |
| Layers | not disclosed |
| Hidden dimension | not disclosed |
| FFN inner dimension | not disclosed |
| FFN ratio | not disclosed |
| Query heads | not disclosed |
| KV heads | n/a[5] |
| Head dimension | not disclosed |
| Vocabulary size | not disclosed |
| Embedding tying | not disclosed |
Core block
| Block type | sparse-MoE[6] |
|---|---|
| Attention variant | MLA[7] |
| Attention layer pattern | uniform |
| Depth mixing | sequential-residual |
| Activation | not disclosed |
| Expert count | not disclosed |
|---|---|
| Experts per token | not disclosed |
| Shared experts | not disclosed |
| Routing | not disclosed |
| Load balancing | not disclosed |
| Expert granularity | not disclosed |
| Method | not disclosed |
|---|---|
| RoPE base frequency | not disclosed |
| Partial RoPE | not disclosed |
| Type | not disclosed |
|---|---|
| Placement | not disclosed |
| QK-norm | not disclosed |
| Attention sinks | no |
|---|---|
| Softcapping | no |
| Other | not disclosed |
Context
| Trained context length | 131,072[10] |
|---|---|
| Deployed context length | 131,072[9] |
| Extension method | none |
Tokenizer
| Algorithm | not disclosed |
|---|---|
| Notes | not disclosed |
Training
| Training tokens | 945.8T |
|---|---|
| Data composition | continued pre-training data distribution 'totally aligned with the 128K long context extension data used for DeepSeek-V3.1-Terminus' (§2.1.1); post-training data: specialist-distilled domain data (mathematics, programming, general logical reasoning, general agentic tasks, agentic coding, agentic search, plus writing and general QA, in thinking and non-thinking modes) and agentic RL data (real tools: web search API, coding tools, Jupyter Notebooks; synthesized: 1,827 task-oriented environments with 4,417 general-agent tasks, plus 24,667 code-agent, 50,275 search-agent, 5,908 code-interpreter tasks)[11] |
| Curriculum / staging | two-stage DSA continued pre-training: (1) dense warm-up freezing all parameters except the lightning indexer, trained with a KL objective aligned to the summed head-wise attention distribution (LR 1e-3, 1000 steps); (2) sparse training with fine-grained top-k selection, optimizing all parameters (LR 7.3e-6, 15000 steps, k=2048); then post-training: specialist distillation → mixed GRPO RL (reasoning + agent + alignment in one stage) → continued RL to final checkpoints[12] |
| Optimizer | not disclosed |
| LR schedule | continued pre-training learning rates: 1e-3 (indexer warm-up stage), 7.3e-6 (sparse training stage)[10] |
| Batch schedule | 16 sequences of 128K tokens per step (dense warm-up); 480 sequences of 128K tokens per step (sparse stage)[10] |
| Precision | not disclosed |
| Parallelism | not disclosed |
| Hardware | not disclosed |
| Disclosed compute | not disclosed |
Post-training
| SFT | yes |
|---|---|
| Preference optimisation | GRPO[13] |
| Reasoning training | mixed RL merging reasoning, agent, and human-alignment training into one GRPO stage; rule-based outcome reward, length penalty and language consistency reward for reasoning/agent tasks, generative reward model with per-prompt rubrics for general tasks; RL compute >10% of pre-training cost; specialist distillation producing thinking (long-CoT) and non-thinking data; RL stabilizers: unbiased (importance-sampling corrected) KL estimate, off-policy sequence masking, keep routing, keep sampling mask; thinking-in-tool-use via cold-start (DeepSeek-V3 methodology) and thinking context management (reasoning retained across tool messages, discarded only on new user messages, tool-call history preserved); DeepSeek-V3.2-Speciale variant trained exclusively on reasoning data with reduced length penalty plus DeepSeekMath-V2 dataset/rewards for mathematical proofs[14] |
| Distillation | specialist distillation: per-domain specialist models (fine-tuned from the same pre-trained V3.2 base with large-scale RL) generate domain-specific data for the final checkpoint; 'models trained on the distilled data achieve performance levels only marginally below those of domain-specific specialists', with the gap eliminated by subsequent RL[13] |
Modality
| Type | text-only |
|---|---|
| Attachment | n/a[15] |
Inference efficiency
| KV-cache design | MLA: low-rank latent KV cache; DSA retrieves only the top-k=2048 key-value entries per query token, so attention computation touches a sparse subset of the latent cache; MQA-mode MLA shares each latent vector across all query heads for kernel efficiency |
|---|---|
| Quantisation shipped | none disclosed |
| Speculative decoding | none disclosed |
| Serving optimisations | masked MHA mode simulating DSA for short-sequence prefilling (higher efficiency under short contexts), plus 'our optimized implementation' of DSA achieving significant end-to-end speedup in long-context scenarios; token-cost benchmarks from the deployed service on H800 clusters at $2 per GPU hour[16] |
Evaluation
Benchmarks (report's own numbers only)
| Benchmark | Value | Ref |
|---|---|---|
| MMLU-Pro (EM) | 85.0 | Table 2 (DeepSeek-V3.2 Thinking) |
| GPQA Diamond (Pass@1) | 82.4 | Table 2 (DeepSeek-V3.2 Thinking) |
| HLE (Pass@1) | 25.1 | Table 2 (DeepSeek-V3.2 Thinking) |
| LiveCodeBench (Pass@1-COT) | 83.3 | Table 2 (DeepSeek-V3.2 Thinking) |
| Codeforces (Rating) | 2386 | Table 2 (DeepSeek-V3.2 Thinking) |
| AIME 2025 (Pass@1) | 93.1 | Table 2 (DeepSeek-V3.2 Thinking) |
| HMMT Feb 2025 (Pass@1) | 92.5 | Table 2 (DeepSeek-V3.2 Thinking) |
| HMMT Nov 2025 (Pass@1) | 90.2 | Table 2 (DeepSeek-V3.2 Thinking) |
| IMOAnswerBench (Pass@1) | 78.3 | Table 2 (DeepSeek-V3.2 Thinking) |
| Terminal Bench 2.0 (Acc) | 46.4 | Table 2 (DeepSeek-V3.2 Thinking; Claude Code framework) |
| SWE Verified (Resolved) | 73.1 | Table 2 (DeepSeek-V3.2 Thinking) |
| SWE Multilingual (Resolved) | 70.2 | Table 2 (DeepSeek-V3.2 Thinking) |
| BrowseComp (Pass@1, with/without context management) | 51.4/67.6* | Table 2 (DeepSeek-V3.2 Thinking) |
| BrowseCompZh (Pass@1) | 65.0 | Table 2 (DeepSeek-V3.2 Thinking) |
| HLE (Pass@1, search agent) | 40.8 | Table 2 (DeepSeek-V3.2 Thinking) |
| τ²-Bench (Pass@1) | 80.3 | Table 2 (DeepSeek-V3.2 Thinking) |
| MCP-Universe (Success Rate) | 45.9 | Table 2 (DeepSeek-V3.2 Thinking) |
| MCP-Mark (Pass@1) | 38.0 | Table 2 (DeepSeek-V3.2 Thinking) |
| Tool-Decathlon (Pass@1) | 35.2 | Table 2 (DeepSeek-V3.2 Thinking) |
| AIME 2025 (Pass@1, V3.2-Speciale) | 96.0 | Table 3 (DeepSeek-V3.2-Speciale) |
| GPQA Diamond (Pass@1, V3.2-Speciale) | 85.7 | Table 3 (DeepSeek-V3.2-Speciale) |
| HLE (Pass@1, V3.2-Speciale) | 30.6 | Table 3 (DeepSeek-V3.2-Speciale) |
| Codeforces (Rating, V3.2-Speciale) | 2701 | Table 3 (DeepSeek-V3.2-Speciale) |
| IMO 2025 (points, V3.2-Speciale) | 35/42 (Gold) | Table 4 (DeepSeek-V3.2-Speciale) |
| IOI 2025 (points, V3.2-Speciale) | 492/600 (Gold, 10th place) | Table 4 (DeepSeek-V3.2-Speciale) |
Provenance references
- stated · Title
- stated · Title page byline (DeepSeek-AI)
- stated · arXiv header ('arXiv:2512.02556v1 [cs.CL] 02 Dec 2025'); confirmed via export.arxiv.org API (published 2025-12-02)
- stated · pipeline
- n/a · §2.1
- stated · §3.1
- stated · §2.1
- n/a · §2.1.1
- stated · §4.1
- stated · §2.1.1
- stated · §2.1.1, §3, Table 1
- stated · §2.1.1, §3
- stated · §3
- stated · §3, §3.1, §3.2
- n/a · §1
- stated · §2.3