Kimi K2 is a 61-layer, 1.04T-parameter ultra-sparse MoE (32.6B active per token) that largely follows DeepSeek-V3's blueprint: Multi-head Latent Attention in every layer (latent KV compression, partial rotary encoding), a hidden width of 7168, and a shared-expert MoE — but with sparsity scaled up to 48 via a dedicated scaling law (384 routed experts of width 2048, 8 active per token, 1 shared expert), attention heads halved to 64 to cut long-context inference FLOPs, and expert grouping dropped. The headline novelty is MuonClip: the Muon optimizer (inherited from Moonlight) augmented with per-head QK-Clip, which rescales query/key projection weights whenever max attention logits exceed τ=100, stabilizing Muon's tendency to explode logits and yielding a spike-free 15.5T-token run (4k context, WSD schedule, 67M-token batches, annealing at 4k/32k, then YaRN to 128k). Post-training is a three-stage story: SFT on a large instruction set with a large-scale agentic tool-use synthesis pipeline (real MCP tools plus evolved synthetic tools, rubric-verified trajectories), then joint RL combining verifiable rewards with a self-critique rubric reward and per-sample budget control, PTX loss, and temperature decay, with long-horizon agentic rollout in real sandboxes. The report positions K2 as the strongest open non-thinking model, particularly on SWE and tool-use. In the atlas, K2 is the direct ancestor of moonshot-kimi-k3, whose report explicitly carries over K2's weight-clipping and data pipelines; note this record's own report is text-only and predates the K3 report by a year.
What the report claims is novel
MuonClip: a novel optimizer integrating the token-efficient Muon algorithm with a QK-Clip stability mechanism (per-head rescaling of query/key projection weights), enabling pre-training on 15.5T tokens with zero loss spikes [Abstract, §2.1]
A large-scale agentic data synthesis pipeline that systematically generates tool-use demonstrations via simulated and real-world environments (3,000+ real MCP tools, 20,000+ synthetic tools, rubric-based task generation, LLM-judge filtering) [Abstract, §3.1.1]
A joint reinforcement learning framework combining verifiable rewards (RLVR) with a Self-Critique Rubric Reward mechanism that extends alignment from static tasks into open-ended domains [Abstract, §3.2.2]
A sparsity scaling law for the MoE family under Muon showing that sparsity 48 cuts FLOPs by 1.69×/1.39×/1.15× vs sparsity 8/16/32 at equal loss, motivating 384 experts with 8 active [§2.3]
State-of-the-art performance among open-source non-thinking models, with strengths in agentic capabilities (Tau2-Bench 66.1, ACEBench 76.5, SWE-bench Verified 65.8) and top-1 open-source ranking on LMSYS Arena (July 17, 2025) [Abstract, §1, §4.1.2]
Open release of base and post-trained checkpoints plus the checkpoint-engine source code to facilitate agentic-intelligence research [Abstract, §3.3.2]
Atlas assessment: Architecturally, K2 is a direct refinement of DeepSeek-V3 (the closest atlas peer, deepseek-v3): the same MLA attention and shared-expert MoE skeleton, with sparsity pushed to 48 (384 routed + 1 shared expert vs 256 + 1), attention heads halved 128→64, expert grouping removed, and one unexplained dense layer among 61. The genuinely novel contributions sit at the optimizer and data/RL level: QK-Clip is a distinct stability mechanism (post-update per-head weight rescaling) that the report explicitly contrasts with both logit soft-capping and QK-Norm, and the agentic data-synthesis + self-critique rubric RL pipeline is a training contribution, not a block change. The 15.5T-token zero-loss-spike claim is internally validated only by a loss curve. Within the atlas, K2 is the direct ancestor of moonshot-kimi-k3, whose report inherits the weight-clipping lineage and rephrasing pipelines.
Relation to the atlas
Lineage in
Multi-head Latent Attention (MLA) with partial rotary encoding from DeepSeek-V2 / DeepSeek-V3
Ultra-sparse MoE with one shared expert and 8 routed experts active from DeepSeek-V3
Muon optimizer with consistent update RMS scaling from Moonlight (Moonshot)
WSD (warmup-stable-decay) learning-rate schedule from MiniCPM
YaRN context extension to 128k from YaRN (Peng et al.)
K1.5 policy-optimization RL objective with mean-reward baseline and KL regularization from Kimi K1.5
Style-/perspective-diverse knowledge rephrasing (WRAP-inspired) and learning-note math rewriting (SwallowMath-style) from WRAP / SwallowMath
Influence out
moonshot-kimi-linear-48b-a3b: Pretraining recipe from Kimi K2: MuonClip optimizer, WSD schedule, annealing + long-context activation phase, K2 pretraining corpus
Notable omissions
Vocabulary size and tokenizer algorithm not disclosed (only the tool-calling token template appears, Appendix B)
Router gating function and load-balancing mechanism not described (only expert counts and sparsity 48)
MLA latent dimensions (compressed-KV and rotary widths) not disclosed; head dim only derivable as 7168/64 = 112
RoPE base frequency and YaRN scaling factor not disclosed
Training compute (FLOPs or GPU-hours) not disclosed
No license named for the released checkpoints
Input/output embedding tying not disclosed
Normalization placement (pre/post) not described
The single dense layer among 61 layers is never explained
Per-domain proportions of the 15.5T-token corpus not given
No ablation of the QK-Clip threshold τ on the full model (τ=100 only)
Inference serving stack (batching, quantization, speculative decoding) not described; FP8 appears only as training-time activation storage
YaRN after an annealing phase (400B tokens at 4k, then 60B tokens at 32k sequence) and a long-context activation stage[25]
Tokenizer
Algorithm
not disclosed
Notes
No tokenizer details disclosed; Appendix B documents the tool-calling token template (<|im_begin|>, <|tool_call_section_begin|>, <|tool_call_begin|>, <|tool_arguments_begin|>, <|tool_call_end|>, <|im_end|>) with TypeScript tool declarations and a constrained-decoding 'enforcer' module[26]
15.5T tokens of curated high-quality data over four domains — Web Text, Code, Mathematics, Knowledge; most pipelines follow Kimi K1.5; Knowledge rephrased with WRAP-inspired style-/perspective-diverse prompting, chunk-wise autoregressive rewriting and fidelity verification; Mathematics rewritten into SwallowMath-style learning notes and translated into English; each corpus rephrased at most twice[28]
Curriculum / staging
annealing phase (400B tokens at 4k sequence, then 60B tokens at 32k) followed by a long-context activation stage; YaRN extension to 128k; batch size held constant throughout[25]
Optimizer
MuonClip: Muon with weight decay, consistent update RMS scaling, and per-head QK-Clip (τ = 100)[29]
LR schedule
WSD: 500-step warm-up then constant 2e-4 for the first 10T tokens, cosine decay 2e-4 → 2e-5 over the remaining 5.5T; annealing decays 2e-5 → 7e-6[25]
Batch schedule
global batch size held at 67M tokens throughout pre-training and annealing[25]
Precision
parameters stored in BF16 with FP32 gradient-accumulation buffer; FP8-E4M3 used only for storing insensitive activations (1×128 tiles, FP32 scales), never in computation[30]
Parallelism
16-way Pipeline Parallelism with virtual stages (interleaved 1F1B; weight-gradient computation decoupled from backward and overlapped with PP communication), 16-way Expert Parallelism, ZeRO-1 Data Parallelism; EP all-to-all overlapped with computation; trainable on any node count multiple of 32[31]
Hardware
NVIDIA H800 cluster: 8 GPUs per node with NVLink/NVSwitch and 2 TB RAM per node, 8×400 Gbps RoCE inter-node interconnect[32]
joint RL stage combining verifiable rewards (math/STEM/logic, complex instruction following, faithfulness via a sentence-level judge, coding/SWE, safety) with self-critique rubric reward; per-sample token budget control, PTX loss, temperature decay; long-horizon agentic rollout in real and synthetic environments with partial rollouts[35]
MLA low-rank latent KV cache (keys/values reconstructed from a compressed per-token latent, DeepSeek-V3-style); attention heads deliberately cut from 128 to 64 to reduce long-context inference overhead (83% inference-FLOP increase estimated for 128 heads at 128k)
Quantisation shipped
none disclosed
Speculative decoding
none disclosed
Serving optimisations
none disclosed
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
Tau2-Bench (Pass@1)
66.1
Abstract
ACEBench (En) (Acc.)
76.5
Abstract, Table 3
SWE-bench Verified — Agentic Single Attempt (Pass@1)
SWE-bench Verified — Agentless Single Patch (Pass@1)
51.8
Table 3
MultiPL-E (Pass@1)
85.7
Table 3
Multi-SWE-bench (Pass@1)
18.3
Table 3
SWE-Lancer (Pass@1)
39.1
Table 3
PaperBench Code-Dev (Acc.)
27.8
Table 3
Terminal Bench Terminus (Acc.)
25.0
Table 3
Terminal Bench In-House (Acc.)
30.0
Table 3, Appendix C
Aider-Polyglot (Acc.)
60.0
Table 3
Tau2 retail (Avg@4)
70.6
Table 3
AIME 2024 (Avg@64)
69.6
Table 3
MATH-500 (Acc.)
97.4
Table 3
HMMT 2025 (Avg@32)
38.8
Table 3
ZebraLogic (Acc.)
89.0
Table 3
AutoLogi (Acc.)
89.5
Table 3
SuperGPQA (Acc.)
57.2
Table 3
MMLU (EM)
89.5
Table 3
MMLU-Redux (EM)
92.7
Table 3
MMLU-Pro (EM)
81.1
Table 3
IFEval (Prompt Strict)
89.8
Table 3
Multi-Challenge (Acc.)
54.1
Table 3
SimpleQA (Correct)
31.0
Table 3
LiveBench (Pass@1)
76.4
Table 3
Arena Hard v2.0 Hard Prompt (Win rate)
54.5
Table 3
Arena Hard v2.0 Creative Writing (Win rate)
85.0
Table 3
FACTS Grounding (Adjusted)
88.5
Table 3
HHEM v2.1 (1-Hallu.)
98.9
Table 3
FaithJudge (1-Hallu.)
92.6
Table 3
DROP (Acc.)
93.5
Table 3
MRCR (Acc.)
55.0
Table 3
FRAMES (Acc.)
77.1
Table 3
LongBench v2 (Acc.)
49.1
Table 3
MMLU (Base, 5-shot)
87.79
Table 4
EvalPlus (Base)
80.33
Table 4
Provenance references
stated · Title
stated · Title byline; fn 1 (github.com/MoonshotAI/checkpoint-engine); Appendix C ('Moonshot's in-house agentic framework')
stated · pipeline
stated · Abstract: 'We release our base and post-trained model checkpoints'; fn 1 (huggingface.co/moonshotai/Kimi-K2-Instruct)
stated · Table 4 (# Total Params 1043B)
stated · Table 2 (Activated Parameters 32.6B)
stated · Table 2 (#Layers 61)
stated · §2.3 (model hidden dimension of 7168)
stated · §2.3 (MoE expert hidden dimension of 2048)
stated · §2.3, Table 2 (Attention Heads 64)
n/a · §2.3 (MLA: keys/values not materialized per head; low-rank latent KV compression)
stated · §2.3 ('employing Multi-head Latent Attention (MLA) as the attention mechanism')
stated · §2.3, Table 2 (Number of Dense Layers 1)
stated · §2.4.3 (recomputation of 'SwiGLU'; FP8 storage for 'inputs of MoE up-projections and SwiGLU')
stated · §2.3, Table 2 (Experts total 384)
stated · §2.3, Table 2 (Experts Active per Token 8)
stated · Table 2 (Shared Experts 1)
stated · §2.5 ('To extend the context window to 128k, we employed the YaRN method')
stated · §2.5 (context window extended to 128k)
stated · §2.1 (MLA decomposed into compressed qC/kC and rotary qR/kR components); §2.5 (YaRN extension applied to the rotary encoding)
stated · Appendix E ('RMS-Norm keeps ‖xi‖‖xj‖ bounded'); §2.4.3 names 'LayerNorm' for recomputation, presumably the same norm
stated · §2.1 (QK-Norm 'is not applicable to MLA'; Kimi K2 instead applies QK-Clip weight clipping)
stated · §2.1 (logit soft-cap described as an insufficient mitigation — 'the dot products between queries and keys can still grow excessively before capping is applied'; K2 uses QK-Clip instead)
stated · §2.5 ('We pre-trained the model with a 4,096-token context window')
stated · §2.5
stated · Appendix B
stated · Abstract, §2 ('pre-trained on 15.5 trillion tokens')
stated · §2.2
stated · §2.1
stated · §2.4.2, §2.4.3
stated · §2.4.2
stated · §2.4.1
stated · §3.1 (large-scale instruction-tuning dataset; agentic tool-use data synthesis pipeline with 3,000+ real MCP tools and 20,000+ synthetic tools)
stated · §3.2.3 (Kimi K1.5 policy-optimization objective with mean-reward baseline and KL regularization; RLVR for verifiable tasks plus Self-Critique Rubric Reward for subjective tasks; no RLHF/PPO/DPO/GRPO named)