Kimi Linear is a 48B-parameter sparse MoE (3B active) that interleaves token mixing at a uniform 3:1 ratio: three Kimi Delta Attention (KDA) layers for every one full MLA layer. KDA is a gated-delta-rule linear attention — St = (I − βt kt ktᵀ) Diag(αt) S_{t−1} + βt kt vtᵀ — that fuses Gated DeltaNet's delta-rule recurrence with GLA-style channel-wise forgetting: each of the 128 key/value dimensions has its own decay αt, parameterized by a low-rank projection through a bounded decay function, giving fine-grained control of the fixed 128×128 per-head recurrent state. Queries/keys pass through ShortConv + Swish + L2Norm for eigenvalue stability, and the head output gets RMSNorm plus a low-rank data-dependent Sigmoid gate (which the report ties to alleviating attention sink). The kernel story is the second half of the contribution: by binding the DPLR factors to k (a = b = k), the chunkwise algorithm avoids two secondary-chunking steps and three matrix multiplications, running ~2× faster than general DPLR while remaining mathematically the classical delta rule. The whole model is NoPE — MLA layers carry no positional encoding and KDA's learnable decay acts as a multiplicative positional encoding — which also lets MLA convert to pure MQA at inference. The MoE (8/256 experts plus one shared, first layer dense) and the entire recipe (MuonClip, WSD, K2 corpus, annealing/long-context activation) are inherited from Moonlight and Kimi K2. Released with open kernels, vLLM integration, and checkpoints; the same KDA recurrence later becomes the linear-attention layer of Kimi K3, the atlas's next family record.
What the report claims is novel
Kimi Delta Attention (KDA): a gated-delta-rule linear attention with channel-wise (fine-grained) gating, extending Gated DeltaNet for more effective use of finite-state RNN memory [Abstract, §3]
A bespoke chunkwise-parallel algorithm based on a constrained Diagonal-Plus-Low-Rank (DPLR) transition, roughly 2× faster than general DPLR kernels (≈100% operator-efficiency gain) while staying consistent with the classical delta rule [Abstract, §3.2, §6.2]
The first hybrid linear-attention architecture to outperform full attention under fair comparisons across short-context, long-context and RL scaling regimes (1.4T-token matched runs) [Abstract, §1, §5]
A 3:1 KDA-to-MLA layerwise hybrid that reduces memory and KV-cache usage by up to 75% and achieves up to 6× decoding throughput at 1M context, positioned as a drop-in replacement for full-attention architectures [Abstract, §1, §4]
Open release of KDA kernels (flash-linear-attention), vLLM integration, and pre-trained plus instruction-tuned checkpoints, drop-in compatible with existing caching/scheduling interfaces [Abstract, §1]
Atlas assessment: KDA is a synthesis inside the delta-rule family rather than a new mixing family: Gated DeltaNet's recurrence with GLA-style channel-wise gates and DeltaNet's Householder/delta update, plus a genuinely useful kernel contribution (the a=b=k-constrained DPLR chunkwise algorithm, ~2× over general DPLR). As a system, Kimi Linear follows the inter-layer hybrid paradigm already present in the atlas via ai21-jamba (Transformer+Mamba 52B), but it is the first delta-rule-based (KDA) hybrid in the atlas and the first entry whose linear component beats the full-attention baseline under matched training. The NoPE-global-attention choice is shared with recent hybrids (Falcon-H, SwanGPT, and the report's own ablation vs RoPE) and is well motivated. The 'outperforms full attention' claim is internally consistent (identical recipe, 1.4T matched runs) but validated only at 48B/3B scale. The KDA mechanism is the direct ancestor of the KDA layers in moonshot-kimi-k3, whose report refines the same recurrence; the K3 report in this atlas does not credit Kimi Linear by name, but the mechanism is unambiguous.
Relation to the atlas
Lineage in
Gated DeltaNet (GDN) — KDA extends its gated delta rule with fine-grained channel-wise gating from Gated Delta Networks
Channel-wise (diagonal) gating from Gated Linear Attention (GLA) from GLA
Delta rule / fast-weight programming from DeltaNet (Householder-style rank-1 updates, WY representation) from DeltaNet
Backbone architecture following Moonlight (incl. Muon optimizer for scaling-law experiments) from Moonlight (Moonshot)
Pretraining recipe from Kimi K2: MuonClip optimizer, WSD schedule, annealing + long-context activation phase, K2 pretraining corpus from Kimi K2
RL algorithm from Kimi K1.5 (with truncated importance sampling and adaptive KL/mini-batch additions) from Kimi K1.5
ShortConv (lightweight depthwise convolution) on attention inputs from Gated Delta Networks / Mamba-style conv
Influence out
upstage-solar-open-2: KDA (Kimi Delta Attention) — gated delta-rule linear attention with fixed-size recurrent state for the 36 linear layers
Notable omissions
Layer count, hidden dimension, head count, and expert hidden dimension of the released 48B-A3B model are never printed (only scaling-law sizes 653M–1.7B active, Table 2)
Vocabulary size and tokenizer not described
Router gating function and expert load-balancing not described
MoE FFN activation function not stated (only the attention-path Swish)
Training precision, parallelism strategy, hardware, and total compute not disclosed
No license named for released checkpoints/kernels
Normalization placement (pre/post) not described; embedding tying not disclosed
MLA latent dimension not disclosed (only KDA's dk = dv = 128)
Exact decay function f(·) parameterization not specified in the text
Source extraction: arXiv HTML unavailable (arxiv.org/html 404; ar5iv fatal conversion error), so the PDF was used — inline math and some table cells are partially garbled in the stored text
Long-context results at 1M reported only for the 5.7T Instruct checkpoint (RULER), not the 1.4T matched runs
uniform 3:1 interleaving — 3 KDA (linear-attention) layers per 1 full MLA layer, repeated through the stack (N = 3, Figure 3); the first layer is dense (no MoE); 3:1 chosen by hybrid-ratio ablation[11]
RLVR on mathematics, code and STEM prompt sets pre-selected at moderate difficulty; truncated importance sampling to counter train/inference precision mismatch; dynamic KL penalty and mini-batch adjustment to avoid entropy collapse; PTX loss for general-capability retention[29]
hybrid cache: KDA layers keep a fixed-size recurrent state (dk × dv = 128 × 128 per head, independent of sequence length) instead of a growing KV cache; MLA layers (1 in 4) keep the low-rank latent KV cache and can be converted to pure MQA at inference under NoPE — up to 75% KV-cache reduction vs full MLA, up to 6× decoding throughput at 1M context
Quantisation shipped
none disclosed
Speculative decoding
none disclosed
Serving optimisations
open-source KDA kernels integrated with vLLM, drop-in compatible with existing caching and scheduling interfaces; prefill uses the FLOP-intensive chunkwise kernel, autoregressive generation switches to the recurrent kernel; hybrid I/O-bounded decoding approaches a 3:1 efficiency ratio vs full attention[31]
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
MMLU-Pro (4k)
51.0
Figure 1, Table 3
RULER (128k)
84.3
Figure 1, Table 5
HellaSwag (Base)
82.9
Table 3
ARC-Challenge (Base)
67.3
Table 3
Winogrande (Base)
78.6
Table 3
BBH (Base)
72.9
Table 3
MMLU (Base)
73.8
Table 3
TriviaQA (Base)
71.7
Table 3
GSM8K (Base)
83.9
Table 3
MATH (Base)
54.7
Table 3
EvalPlus (Base)
60.2
Table 3
CRUXEval-I-cot (Base)
56.6
Table 3
CRUXEval-O-cot (Base)
62.0
Table 3
CEval (Base)
79.5
Table 3
CMMLU (Base)
80.8
Table 3
BBH (Instruct)
69.4
Table 4
MMLU (Instruct)
77.0
Table 4
MMLU-Pro (Instruct)
67.4
Table 4
MMLU-Redux (Instruct)
80.3
Table 4
GPQA-Diamond (Avg@8, Instruct)
62.1
Table 4
LiveBench (Pass@1, Instruct)
45.2
Table 4
AIME 2025 (Avg@64, Instruct)
21.3
Table 4
MATH500 (Acc., Instruct)
81.2
Table 4
HMMT 2025 (Avg@32, Instruct)
12.5
Table 4
PolyMath-en (Avg@4, Instruct)
43.6
Table 4
LiveCodeBench v6 (Pass@1, Instruct)
26.0
Table 4
MRCR (128k)
29.6
Table 5
HELMET-ICL (128k)
90.0
Table 5
LongBench V2 (128k)
35.0
Table 5
Frames (128k)
58.8
Table 5
RepoQA (128k)
68.5
Table 5
Long Code Arena (128k)
37.1
Table 5
RULER@1M (Instruct, 5.7T)
94.8
Appendix D, Table 9
MMLU-Pro (Base, 5.7T)
54.8
Appendix D, Table 8
GPQA-Diamond (Avg@8, Instruct, 5.7T)
71.7
Appendix D, Table 9
Provenance references
stated · Title
stated · Title byline; Appendix A ('The project is developed at Moonshot AI'); header (github.com/MoonshotAI/Kimi-Linear)
stated · Title page: 'arXiv:2510.26692v2 [cs.CL] 1 Nov 2025'
stated · pipeline
stated · Abstract: 'we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints'; fn 2 (huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct)
stated · Abstract, §5.4 ('48 billion total parameters')
stated · Abstract, §5.4 ('3 billion active parameters per forward pass')
n/a · §4 (MLA layers cache a low-rank latent — no KV heads; KDA layers keep a recurrent state — no KV cache; NoPE MLA converts to pure MQA at inference, still no KV heads)
stated · §4 ('dk, dv represent the key and value head dimensions, which are set to 128 for all experiments')
stated · §4 (layerwise hybrid of KDA linear attention and full MLA global attention)
stated · §4 (q, k, v = Swish(ShortConv(·))); MoE FFN activation not described
stated · §5.4 ('Each model activates 8 out of 256 experts')
stated · §5.4 ('activates 8 out of 256 experts')
stated · §5.4 ('including one shared expert')
stated · §4, §5.4.1
n/a · §4 (no rescaling factor applies; no RoPE)
stated · §5.4.1 ('the final checkpoint supports a context length of up to 1 million tokens')
stated · §4 ('we apply NoPE to all full attention (MLA) layers'); §6.1 (KDA's data-dependent decay acts as a learnable multiplicative positional encoding, so no explicit PE anywhere)
n/a · §4 (no RoPE in the model; 'Kimi Linear (RoPE)' is an ablation baseline only, §5.4)
n/a · §4 (no RoPE)
stated · §4 (head-wise RMSNorm before the output projection)
stated · §5.4.1 ('All models are pretrained using a 4,096-token context window')
stated · §5.4.1, §4
stated · §5.4.1 ('an expanded total of 5.7 trillion tokens to match the pretraining tokens of Moonlight')
stated · §5.4.1
stated · §5.4.2 (multi-stage SFT: broad instruction-following data first, then scheduled reasoning-intensive data; dataset extends the Kimi K2 SFT data)
stated · §5.4.2 (same RL algorithm as Kimi K1.5, with truncated importance sampling, dynamically adjusted KL penalty and mini-batch size; RLVR on math/code/STEM prompt sets; no RLHF/PPO/DPO/GRPO named)