Kimi K3 is a 93-layer, 2.78T-parameter sparse MoE (104.2B active per token) that mixes information along three axes at once. For sequence length, each block interleaves three Kimi Delta Attention layers — a delta-rule linear attention with a channel-wise forget gate, lower-bounded (g_min = −5) decay that keeps chunkwise computation inside the BF16 range, and full-rank output gating — with one Gated MLA layer (DeepSeek-V2-style latent KV, NoPE, full-rank gate) for unrestricted global interaction; 23 blocks plus a final Gated MLA give 69 KDA / 24 MLA layers. For depth, Attention Residuals let each layer attend over prior layer outputs via a learned pseudo-query, partitioned into 8 blocks of 12 layers (plus a partial final block) to bound memory. For width, Stable LatentMoE routes each token through 16 of 896 fine-grained experts operating in a 3584-wide latent space (two full-width shared experts carry common transformations), stabilized by RMSNorm before the up-projection, the bounded SiTU-GLU activation, and Quantile Balancing, an auxiliary-loss-free bias update that sets each expert's bias from the quantile of router-score margins. Native vision arrives through MoonViT-V2, a 27-layer ViT trained from scratch with the next-token objective and projected into the shared embedding space; the 1M context is reached by progressive curriculum (8K→64K pre-training, 256K→1M cooldown) with no positional-encoding changes — the model is fully NoPE, with position carried implicitly by KDA's decay. Post-training runs SFT → domain/effort-specialized RL (general, agentic, coding × low/high/max) → multi-teacher on-policy distillation into one model, with MXFP4 QAT from SFT onward.
What the report claims is novel
First open 3T-class model: a 2.8T-parameter native multimodal MoE with 104B activated parameters and a 1M-token context window [§1, §8]
Hybrid attention: three KDA layers (delta-rule linear attention with channel-wise forget gate, lower-bounded decay, full-rank output gating) per one Gated MLA layer, giving efficient long-sequence mixing with periodic global interaction [§2.1]
Attention Residuals (AttnRes) with a Block variant: selective, learned attention over prior layer outputs along the depth axis [§2.2]
Stable LatentMoE: 896 routed experts with 16 active per token (56× sparsity), stabilized by RMSNorm, SiTU-GLU, and Quantile Balancing [§2.3]
MoonViT-V2 vision encoder trained entirely from scratch with next-token prediction, matching SigLIP-initialized baselines without contrastive pre-training [§2.4]
RL across general, agentic, and coding domains at three reasoning-effort levels, consolidated into a unified model via Multi-Teacher On-Policy Distillation [§4.1]
Infrastructure: FlashKDA kernels and KDA Context Parallelism, MoonEP perfectly balanced expert-parallel training, million-token agentic RL with AgentENV resumable sandboxes, and KDA-aware prefix caching for serving [§5]
Approximately 2.5× improvement in overall scaling efficiency over Kimi K2 [Abstract, §3.2]
Atlas assessment: The block is a deliberate three-axis hybrid (token/depth/channel). vs DeepSeek-V3, the closest atlas peer: same MLA + shared/routed MoE lineage, but K3 replaces the all-softmax-attention backbone with a 3:1 KDA/MLA hybrid, scales routing to 896/16 experts in a latent width (vs 256/8 at full width), and replaces DeepSeek-V3's sign-based bias updates with quantile-based QB. The lower-bounded decay parameterization is a numerical-stability refinement of the delta-rule recurrence family, not a new mixing family. The claimed 2.5× scaling-efficiency gain over Kimi K2 is internal to the report and not independently verifiable. From-scratch vision training with a next-token objective (no contrastive initialization) is the first such claim in the atlas; the report's own evidence is that it matches the SigLIP-initialized baseline with better gradient stability. Depth-wise AttnRes is a genuinely new axis for the atlas (no peer describes attention over layers).
Relation to the atlas
Lineage in
Multi-head Latent Attention (MLA) — latent KV compression retained in the Gated MLA layers from DeepSeek-V2
Auxiliary-loss-free bias-based load balancing (QB replaces the fixed-step sign update with a quantile update) from DeepSeek-V3
LatentMoE — routed experts operating in a compact latent space from LatentMoE (not in atlas)
EAGLE-3-style speculative decoding — draft model fine-tuned from the MTP layer from EAGLE-3 (not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
No total pre-training token count disclosed (tokens-per-parameter ratio retuned but never printed)
No training compute (FLOPs/GPU-hours) or training hardware disclosed
Master training precision (BF16/FP8) not stated
Tokenizer algorithm not described (only the 160K vocab size appears in Table 1)
Batch size and peak learning rate values not disclosed
No license named for the weights release
Normalization placement (pre/post) not described
Input/output embedding tying not disclosed
Head dimension and KDA state dimensions (d_k, d_v) not disclosed
The single 'dense layer' counted in Table 1 is never explained in the text
RL algorithm details deferred to Kimi K2.5 ('follows the algorithm in Kimi K2.5') without naming the method
Third-party benchmark scores (Table 5, several Table 2 rows) are cited from Artificial Analysis / Vals AI / official leaderboards rather than measured in-house
per block: 3 KDA layers then 1 Gated MLA layer, repeated throughout the backbone; 69 KDA + 24 MLA layers total, with an additional Gated MLA at the end of the backbone so the final layer is global attention[11]
sigmoid router scores with biased top-k selection; mixture weights renormalized over selected experts (bias excluded from weights)[17]
Load balancing
Quantile Balancing (QB): auxiliary-loss-free per-expert bias set from the (1 − k/n)-quantile of router-score margins, estimated from per-expert histograms over the global batch; bias frozen at inference[18]
Expert granularity
fine-grained LatentMoE: 896 routed experts operate in a compact latent width 3584 (0.5× hidden); 2 full-width shared experts[19]
curated corpus over four text domains — Web Text, Code, Mathematics, Knowledge — plus a large-scale vision corpus (captions, interleaved image–text documents, OCR, perception, video, visual coding); knowledge and mathematics corpora rephrased with style/perspective-diverse prompting and fidelity verification (Kimi K2 recipe); substantial programmatic multimodal data coupling code with rendered visuals (SVG, 3D assets, Webpage, Game, CAD schematics); coordinate supervision in absolute and normalized formats; pipelines build on Kimi K2/K2.5[28]
Curriculum / staging
native multimodal training from the start (vision and language jointly optimized, interleaved tokens under one next-token objective); progressive context extension (8K → 64K pre-training; 256K → 1M cooldown); long-context data upsampled and synthesized (permuted/concatenated multimodal documents) so tasks require full-context attention[29]
Optimizer
Per-Head Muon (Muon with per-head Newton–Schulz orthogonalization for attention projections) with weight clipping (from Kimi K2)[30]
LR schedule
cosine decay with 1% linear warmup (scaling-law study favored cosine over Warmup Stable Decay under per-schedule optimal hyperparameters); weight decay 0.1 throughout[31]
Batch schedule
not disclosed
Precision
not disclosed
Parallelism
Pipeline Parallelism with virtual stages, Expert Parallelism (MoonEP), ZeRO-1 Data Parallelism, Pipeline ZeRO-2 gradient sharding (CPU offload), and Context Parallelism (KDA Context Parallelism / KCP); all-to-all dispatch overlapped with computation; P2P Muon orthogonalization[32]
multi-effort RL: three domain experts (general, general agents, coding agents) × three reasoning-effort levels (low/high/max) with per-problem token-budget control and τ annealing; long-horizon agentic RL at up to 1M context (partial rollouts, persistent sandbox states); verifiable environments: knowledge-graph-guided task synthesis, autonomous execution tasks (AET), kernel optimization, web development, personal assistant workflows, vision-in-the-loop tool use[35]
Distillation
Multi-Teacher On-Policy Distillation (MOPD): the nine domain × effort expert policies are consolidated into the unified model via per-token on-policy distillation rewards (clipped log-ratio), with effort-conditioned training[36]
hybrid cache: KDA keeps a fixed-size recurrent state (d_k × d_v per head) instead of a growing KV cache; Gated MLA keeps the low-rank latent KV cache (per-token latent, reconstructed keys/values); both managed jointly in a unified paged prefix-cache pool with KDA checkpoints at 512-token hash boundaries[39]
Quantisation shipped
MXFP4 MoE expert weights with MXFP8 activations at deployment (QAT throughout SFT and RL); all non-expert components (attention projections, latent MoE projections, shared experts, routers) remain in higher precision[40]
Speculative decoding
EAGLE-3-style draft model fine-tuned from the pre-trained MTP layer (draft mirrors one backbone block; 7-step unroll; LK loss directly maximizes acceptance rate; draft input fuses features from AttnRes blocks 1, 4, and final)[40]
stated · §2.2 (AttnRes: each layer attends over prior layer outputs with a learned pseudo-query; Block variant partitions 93 layers into 8 blocks of 12 (partial final block), 9 total blocks counting the embedding)
stated · §2.3.2, Table 1
stated · Table 1 (Routed Experts)
stated · Table 1 (Experts Active per Token)
stated · §2.3 (N_s = 2 full-width shared experts in every layer)
stated · §4.1.1 (three-stage pipeline begins with SFT; expanded agentic SFT dataset synthesized with prior Kimi models plus multi-stage verification and human-in-the-loop annotation; XTML chat template; QAT from the SFT stage onward)
stated · §4.1.2 (policy optimization follows the Kimi K2.5 algorithm with per-token regularization for stale-data tolerance; agentic generative reward model (GRM) with tournament-style group reward and binary comparisons, verbosity-budgeted; report never names RLHF/PPO/DPO/GRPO)
stated · §4.1.2, §4.2
stated · §4.1.3
stated · §2.4 (text, images, and videos processed by a single shared backbone within one context)
stated · §2.4 ('natively multimodal… no post-hoc modality-alignment stage'); MoonViT-V2 (27-layer ViT, ~0.4B/401M params, patch 14, 12 heads) trained from scratch with next-token prediction + lightweight MLP projector; pixel-shuffle 2×2 token downsampling (up to 3584×3584 px); factorized intra-frame spatial / inter-frame temporal attention with temporal pooling; language and vision jointly optimized from the start of training (§3.3)