MiMo-V2-Flash is Xiaomi's 309B-parameter sparse MoE (15B active per token) built around a deliberately aggressive hybrid attention schedule. Eight hybrid blocks each stack five sliding-window-attention layers (128-token window, 64 query / 8 KV heads) with one global-attention layer (64 query / 4 KV heads), giving 39 SWA and 9 GA layers; only the very first block is global with a dense FFN. A learnable attention sink bias in the softmax denominator — following OpenAI's gpt-oss — lets each head dump surplus probability mass, which the authors show is what lets the 5:1 ratio and 128-token window match or beat an all-global baseline on general, long-context, and reasoning benchmarks. Position comes from partial RoPE (first 64 QK dimensions); the GA RoPE base is raised from 640,000 to 5,000,000 when the context is extended from 32K to 256K, while SWA keeps base 10,000. Every MoE layer routes each token through 8 of 256 experts (no shared experts, per-expert width 2048) under a sequence auxiliary loss plus expert-bias updates. Training runs FP8 mixed precision over 27T tokens in three stages (general at 32K, code/reasoning-heavy mid-training, 256K context extension). The headline post-training contribution is MOPD, multi-teacher on-policy distillation: domain-specialized RL teachers supply dense token-level reverse-KL rewards combined with GRPO-style outcome rewards so one student absorbs every teacher's peak capability. The lightweight 0.33B MTP blocks (dense FFN + SWA) double as a self-speculative draft model — ~3.6-token acceptance length, 2.6x decoding speedup — and are open-sourced alongside the weights.
What the report claims is novel
Hybrid SWA architecture: 5:1 local:global interleaving with a 128-token window and learnable attention sink bias, yielding ~6x KV-cache/compute reduction while matching or surpassing all-global baselines on general, long-context, and reasoning benchmarks [§1, §2.2]
Multi-Teacher On-Policy Distillation (MOPD): a three-stage post-training paradigm (SFT, domain-specialized RL/SFT teachers, on-policy distillation with dense token-level rewards plus ORM advantages) that lets the student master peak teacher capabilities without see-saw trade-offs [§4.1]
Lightweight MTP repurposed as a native draft model for self-speculative decoding (up to 3.6 acceptance length, 2.6x decoding speedup) and for enabling small-batch on-policy RL rollouts [§2.3, §5]
Competitive with DeepSeek-V3.2 and Kimi-K2 using 1/2-1/3 of their total parameters; leading open-source software-engineering results (73.4% SWE-Bench Verified, 71.7% SWE-Bench Multilingual) [Abstract, §4.5.2]
Long-context robustness: near-100% NIAH retrieval success from 32K to 256K and superior GSM-Infinite retention vs sparse-attention peers; surpasses larger full-attention models on LongBench V2 and MRCR [§1, §3.3.2]
RL infrastructure for ultra-scaled agentic training: Rollout Routing Replay (R3), fine-grained Data Scheduler with partial rollout, and Toolbox/Tool Manager for 300K+ agentic tasks [§4.6]
Open-sourced model weights and the three-layer MTP weights [Abstract, §1]
Atlas assessment: The block is a competent synthesis of known mechanisms rather than a new mixing family: the 5:1 SWA/GA schedule extends the Gemma-style local/global interleave, the learnable attention sink bias is taken from OpenAI gpt-oss (in atlas), and FP8 mixed precision + MTP + GRPO-style ORMs follow the DeepSeek-V3/R1 playbook. The genuinely distinctive contributions are (a) the aggressive 128-token window at 5:1 validated with sink bias, and (b) MOPD, a named multi-teacher on-policy distillation paradigm — the same acronym and concept later appear in Moonshot Kimi K3 (arXiv 2607.24653, in atlas), which postdates this report; MiMo's January 2026 publication predates it, so the atlas now holds two independent MOPD claims. The 'rivals DeepSeek-V3.2/Kimi-K2 at 1/2-1/3 parameters' claim is internal and not independently verifiable. Report is unusually silent on normalization, activation function, router function, vocab size, and training hardware — architecture sections that peers almost always fill.
Relation to the atlas
Lineage in
Learnable attention sink bias in the softmax denominator from OpenAI gpt-oss
FP8 mixed-precision framework (BF16 attention output projections and embeddings/output head, FP32 router) from DeepSeek-V3
Multi-Token Prediction (MTP) training objective from DeepSeek-V3 / MiMo-7B
GRPO-style outcome-reward advantages combined with distillation rewards from DeepSeek (GRPO / DeepSeekMath)
Influence out
No descendants recorded yet.
Notable omissions
Normalization type and pre/post placement never named (no RMSNorm/LayerNorm mention)
FFN activation function never named (no SwiGLU/GELU)
Vocabulary size and tokenizer not disclosed
MoE router function not described (only bias/aux-loss hyperparameters)
Training hardware and compute (FLOPs/GPU-hours) not disclosed
Training parallelism strategy not disclosed
No license named for the weights release
Embedding/output-head tying not disclosed
MoE expert granularity not discussed
SWA/GA per-head dims (192/128) do not match hidden 4096 / 64 heads = 64, and the discrepancy is unexplained
RL reward-model training details (beyond GRPO mention) deferred
Context extension relies on RoPE base change only; no comparison to YaRN/NTK given
sequence-wise MoE auxiliary loss (coefficient 1.0e-5 all stages) plus per-expert bias updates (bias update factor 0.001 in Stages 1-2, 1.0e-5 in Stage 3; 1.0e-4 during SFT)[18]
diverse high-quality corpus: public web content, books, academic papers, code, mathematics, and broader STEM; deliberate shift toward long-range dependencies (long-form web documents, repository-level code, pull requests, issues, commit histories); ~5% synthetic reasoning data added in Stage 2; pipeline largely follows MiMo-7B[29]
Curriculum / staging
three stages: Stage 1 (0-22T) general corpus at 32K context; Stage 2 (22-26T) code upsampling + ~5% synthetic reasoning data; Stage 3 (26-27T) context extension to 256K with long-range data upsampled[30]
Optimizer
AdamW with beta1=0.9, beta2=0.95, weight decay 0.1, gradient clipping max norm 1.0[31]
LR schedule
Stage 1: linear warmup 0 -> 3.2e-4 over 50B tokens, constant 3.2e-4 for 12T, cosine decay to 1.0e-4 over 10T; Stage 2: cosine 1.0e-4 -> 3.0e-5 over 4T; Stage 3: cosine 3.0e-5 -> 1.0e-5[31]
Batch schedule
batch size warms up linearly to 2048 over the first 500B tokens and stays constant through Stages 1-2; fixed batch size 256 in Stage 3; SFT batch size 128[18]
Precision
FP8 mixed precision (DeepSeek-V3-style): BF16 for attention output projections and embedding/output head, FP32 for MoE router; FP8 used for both training and inference[32]
large-scale RL: non-agentic RL on verifiable domains (programmatic verifiers + LLM judge, rubric-based judge for subjective qualities); agentic RL across code (120K environments), terminal (~30K), web development, search (150K), function-calling (50K) agents; all consolidated via MOPD distillation[35]
Distillation
Multi-Teacher On-Policy Distillation (MOPD): student samples from its own distribution and receives dense token-level reverse-KL rewards from domain-specialized teacher policies (RL or SFT trained), combined with ORM advantages; training-inference importance sampling with token discarding (IcePop2025)[36]
hybrid: 39/48 layers are SWA with a 128-token window so their KV cache is bounded by the window; 9 GA layers keep full cache; report claims nearly 6x reduction in KV-cache storage and attention computation for long contexts[38]
self-speculative decoding with MTP as native draft model: single MTP head during pre-training, replicated to K=3 jointly-trained heads in post-training; up to 3.6 average acceptance length and 2.6x decoding speedup[40]
stated · §3.1 Stage 3 ('extend the model's context window to 256K tokens')
stated · §2.1 ('Rotary Positional Embedding (RoPE) is partially applied to the first 64 dimensions of query and key')
stated · §3.2 (Long Context Extension)
stated · §2.1 (RoPE applied to first 64 QK dimensions)
stated · §2.2 (learnable attention sink bias s_sink added to the softmax denominator per head, following gpt-oss)
stated · §3.1 Stage 3 (context extended to 256K in the 26-27T range)
stated · Abstract, §1 ('native 32k context length and subsequently extended to 256k')
stated · §3.1, §3.2
stated · Abstract, §3 ('27 trillion tokens')
stated · §3, §3.1
stated · §3.1
stated · §3.2
stated · §2.1, §4.6
stated · §4.1 Stage 1, §4.2 (millions of samples across conversation, reasoning, coding, agent tasks; thinking and non-thinking modes)
stated · §4.4 ('we combine the advantages of MOPD with other types of advantages, such as those computed using Outcome Reward Models (ORMs), including GRPO [shao2024deepseekmath]')
stated · §4.3, Table 8
stated · §4.1 Stage 3, §4.4 (Eq. 5-9)
n/a · modality.type (text-only)
stated · §1, §2.1
stated · §4.6 ('We adopt FP8 for both training and inference')
stated · §2.3, Abstract, §5
stated · report is silent on serving optimizations beyond MTP-based speculative decoding