Mellum 2 is a 28-layer, 12B-parameter MoE (2.5B active per token) built as a scaled-down Qwen3-MoE: hidden 2,304, 32 query heads / 4 KV heads (GQA) at head dim 128 with QK-Norm (RMSNorm), pre-RMSNorm, SiLU-gated MLPs, RoPE base 500,000, and 64 experts per layer (expert intermediate 896) with top-8 routing, no shared expert, untied 98,304-token embeddings. Three deployment-oriented modifications distinguish it: a 3:1 sliding-window pattern (1,024-token window on three of every four layers, full attention on the fourth), a single MTP head (one extra transformer layer, loss weight 0.1) that doubles as a speculative-decoding draft, and a latency budget — matching Qwen2.5-7B on one H100 — against which every choice was ablated (dense variants, MLA, hybrids, shared experts, and auxiliary-loss-free balancing were all tried and rejected). Pre-training runs ~10.65T tokens in three phases (web 70→44→23%, code 23→42→59%) with a 50/10/50% FIM schedule, distributed Muon (Moonlight config) under BF16+FP8 hybrid precision, and a Warmup-Hold-Decay schedule with linear decay to zero. The 128K context comes from layer-selective YaRN — frequency re-mapping applied only to the global layers, per the Gemma-3/OLMo-3 recipe — over a ~117B-token extension run. Post-training splits into two variants from one long-context base: Instruct (direct answers, ~47B SFT tokens) and Thinking (explicit reasoning traces, ~167B SFT tokens, last-turn loss, turn unfolding), each refined by GRPO-style RLVR on verifiable math/code/tool tasks with IcePop truncation and a concision penalty. Released under Apache 2.0 with an unusually candid discussion of what was ablated and why.
What the report claims is novel
An efficiency-aware architecture: every design choice (MoE vs dense, 8-of-64 expert sparsity, 4-KV-head GQA, 3:1 SWA, single MTP head) selected by ablation against a fixed inference budget — matching the single-H100 latency of Qwen2.5-7B — yielding a 12B/2.5B-active model that matches Qwen2.5-7B sync latency and exceeds it by 21% in throughput [§1, §2, §6, §7]
A three-phase pre-training curriculum on ~10.6T tokens ('web early, curated late'): the mixture shifts from diverse web content to curated code and math (code ratio 23% → 42% → 59%) with batch-size doubling and a capability-sharpening phase that decays the LR linearly to zero [§1, §3.2]
A Muon + FP8 hybrid training recipe at production scale (distributed Muon in the Moonlight configuration, tensorwise FP8 with most-recent amax, FP32 gradient reduction) with training-stability observations across the full ten-trillion-token run [§1, §3.4]
Long-context extension to 131,072 tokens via layer-selective YaRN — frequency re-mapping applied only to the global (full-attention) layers — with empirical findings on data-mix transfer (Longmino mix failed to replicate; ~30B tokens sufficient, 117B run for router equilibration) and MoE router dynamics during extension [§1, §4]
Two post-trained variants (Instruct and Thinking) from a shared long-context base via SFT followed by RLVR with a GRPO variant (IcePop truncation, clip-higher, no KL anchor, verifiable rewards only) [§1, §5]
Open release under Apache 2.0: base, instruct, and thinking checkpoints (plus a pre-long-context base and SFT checkpoints) with the full architecture/data/recipe documentation [§1, §7]
Atlas assessment: Architecturally derivative by design — the report openly frames Mellum 2 as the Qwen3-MoE recipe (64 experts/8 active, GQA, QK-Norm, RMSNorm, SiLU, RoPE) plus three known components: Mistral-style sliding-window attention (3:1, window 1,024), Gloeckle-style MTP, and Gemma-3/OLMo-3-style layer-selective context scaling (with YaRN instead of PI). No mechanism is new to the atlas. The genuine contributions are process-level: every choice is tied to an ablation against a fixed single-H100 latency budget (with rejected alternatives documented — MLA, dense sweeps, hybrid Gated DeltaNet backbones, shared experts, aux-loss-free balancing), the three-phase curriculum with an unusually transparent FIM schedule, and a very complete RL recipe (IcePop, clip-higher, concision penalty). vs atlas peers it sits between mistral-mixtral-8x7b (MoE, no SWA/GQA/MTP) and google-gemma-3 (SWA + layer-selective scaling but dense, 5:1 pattern); it is the first small-MoE coding-assistant record and the first Qwen3-MoE-style record (the existing qwen-qwen3-moe record predates the current schema).
Relation to the atlas
Lineage in
Qwen3-MoE recipe — 'closely follows the Qwen3-MoE recipe' (MoE-in-every-layer, GQA, QK-Norm, RMSNorm, SiLU, RoPE); scaled down from Qwen3-30B-A3B from Qwen3 (not in atlas)
Sliding Window Attention (3:1 pattern, 1,024-token window) from Mistral 7B / Longformer (not in atlas)
Multi-Token Prediction head (auxiliary objective + draft model for speculative decoding) from Gloeckle et al. (not in atlas)
Layer-selective context scaling — extension applied only to global-attention layers ('first reported in the Gemma 3 technical report… subsequently adopted by OLMo 3'); Mellum 2 uses YaRN as the scaling method from Gemma 3 / OLMo 3
Muon optimizer in the distributed Moonlight configuration from Moonlight (not in atlas)
GRPO recipe elements — token-level loss, DAPO-style clip-higher and soft overlong penalty, Dr. GRPO-style leave-one-out baseline, IcePop truncation from DeepSeekMath-GRPO / DAPO / Dr. GRPO (not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
Total and active parameter counts are approximate only (≈12B / ≈2.5B); no exact figure or per-component breakdown
No training compute (FLOPs/GPU-hours) disclosed
Tokenizer algorithm not named (custom tokenizer, shared with Mellum-4B)
Partial-RoPE configuration not stated (RoPE base 500,000 given, dimension fraction not)
YaRN scaling factor for the context extension not stated
MTP head parameter count not stated (removed at inference, so deployment cost is clean, but training overhead beyond the stated 7% is not quantified)
Pretraining batch-size ramp (2,048→4,096 sequences) is disclosed but no token-level batch in FLOPs
The report acknowledges weaknesses: broad world knowledge (GPQA Diamond 40.9 vs 79.8 for Qwen3.5-9B), BS-Bench false-premise pushback (14-24 vs 56-70), and a HarmBench regression after RL (8.4 → 23.1)
RULER QA subsets were depressed by a prompt-formatting issue during the extension stage; absolute long-context scores are flagged by the authors as conservative
Only seven of eighteen MultiPL-E languages are evaluated
top-8 expert selection; router operates in FP32; dropless routing (no expert capacity factor, no token dropping)[23]
Load balancing
global-batch auxiliary load-balancing loss (coefficient 1e-3) plus router z-loss (coefficient 1e-3); global-batch balancing chosen over per-sequence balancing for flexibility (per-sequence gave marginally better loss on short runs); aux-loss coefficient reduced to 1e-4 during SFT; auxiliary-loss-free bias balancing (DeepSeek-V3 style) considered but rejected for ecosystem fit[24]
Expert granularity
fine-grained: 64 experts per layer with 8 active (12.5% activation ratio), expert intermediate size 896; scaled down proportionally from the Qwen3-30B-A3B architecture
layer-selective YaRN: frequency re-mapping applied only to global (full-attention) layers, sliding-window layers keep original RoPE (Gemma 3 / OLMo 3 recipe); ~117B tokens of long-context training (3,500 iterations, WHD schedule, peak LR 3e-5, 500 decay iterations); RULER @64K 0.64 vs 0.52 uniform θ-bump vs 0.33 unchanged-θ[34]
Tokenizer
Algorithm
not disclosed
Notes
Custom tokenizer with a 98,304-token vocabulary, identical to the tokenizer used in Mellum-4B; designed to provide strong coverage of programming-language tokens and technical terminology[35]
Three categories: (1) web and general knowledge — large-scale synthetic web corpora from Common Crawl, educational web content and PDFs, multilingual reasoning/QA datasets, curated knowledge (SFT data, STEM instruction data, Wikipedia rewrites, synthetic encyclopedic articles); (2) source code — raw permissively licensed code deduplicated at file level, code extracted from Common Crawl, and synthetic/derived datasets (summarization, functionality extensions, translations, test generation, commit messages, task descriptions, QA, rewriting, review, transpilation, educational explanations); (3) mathematical data — math SFT data, quality-tiered math web content, permissively licensed textbooks, math instruction data; raw code corpus seen for 3 epochs (~958B tokens), no dataset repeated more than 4×[37]
Curriculum / staging
Three-phase curriculum aligned with the WHD schedule: Phase 1 Foundation (~6.18T tokens, 58%; web 70% / code 23% / math 6%; warmup→hold), Phase 2 Quality Uplift (~2.79T, 26.2%; web 44% / code 42% / math 14%; hold), Phase 3 Capability Sharpening (~1.69T, 15.9%; web 23% / code 59% / math 18%; decay); FIM rate schedule 50% (all data) → 10% → 50% (code files only); global batch size ramps 2,048 → 4,096 sequences[38]
Optimizer
Distributed Muon (Moonlight configuration: spectral scale mode, extra scale factor 0.2, Newton-Schulz iterations 5, momentum 0.95, Nesterov momentum, blockwise TP mode) with Adam (beta1=0.9, beta2=0.95, epsilon=1e-8) for embedding and output layers; weight decay 0.1; gradient clipping 1.0[39]
LR schedule
Warmup-Hold-Decay (WHD): linear warmup over 2,000 steps to peak 3e-4, held through Phases 1-2, linear decay to zero over 49,306 steps (~15% of training, Phase 3); linear decay-to-zero chosen over cosine-to-nonzero minimum[40]
Batch schedule
Global batch size ramps linearly from 2,048 to 4,096 sequences during the initial phase; at full batch each step processes ~33.6M tokens (4,096 × 8,192); micro-batch size 2; sequence length 8,192[41]
Precision
BF16 base precision with FP8 hybrid mixed precision (tensorwise FP8 recipe, most-recent amax algorithm); gradient reduction in FP32[42]
Parallelism
Expert parallelism 8 (each GPU hosts 8 of 64 experts), tensor parallelism 1, pipeline parallelism 1, gradient reduction and parameter gather overlapped with computation; SFT adds context parallelism 8[43]
Hardware
32 nodes × 8 H200 GPUs (migrated to a 16-node cluster mid-training with effective batch size fixed); RL on a Kubernetes cluster of H200 nodes (small training group + larger generation group); Megatron-LM / Megatron-Bridge, NeMo-RL, vLLM, Ray[44]
Thinking variant: emits an explicit reasoning trace before the final answer; SFT loss on the last assistant turn + reasoning field only (multi-turn conversations unfolded into up to 5 samples); RLVR with a harder thinking mix (difficulty-filtered long-form math subset); max total sequence length 40,960 tokens for thinking RL rollouts[47]
GQA: KV cache proportional to 4 KV heads; the 3:1 SWA pattern bounds per-token KV reads on window layers to the 1,024-token window; KV-cache size under high concurrency was the dominant design driver for choosing 4 KV heads
Quantisation shipped
dynamic FP8 model quantization for serving (vLLM FP8 serving on a single H100 80GB)[49]
Speculative decoding
single Multi-Token Prediction (MTP) head: one additional transformer layer predicting one future token, trained with scaled loss α=0.1, removed at evaluation/inference, and used as a built-in draft model for speculative decoding[50]
Serving optimisations
vLLM serving on a single H100 (80GB); matches Qwen2.5-7B sync-mode latency (192 vs 193 tokens/s) and exceeds it by 21% in throughput mode (5,179 tokens/s; 20.2 req/s sustained vs 16.7 for Qwen2.5-7B and 11.3 for Qwen3-8B); workload shape ISL/OSL = 2,304/256 tokens[51]
stated · Title ('Mellum 2 Technical Report'), Abstract
stated · Title page byline
stated · arXiv header line 16 ('arXiv:2605.31268v1 [cs.CL] 29 May 2026'); title page stamps 'v1.0 · May 2026'
stated · pipeline
stated · Abstract ('released… under the Apache 2.0 license'); §1
stated · Abstract ('We release the base, instruct, and thinking checkpoints'); §1 ('We release base, instruct, and thinking checkpoints under the Apache 2.0 license, together with this report'; 'we release a base model before the long context extension and SFT checkpoints')
stated · §2.1.4, §2.2 (GQA with a 3:1 Sliding Window Attention pattern: 3 of every 4 layers use a 1,024-token sliding window, the remaining layer uses full attention; 32 query heads / 4 KV heads, QK-Norm, RoPE)
stated · §2.1.4, §2.2, Table 2
stated · §2.2 ('SiLU-gated MLPs'), Table 2 ('Activation: SiLU (gated)')
stated · §2.2 ('64 routed experts per layer'), Table 2
stated · §2.2 ('8 active per token (top-8 routing)'), Table 2
stated · §2.2 ('no shared expert'), Table 2 ('Shared expert: None'); Appendix A.2 (a shared expert was ablated and dropped: 'no measurable quality gain… consistently hurt inference performance')
stated · §2.2, §3.4.5, Table 11
stated · §3.4.5, Appendix A.2, §5.1.2, Table 11
stated · §4.1 (layer-selective YaRN: frequency re-mapping applied only to the global (full-attention) layers; sliding-window layers keep their original RoPE parameters — recipe first reported in Gemma 3 and adopted by OLMo 3)
stated · §4 ('extend the effective context length of Mellum 2 from the 8,192-token training context to 131,072 tokens (128K)'), Table 2
stated · §2.2 ('Rotary Position Embeddings (RoPE) with base θ=500,000'), Table 2
stated · §2.2 ('RoPE with base θ=500,000'), Table 2
stated · §2.2 ('pre-RMSNorm (ε=10^-6)'), Table 2; QK-Norm is also RMSNorm (Table 2: 'QK-Norm: Yes (RMSNorm)')
stated · §2.2 ('pre-RMSNorm')
stated · §2.2 ('QK-Norm applied to the query and key projections'), Table 2 ('QK-Norm: Yes (RMSNorm)')
stated · §4 (native 8,192-token pre-training context extended to 131,072 in the long-context stage; SFT and RL train on packed 131,072-token sequences — §5.1.2, Table 6)
stated · §4 ('to 131,072 tokens (128K)'); Abstract ('extended to a 128K context window'); Table 2 ('8,192 / 131,072⋆ after the long-context extension stage')
stated · §5.1 (two SFT variants — Instruct and Thinking — from the same long-context YaRN checkpoint and data mix, differing in chat template, reasoning-trace handling, and loss masking; packed 131,072-token sequences, 3 epochs, peak LR 3e-5 cosine-decaying to 3e-6, Muon optimizer, MoE aux-loss coefficient reduced to 1e-4, EP=8 + CP=8 on 16×8 H200; Instruct run ≈47B tokens, Thinking run ≈167B tokens; MTP head retained with α=0.1)
stated · §5.2.3 (GRPO variant: token-level loss, leave-one-out advantage baseline without std normalization, asymmetric clip-higher [1-0.2, 1+0.28], IcePop band [0.5, 5.0], KL coefficient 0, DAPO-style soft overlong penalty, ARLCP-style concision penalty for Instruct; RLVR rather than RLHF — every prompt has a programmatic verifier; Instruct 500 steps at max seq 16,384, Thinking 100 steps at max seq 40,960; AdamW peak 1e-6 → 1e-7)