Qwen3's MoE branch pairs a deliberately conventional transformer backbone with an aggressive sparsity story. Qwen3-235B-A22B and Qwen3-30B-A3B share the dense branch's Qwen2.5-style stack — GQA attention with QK-Norm, SwiGLU, RoPE with the base raised to 1,000,000 via ABF, pre-RMSNorm — but replace every FFN with a fine-grained MoE: 128 total experts with 8 activated per token and, unlike Qwen2.5-MoE or DeepSeek-V3, no shared expert at all. The report discloses no hidden or per-expert dimensions, no router function, and no embedding tying, so the block's economics are only partially visible; what is stated is that load is balanced by a global-batch load-balancing auxiliary loss (Qiu et al., 2025). Context reaches 128K via YaRN + Dual Chunk Attention over the 32,768-token stage-3 training length. The 235B flagship activates 22B parameters per token — the same active count as DeepSeek-V3 at roughly a third of its total parameters — and the family's headline claims sit in post-training: one model that switches between thinking and non-thinking modes via /think and /no_think chat-template flags, a thinking-budget mechanism that halts reasoning at a user-defined threshold, and strong-to-weak distillation that builds the lightweight members (including the 30B-A3B MoE) at roughly 1/10 of the four-stage RL GPU hours. The flagship itself runs the full pipeline: long-CoT cold start, GRPO reasoning RL on 3,995 query-verifier pairs, thinking-mode fusion SFT, then general RL with rule-based and model-based rewards.
What the report claims is novel
Flagship MoE Qwen3-235B-A22B (235B total / 22B activated) achieves state-of-the-art overall performance among open-source models in both thinking and non-thinking modes and is highly competitive with closed-source leaders (OpenAI-o1, Gemini2.5-Pro, GPT-4o) [§1, §4.6]
MoE design with 128 total experts and 8 activated per token via fine-grained expert segmentation, excluding shared experts entirely (unlike Qwen2.5-MoE), balanced by a global-batch load balancing loss [§2]
Unified thinking / non-thinking modes in a single model with /think and /no_think chat-template flags, plus a thinking budget mechanism that adaptively controls inference-time reasoning compute [§1, §4.3]
Strong-to-weak distillation builds lightweight models (including the Qwen3-30B-A3B MoE) with better performance and ~1/10 the GPU hours of the four-stage RL pipeline [§4.5, §4.7]
Pre-trained on 36 trillion tokens covering 119 languages and dialects (up from 29 in Qwen2.5), with instance-level data-mixture optimization via a multilingual annotation system [§1, §3.1]
Qwen3 MoE base models reach performance similar to Qwen3 dense base models with only 1/5 of the activated parameters on the same pre-training data, and outperform Qwen2.5 MoE base models with less than 1/2 the activated parameters [§3.3]
Atlas assessment: Architecturally the MoE branch is conservative — a Qwen2.5-lineage GQA + SwiGLU + RoPE + pre-RMSNorm transformer (plus QK-Norm) with fine-grained routing. Against DeepSeek-V3, the closest atlas peer (671B total / 37B active, 256 routed experts with 1 shared expert and aux-loss-free bias balancing), Qwen3-MoE is the mirror-image design: fewer and fatter experts (128 total / 8 active), no shared expert at all, and an explicit global-batch load-balancing auxiliary loss rather than an aux-loss-free scheme. Both choices (fine-grained segmentation, global-batch loss) are borrowed and cited (Dai et al. 2024; Qiu et al. 2025), not new mechanisms. The genuinely distinctive contributions sit in post-training — thinking/non-thinking mode unification, the thinking budget, and strong-to-weak distillation — and in scale economics: 22B active parameters matching DeepSeek-V3's active count at 35% of its total parameters. The report's efficiency claims (1/5 activated params vs dense, 1/10 GPU hours for distillation) are internal comparisons, not independently verifiable.
Relation to the atlas
Lineage in
Fine-grained expert segmentation for MoE FFNs from Qwen2.5-MoE
QK-Norm on the attention mechanism (QKV-bias removed) from Dehghani et al., 2023 (QK-Norm)
RoPE base-frequency increase from 10,000 to 1,000,000 via ABF from Qwen2.5
YaRN context extension with Dual Chunk Attention (DCA) from Qwen2.5
Global-batch load balancing loss for expert specialization from Qiu et al., 2025 (not in atlas)
GRPO for Reasoning RL from Shao et al., 2024 (GRPO)
Influence out
No descendants recorded yet.
Notable omissions
Hidden/embedding dimensions and FFN (per-expert) dimensions never disclosed; Table 2 lists only layers, heads, expert counts, and context length
Router function not described (only the global-batch load balancing loss is named)
Input/output embedding tying not stated for the MoE models (Table 2 has no tie-embedding column)
Optimizer not named
Training precision, parallelization strategy, hardware, and total compute (FLOPs/GPU-hours) not disclosed
No per-variant pre-training token allocation for the MoE models (36T total for the whole family)
No quantisation formats, serving optimisations, or speculative-decoding strategy disclosed
Shared-expert removal is asserted without ablation evidence
Head dimension not derivable (hidden dim undisclosed)
119 languages and dialects; domains include coding, STEM, reasoning tasks, books, multilingual texts, and synthetic data; PDF-like documents transcribed by Qwen2.5-VL with Qwen2.5 refinement; trillions of synthetic tokens generated by Qwen2.5, Qwen2.5-Math, Qwen2.5-Coder (textbooks, QA, instructions, code); instance-level data-mixture optimization via a multilingual annotation system applied to over 30 trillion tokens (educational value, fields, domains, safety)[30]
Curriculum / staging
three-stage pre-training: S1 general stage >30T tokens at sequence length 4,096; S2 reasoning stage ~5T higher-quality tokens at 4,096 with accelerated learning-rate decay; S3 long-context stage hundreds of billions of tokens at 32,768 (75% 16,384–32,768-token texts, 25% 4,096–16,384)[26]
Optimizer
not disclosed
LR schedule
learning-rate decay accelerated during S2; optimal learning rate per model predicted via scaling laws (values not printed)[26]
Batch schedule
batch size strategy predicted via scaling laws for each dense/MoE model (values not printed)[26]
long-CoT cold start (responses generated by QwQ-32B, two-phase query/response filtering); Reasoning RL on 3,995 query-verifier pairs (math/code/STEM); thinking budget mechanism halting thinking at a user-defined threshold; /think and /no_think mode switching with <think>/</think> tokens[33]
Distillation
Qwen3-30B-A3B: strong-to-weak distillation (off-policy response distillation from teacher outputs, then on-policy distillation aligning student logits with teacher Qwen3-32B or Qwen3-235B-A22B via KL divergence; ~1/10 the GPU hours of four-stage RL). Flagship Qwen3-235B-A22B is not distilled — it runs the full four-stage post-training pipeline.[34]
stated · §2 (MoE models 'share the same fundamental architecture as the Qwen3 dense models', which use Grouped Query Attention; Table 2 heads Q/KV 64/4 and 32/4)
stated · §2
stated · §2 ('128 total experts'); Table 2 (# Experts Total / Activated: 128 / 8)
stated · §2 ('8 activated experts per token'); Table 2 (# Experts Total / Activated: 128 / 8)
stated · §2 ('We follow Qwen2.5-MoE and implement fine-grained expert segmentation (Dai et al., 2024)')
stated · §3.2, §A.1.1
stated · §A.1.1 ('YARN with a scaling factor=4'); §3.2 ('a four-fold increase in sequence length capacity during inference')
stated · §2 (Rotary Positional Embeddings)
stated · §3.2 ('we increase the base frequency of RoPE from 10,000 to 1,000,000 using the ABF technique')
stated · §2 (RMSNorm with pre-normalization)
stated · §2 ('RMSNorm with pre-normalization')
stated · §2 (QK-Norm (Dehghani et al., 2023) introduced to the attention mechanism; 'The Qwen3 MoE models share the same fundamental architecture as the Qwen3 dense models')
stated · §3.2 stage 3 ('All models are pre-trained on hundreds of billions of tokens with a sequence length of 32,768 tokens')