Qwen3 dense is the standard-bearer of the Qwen2.5 architectural line: each layer is a pre-RMSNorm transformer block with GQA attention, SwiGLU feed-forward, and RoPE, stacked 64 deep in the reference Qwen3-32B (36 layers in 4B/8B, 40 in 14B, 28 in 0.6B/1.7B). The two structural changes over Qwen2.5 are the removal of QKV-bias and the addition of QK-Norm to the attention mechanism, both aimed at training stability. Context is grown in three pre-training stages (4,096 → 4,096 → 32,768 tokens, 36T tokens total across 119 languages), then extended four-fold at inference via YaRN with scaling factor 4 plus Dual Chunk Attention, on top of an ABF base-frequency increase from 10,000 to 1,000,000; the 32B deploys at 128K. Post-training is where Qwen3 makes its mark: a long-CoT cold start with QwQ-32B-generated reasoning traces, GRPO-based reasoning RL on 3,995 verifiable query-verifier pairs, a thinking-mode fusion stage that fuses /think and /no_think behaviors into one model with a budget-controllable thinking process, and a general RL stage with rule-based and model-based rewards. The five smaller dense siblings (0.6B–14B) skip most of this: they are trained via strong-to-weak distillation, first off-policy on teacher outputs and then on-policy against teacher logits, at roughly a tenth of the RL compute.
What the report claims is novel
Integration of thinking mode and non-thinking mode into a single model with dynamic mode switching via chat templates, eliminating the need to alternate between chat and reasoning models [Abstract, §1, §4.3]
Thinking budget mechanism giving users fine-grained control over reasoning effort (token budget) at inference, with performance scaling smoothly with the budget [Abstract, §1, §4.7]
Strong-to-Weak Distillation (off-policy plus on-policy logit distillation) for lightweight models, achieving better performance than RL at about 1/10 of the GPU hours [§4.5, §4.7]
Multilingual expansion from 29 to 119 languages and dialects, with instance-level data-mixture optimization via a large-scale annotation system [Abstract, §3.1]
Architecture improvements over Qwen2.5: QK-Norm added to the attention mechanism and QKV-bias removed for stable training [§2]
Scaling laws developed for optimal hyperparameters (learning-rate scheduler, batch size) across the three pre-training stages [§3.2]
Atlas assessment: The dense block is a conservative Qwen2.5-lineage stack (GQA + SwiGLU + RoPE + RMSNorm pre-norm) with one structural addition, QK-Norm, and one removal, QKV-bias; closest atlas relative is meta-llama-3.1 (same four-mechanism recipe, also RoPE + pre-RMSNorm), from which Qwen3 dense differs mainly by QK-Norm, the 1,000,000 RoPE base with YaRN+DCA 4× extension instead of direct 128K training, and a tie-embedding policy split by size (small models tie, 8B+ do not). The genuinely novel contribution is the post-training pipeline (long-CoT cold start → reasoning RL with GRPO → thinking-mode fusion → general RL) and strong-to-weak distillation for small models; these are training recipes, not new mixing mechanisms. The report discloses no hidden dims or FFN dims (unlike Llama 3.1 / Phi-3 / DeepSeek-V3 reports), so per-variant width comparisons in the atlas are impossible from this source.
Relation to the atlas
Lineage in
Grouped Query Attention (GQA) — shared KV heads across query groups from GQA (Ainslie et al., 2023)
SwiGLU gated activation from Dauphin et al., 2017
Rotary Positional Embeddings (RoPE) from Su et al., 2024
RMSNorm with pre-normalization from Jiang et al., 2023
QK-Norm for attention-logit normalization from Dehghani et al., 2023
Adjusted Base Frequency (ABF) RoPE base increase to 1,000,000 from Xiong et al., 2023
YaRN length extrapolation and Dual Chunk Attention (DCA) from Peng et al., 2023 (YaRN); An et al., 2024 (DCA)
GRPO for reasoning RL from DeepSeekMath (Shao et al., 2024)
Overall dense block design 'similar to Qwen2.5' from Qwen2.5 (not in atlas)
Influence out
jetbrains-mellum2-thinking-12b-a2-5b: Qwen3-MoE recipe — 'closely follows the Qwen3-MoE recipe' (MoE-in-every-layer, GQA, QK-Norm, RMSNorm, SiLU, RoPE); scaled down from Qwen3-30B-A3B
prime-intellect-intellect-3: Chat template with <|system|>/<|user|>/<|assistant|> and <|im_start|>/<|im_end|> control tokens, XML-style tool calls, <|think|> reasoning token
Notable omissions
Hidden/embedding dimensions and FFN intermediate dimensions are never disclosed for any variant (Table 1 lists only layers, heads, tie-embedding, context length) — per-variant width comparisons are impossible from this report
Training optimizer, precision, parallelism strategy, hardware, and total compute (FLOPs/GPU-hours) are not disclosed
Learning rate and batch size values are not printed (only that they were scaling-law-predicted)
No head dimension disclosed
MoE record fields dependent on per-expert dims are unknown (this report's MoE section does not give them either)
Source extraction: arxiv.org/html/2505.09388 is a truncated conversion ending at §4.3 (Thinking Mode Fusion) with Tables 10–37 missing; the stored source was re-extracted from the full 35-page PDF (116K chars), and PDF table columns interleave row-wise in the plain text (verified against context before use)
119 languages and dialects; domains include coding, STEM, reasoning tasks, books, multilingual texts, and synthetic data; PDF-like documents transcribed by Qwen2.5-VL with Qwen2.5 refinement; trillions of synthetic tokens generated by Qwen2.5, Qwen2.5-Math, Qwen2.5-Coder (textbooks, QA, instructions, code); instance-level data-mixture optimization via a multilingual annotation system applied to over 30 trillion tokens (educational value, fields, domains, safety)[27]
Curriculum / staging
three-stage pre-training: S1 general stage >30T tokens at sequence length 4,096; S2 reasoning stage ~5T higher-quality tokens at 4,096 with accelerated learning-rate decay; S3 long-context stage hundreds of billions of tokens at 32,768 (75% 16,384–32,768-token texts, 25% 4,096–16,384)[23]
Optimizer
not disclosed
LR schedule
learning-rate decay accelerated during S2; optimal learning rate per model predicted via scaling laws (values not printed)[23]
Batch schedule
batch size strategy predicted via scaling laws for each dense/MoE model (values not printed)[23]
long-CoT cold start (responses generated by QwQ-32B, two-phase query/response filtering); Reasoning RL on 3,995 query-verifier pairs (math/code/STEM); thinking budget mechanism halting thinking at a user-defined threshold; /think and /no_think mode switching with <think>/</think> tokens[30]
Distillation
Strong-to-Weak Distillation for smaller models (0.6B/1.7B/4B/8B/14B dense and 30B-A3B MoE): off-policy response distillation from teacher outputs, then on-policy distillation aligning student logits with teacher (Qwen3-32B or Qwen3-235B-A22B) via KL divergence; ~1/10 the GPU hours of four-stage RL[31]