GLM-4.5 is Zhipu AI's first MoE model: a 92-layer stack (89 MoE + 3 dense layers) with 355B total and 32B activated parameters, deliberately made deeper and narrower than DeepSeek-V3 or Kimi K2 (5120 hidden dim, 160 routed experts with 8 active plus 1 shared expert, per-expert intermediate dim 1536 against 12288 in the dense layers) because the team found depth helped reasoning. Attention is GQA with an unusual 96 query heads and only 8 KV heads (head dim 128), with QK-Norm on the flagship to stabilize attention logits and partial RoPE whose base is raised from 10,000 to 1,000,000 at the 32K context stage; context grows 4K → 32K → 128K across pre- and mid-training. Routing uses sigmoid gates with DeepSeek-V3-style loss-free bias balancing (bias update rate 0.001 for the first 15T tokens, then 0) plus a small sequence-level balance loss; the MTP layer is itself an MoE layer and doubles as the speculative-decoding draft. Training runs 23T tokens with the Muon optimizer (N=5 Newton-Schulz steps, weight decay 0.1, no dropout) under cosine decay, batch-size warmup 16M→64M tokens, and BF16 with FP8 rollout inference. Post-training is a two-stage expert iteration: three domain experts (Reasoning, Agent, General chat) are trained with SFT + RL (GRPO-family reasoning RL at a single 64K output length, agentic RL with iterative self-distillation, holistic/instruction/function-calling/pathology general RL), then distilled via SFT into one unified model that can answer in thinking or non-thinking mode — the 'hybrid reasoning' that defines the ARC series.
What the report claims is novel
An open-source MoE foundation model unifying agentic, reasoning, and coding (ARC) capabilities, with a hybrid reasoning method supporting both thinking and direct-response modes; ranked 3rd overall and 2nd on agentic benchmarks among all evaluated models [Abstract, §1]
Depth-over-width MoE design: width (hidden dim and routed-expert count) reduced and height (layer count) increased relative to DeepSeek-V3 and Kimi K2, because deeper models exhibited better reasoning capacity [§2.1]
2.5× more attention heads (96 heads for a 5120 hidden dimension) plus QK-Norm: the higher head count does not improve training loss but consistently improves reasoning benchmarks such as MMLU and BBH [§2.1]
Loss-free balance routing with sigmoid gates for MoE layers, with an auxiliary sequence-level balance loss to prevent extreme intra-sequence imbalance [§2.1, §2.4]
An MoE layer as the MTP (Multi-Token Prediction) layer for both models, supporting speculative decoding during inference [§2.1]
Expert model iteration post-training: three domain experts (Reasoning, Agent, General chat) consolidated via self-distillation into a unified hybrid-reasoning generalist [§3]
Novel function-call template encapsulating call keys and values in XML-like special token tags, substantially reducing character escaping for code segments without hurting function-call execution [§3.1]
Reasoning-RL recipes validated on a smaller model: two-stage difficulty-based curriculum, single-stage RL directly at 64K output length (multi-stage progressive-length RL found inferior), dynamic sampling temperature, and token-weighted mean loss for code RL [§3.2]
Slime, an open-source RL infrastructure supporting colocated synchronous and disaggregated asynchronous modes, BF16 training with FP8 rollout inference, and decoupled agentic rollout engines (Docker-based, unified HTTP endpoint, centralized data pool) [§3.5]
Open-sourced evaluation toolkit (glm-simple-evals) for reproducibility of benchmark results [§1]
Atlas assessment: Architecturally a conservative DeepSeek-V3-style MoE (sigmoid gating, loss-free bias-based balancing, one shared expert, MTP layer) with three notable deltas: GQA with 8 KV heads plus QK-Norm instead of DeepSeek-V3's MLA (the only atlas MoE besides K3 with QK-normed global attention), a depth-over-width geometry (5120 hidden, 92 layers, 160 experts vs DeepSeek-V3's 7168/61/256), and the Muon optimizer. Nothing in the block is genuinely new relative to atlas peers — kimi-k3's KDA/AttnRes axes are absent here; the report's novelty is concentrated in post-training (domain-expert iteration with self-distillation into a hybrid thinking/non-thinking generalist) and in the unusually aggressive head-count/QK-norm choice. The 3rd-overall/2nd-agentic ranking claims are internal to the report. The activation function, tokenizer, vocab size, and training compute are all undisclosed, which limits comparability with peers.
Relation to the atlas
Lineage in
Sigmoid gating for MoE layers from DeepSeek-V3
Loss-free (auxiliary-loss-free) balance routing with expert biases from DeepSeek (aux-loss-free balancing line)
Shared + routed expert MoE organization and MTP-layer parameter counting convention (Table 1 columns mirror DeepSeek-V3's) from DeepSeek-V3
QK-Norm for attention-logit stabilization from Query-Key Normalization for Transformers (Henry et al. 2020)
Multi-Token Prediction (MTP) layer from Multi-token prediction (Gloeckle et al. 2024)
GRPO (group relative policy optimization) as the RL backbone, KL term excluded from DeepSeekMath (DeepSeek)
Muon optimizer for hidden-layer parameters from Muon (Jordan et al. 2024 / Liu et al. 2025)
Quality-bucketed web corpus construction with up-sampling of high-quality buckets from Nemotron-CC / SemDedup / Fineweb-2
Influence out
zai-glm-4-5-air: Shared architecture and post-training pipeline with the flagship GLM-4.5 (same series, smaller scale, no QK-Norm)
Notable omissions
FFN activation function never named anywhere in the report
Tokenizer and vocabulary size not disclosed
Input/output embedding tying not disclosed
Normalization placement (pre/post) not described
Training hardware and total compute (FLOPs/GPU-hours) not disclosed
Parallelism strategy (TP/PP/EP/CP) not described — only 'Megatron' is named in the infra diagram (Fig. 10)
No license named for the weights release
GLM-4.5-Air's own pre-training token budget not disclosed (23T is stated for GLM-4.5)
SFT dataset size given only as 'millions of samples'; most RL data sizes/rollout counts undisclosed
No long-context benchmark results (e.g. RULER-style) despite the 128K window claim
Expert granularity terminology (fine-grained vs grouped) absent; per-expert width must be inferred from Table 1
GLM-4.5-Air receives no human-evaluation or translation coverage (those sections cover GLM-4.5 only)
loss-free balance routing: expert bias update rate 0.001 for the first 15T tokens, then 0.0; auxiliary sequence-level balance loss with weight 0.0001 to avoid extreme imbalance within a single sequence[18]
Expert granularity
compact per-expert FFN (MoE intermediate dim 1536, 0.3× hidden) vs dense-layer intermediate 12288; report uses no fine-grained/grouped terminology
corpus of webpages, social media, books, papers, and code repositories: web documents bucketed by quality scores (Nemotron-CC-inspired; top bucket contributes over 3.2 epochs, lowest bucket discarded; MinHash + SemDedup deduplication); multilingual documents from crawled web and Fineweb-2 with educational-utility classifier up-sampling; code from GitHub and code hosting platforms with tiered quality models (high/medium/low) and Fill-In-the-Middle objective on all source code; math & science documents scored by an LLM and up-sampled above a threshold; two-stage pre-training: stage 1 general web documents, stage 2 up-sampled code, math, and science[30]
Curriculum / staging
two-stage pre-training (general documents, then up-sampled code/math/science); mid-training stages: repo-level code training (4K→32K, concatenated same-repo files, filtered issues/PRs/commits in diff format), synthetic reasoning data training, long-context & agent training (32K→128K, up-sampled long documents, synthetic agent trajectories); best-fit packing used only in mid-training (random truncation kept in pre-training)[31]
Optimizer
Muon for all parameters except word embeddings, biases, and RMSNorm weights (Newton-Schulz iterations N=5, momentum 0.95, scaled Muon update RMS 0.2); weight decay 0.1; no dropout[32]
LR schedule
cosine decay (chosen over warmup-stable-decay after experiments); warmup 0 → 2.5e-4, decaying to 2.5e-5 until the end of mid-training[32]
Batch schedule
batch size warmup: gradually increased from 16M to 64M tokens over the first 500B tokens, then constant[32]
Precision
BF16 training with FP8 inference for RL rollouts (online block-wise FP8 quantization of parameters before rollout dispatch)[33]
Parallelism
Megatron-based training (per Slime infra description); detailed TP/PP/EP/CP strategy not described[34]
hybrid reasoning modes (thinking mode for complex reasoning/agentic tasks, non-thinking mode for instant responses); reasoning RL: two-stage difficulty-based curriculum (stage 2 problems verified pass@8==0, pass@512>>0), single-stage RL directly at 64K max output length (multi-stage progressive-length RL found inferior), dynamic sampling temperature (raised on convergence, capped by ≤1% validation drop), token-weighted mean loss for code RL, expert-verified MCQ-only data for science RL; agentic RL on web-search and SWE tasks with process format penalty and iterative self-distillation; test-time compute scaled via interaction turns; general RL: holistic RL (~5,000 prompts, 7/33/139 category taxonomy), instruction-following RL (7 major/151 minor constraint taxonomy, rule+RM+critique feedback), function-calling RL (step-wise rule-based + end-to-end multi-turn), pathology RL[37]
Distillation
self-distillation (no external teacher): Stage 1 trains three domain experts (Reasoning, Agent, General chat); Stage 2 overall SFT distills expert outputs into one unified hybrid-reasoning generalist; agentic RL additionally uses iterative self-distillation (cold-start SFT data replaced by RL-trained responses before further RL)[38]
MTP (Multi-Token Prediction) MoE layer supporting speculative decoding during inference; MTP loss weight λ = 0.3 for the first 15T tokens, 0.1 after[18]
stated · Title; §5 ('the GLM-4.5 model series, including GLM-4.5 and GLM-4.5-Air')
stated · Title page byline
stated · pipeline
stated · Abstract ('We release both GLM-4.5 (355B parameters) and a compact version, GLM-4.5-Air (106B parameters)'); §1 (huggingface.co/zai-org/GLM-4.5); §5 ('We release the model weights of GLM-4.5 and GLM-4.5-Air')
stated · Abstract; Table 1
stated · Table 1 (Hidden Dim)
stated · Table 1 (MoE Intermediate Dim)
stated · Table 1 (# Attention Heads); §2.1 (96 heads for a 5120 hidden dimension)
stated · Table 1 (# Key-Value Heads) — GQA
stated · Table 1 (Attention Head Dim)
stated · §2.1 ('we adopt the MoE architecture'); Table 1
stated · §2.1 (GQA in the self-attention component; no per-layer hybrid described)
stated · Table 1 (# Experts (total) 160)
stated · Table 1 (# Experts Active Per Token)
stated · Table 1 (# Shared Experts)
stated · §2.1 ('We employ loss-free balance routing [40] and sigmoid gates for MoE layers [23]')
stated · §2.1, §2.4
stated · §2.3, §2.4
n/a · §2.4 (base-frequency change, not an interpolation/rescaling factor)
stated · §2.4 (max sequence length extended to 32,768 and 131,072 during mid-training)
stated · §2.1 (partial RoPE); §2.4 (RoPE base frequency adjustment)
stated · §2.4 ('we also adjusted RoPE's base frequency from 10,000 to 1,000,000' when extending to 32K)
stated · §2.1 (Grouped-Query Attention with partial RoPE)
stated · §2.4 ('weights for RMSNorm' excluded from the Muon optimizer)
stated · §2.1 ('We also incorporate QK-Norm [15] to stabilize the range of attention logits'); Table 1 (QK-Norm: Yes)
stated · §2.4 (max sequence length 4,096 during pre-training, extended to 32,768 and 131,072 in mid-training); §3.1 (overall SFT at max 128K)
stated · §4.2.3 (evaluation 'prevent exceeding the 128K context limit')
stated · Abstract ('multi-stage training on 23T tokens')
stated · §2.2
stated · §2.2, §2.3
stated · §2.4
stated · §3.5
stated · §3.5 (Fig. 10: 'Training (Megatron) – handles the main training process')
stated · §3.1 (cold-start SFT with extended CoT responses; overall SFT on millions of samples covering reasoning, general chat, agentic, and long-context tasks at 128K max context; function-call template with XML-like special token tags; rejection sampling; prompt selection with response-level scaling)
stated · §3.2 ('Our overall RL algorithm builds upon the GRPO [31] framework, excluding the KL loss term'); §3.4 ('crucial during GRPO training')