GLM-4.5-Air is the compact sibling in Zhipu AI's GLM-4.5 series: a 46-layer sparse MoE (45 MoE + 1 dense layer) with 106B total and 12B activated parameters, hidden dim 4096, 128 routed experts (8 active) plus one shared expert, and per-expert intermediate dim 1408 against 10944 in its single dense layer. It shares the flagship's block design — GQA with 96 query heads over 8 KV heads (head dim 128), partial RoPE with the base raised from 10,000 to 1,000,000 at the 32K context stage, sigmoid gating with loss-free bias-based load balancing plus a small sequence-level balance loss, RMSNorm, and a dedicated MoE MTP layer used for speculative decoding — with one notable exception: Table 1 marks QK-Norm as 'No' for Air, the only stated architectural difference from GLM-4.5. Training follows the same series recipe (two-stage pre-training, repo-level code / synthetic reasoning / long-context-agent mid-training, 4K→128K context growth, Muon optimizer, cosine decay, BF16 with FP8 rollouts), though the report gives no token budget specific to Air. Post-training is the shared expert-model-iteration pipeline: domain experts for Reasoning, Agent, and General chat are trained and then distilled via SFT into one hybrid-reasoning model with thinking and non-thinking modes. The report positions Air as a 100B-scale parameter-efficiency play, matching or exceeding Qwen3-235B-A22B and MiniMax-M1 on ARC benchmarks while activating only 12B parameters.
What the report claims is novel
GLM-4.5-Air: a compact 106B-parameter MoE representing a significant leap among 100B-scale models, matching or exceeding Qwen3-235B-A22B and MiniMax-M1, and ranked 6th overall on the ARC benchmark set [§1, Fig. 1]
Both models lie on the Pareto frontier of SWE-bench Verified vs model parameters among open-source models [§1, Fig. 2]
Series-level: depth-over-width MoE design, 2.5× attention heads with QK-Norm (flagship only), loss-free balance routing with sigmoid gates, MoE MTP layer for speculative decoding, expert-model-iteration post-training with hybrid reasoning modes, and the Slime RL infrastructure [§2, §3, §3.5]
Atlas assessment: GLM-4.5-Air is the same block as GLM-4.5 at smaller scale (4096 hidden, 46 layers, 128 experts) with exactly one stated architectural difference: QK-Norm is absent (Table 1). The record exists separately from zai-glm-4-5 because its config differs, not because its block type differs — the task premise that Air is 'dense' is contradicted by the report, which explicitly calls it an MoE ('GLM-4.5-Air is a smaller MoE model with 106B parameters', §1; 45/46 MoE layers, Table 1). Relative to the 100B-scale atlas class (qwen-qwen3-moe 235B-A22B), Air is narrower and deeper with 96/8 GQA heads and per-expert width 1408; its 12B active params are well below Qwen3's 22B, which the report leverages as a parameter-efficiency claim. The QK-Norm ablation between flagship and Air is disclosed but never analyzed, which weakens the causal story for Air's lower scores.
Relation to the atlas
Lineage in
Sigmoid gating and loss-free (auxiliary-loss-free) balance routing for MoE layers from DeepSeek-V3
Shared + routed expert MoE organization and MTP layer (Table 1 columns mirror DeepSeek-V3's) from DeepSeek-V3
Multi-Token Prediction (MTP) layer from Multi-token prediction (Gloeckle et al. 2024)
GRPO as the RL backbone, KL term excluded from DeepSeekMath (DeepSeek)
Muon optimizer for hidden-layer parameters from Muon (Jordan et al. 2024 / Liu et al. 2025)
Shared architecture and post-training pipeline with the flagship GLM-4.5 (same series, smaller scale, no QK-Norm) from GLM-4.5 (same report)
Influence out
prime-intellect-intellect-3: Base model and full architecture: SFT and RL run on top of the GLM-4.5-Air base (all block-level design inherited)
Notable omissions
Air's own pre-training token budget not disclosed (23T stated only for GLM-4.5)
The QK-Norm difference between flagship and Air (Table 1) is never analyzed or ablated in the text
FFN activation function never named
Tokenizer and vocabulary size not disclosed
Input/output embedding tying not disclosed
Normalization placement (pre/post) not described
Training hardware and total compute (FLOPs/GPU-hours) not disclosed
Parallelism strategy not described (only 'Megatron' named in Fig. 10)
No license named for the weights release
No human-evaluation, CC-Bench, or translation results for Air (those sections cover GLM-4.5 only)
No long-context benchmark results despite the 128K window claim
loss-free balance routing: expert bias update rate 0.001 for the first 15T tokens, then 0.0; auxiliary sequence-level balance loss with weight 0.0001[18]
Expert granularity
compact per-expert FFN (MoE intermediate dim 1408, 0.34× hidden) vs dense-layer intermediate 10944; report uses no fine-grained/grouped terminology
trained at extended lengths: 4K → 32K → 128K across pre-/mid-training, with RoPE base frequency raised from 10,000 to 1,000,000 at the 32K stage[19]
Tokenizer
Algorithm
not disclosed
Notes
not disclosed
Training
Training tokens
not disclosed
Data composition
series-level corpus (report describes the GLM-4.5 series jointly): webpages, social media, books, papers, and code repositories; web documents bucketed by quality (top bucket >3.2 epochs, lowest discarded, MinHash + SemDedup); multilingual from crawled web and Fineweb-2 with quality-classifier up-sampling; code from GitHub with tiered quality models and Fill-In-the-Middle; math & science up-sampled above an LLM-scored threshold; two-stage pre-training (general documents, then up-sampled code/math/science)[29]
Curriculum / staging
two-stage pre-training; mid-training stages: repo-level code (4K→32K), synthetic reasoning data, long-context & agent training (32K→128K); best-fit packing in mid-training only (series-level recipe)[30]
Optimizer
Muon for all parameters except word embeddings, biases, and RMSNorm weights (Newton-Schulz iterations N=5, momentum 0.95, scaled update RMS 0.2); weight decay 0.1; no dropout (series-level recipe)[31]
LR schedule
cosine decay (chosen over warmup-stable-decay); warmup 0 → 2.5e-4, decaying to 2.5e-5 until the end of mid-training (series-level recipe)[31]
Batch schedule
batch size warmup: 16M → 64M tokens over the first 500B tokens, then constant (series-level recipe)[31]
Precision
BF16 training with FP8 inference for RL rollouts (online block-wise FP8 quantization before rollout dispatch)[32]
Parallelism
Megatron-based training (per Slime infra description); detailed TP/PP/EP/CP strategy not described[33]
hybrid reasoning modes (thinking + non-thinking); reasoning RL with two-stage difficulty-based curriculum, single-stage RL at 64K max output length, dynamic sampling temperature, token-weighted mean loss for code RL; agentic RL (web-search + SWE) with process format penalty and iterative self-distillation; general RL (holistic, instruction-following, function-calling, pathology); test-time compute via interaction turns (series-level pipeline, §3.2–3.4)[36]
Distillation
self-distillation (no external teacher): Stage 1 domain experts (Reasoning, Agent, General chat) consolidated into a unified hybrid-reasoning model in Stage 2 via SFT distillation; iterative self-distillation in agentic RL (series-level pipeline)[37]
MTP (Multi-Token Prediction) MoE layer supporting speculative decoding during inference; MTP loss weight λ = 0.3 for the first 15T tokens, 0.1 after (series-level recipe)[39]
stated · Title; §1 ('This paper introduces two new models: GLM-4.5 and GLM-4.5-Air')
stated · Title page byline
stated · pipeline
stated · Abstract ('We release both GLM-4.5 (355B parameters) and a compact version, GLM-4.5-Air (106B parameters)'); §1 (huggingface.co/zai-org/GLM-4.5); §5 ('We release the model weights of GLM-4.5 and GLM-4.5-Air')
stated · Abstract; Table 1
stated · Table 1 (Hidden Dim)
stated · Table 1 (MoE Intermediate Dim)
stated · Table 1 (# Attention Heads)
stated · Table 1 (# Key-Value Heads) — GQA
stated · Table 1 (Attention Head Dim)
stated · §1 ('GLM-4.5-Air is a smaller MoE model with 106B parameters'); §5 ('Both models adopt the MoE architecture'); Table 1
stated · §2.1 (GQA in the self-attention component; no per-layer hybrid described)
stated · Table 1 (# Experts (total) 128)
stated · Table 1 (# Experts Active Per Token)
stated · Table 1 (# Shared Experts)
stated · §2.1 ('In the GLM-4.5 series, we adopt the MoE architecture... We employ loss-free balance routing [40] and sigmoid gates for MoE layers [23]')
stated · §2.1, §2.4 (series-level recipe)
stated · §2.3, §2.4
n/a · §2.4 (base-frequency change, not an interpolation/rescaling factor)
stated · §2.4 (max sequence length extended to 32,768 and 131,072 during mid-training)
stated · §2.1 (partial RoPE); §2.4 (RoPE base frequency adjustment)
stated · §2.4 ('we also adjusted RoPE's base frequency from 10,000 to 1,000,000' when extending to 32K)
stated · §2.1 (Grouped-Query Attention with partial RoPE)
stated · §2.4 ('weights for RMSNorm' excluded from the Muon optimizer; series-level recipe)
stated · Table 1 (QK-Norm: No)
stated · §2.4 (series-level recipe: max sequence length 4,096 during pre-training, extended to 32,768 and 131,072 in mid-training); §3.1 (overall SFT at max 128K)
stated · §4.2.3 (evaluation 'prevent exceeding the 128K context limit')
stated · §2.2
stated · §2.2, §2.3
stated · §2.4
stated · §3.5
stated · §3.5 (Fig. 10)
stated · §3.1 (cold-start SFT + overall SFT at 128K max context; series-level pipeline applying to both models)
stated · §3.2 ('Our overall RL algorithm builds upon the GRPO [31] framework, excluding the KL loss term'); §3.4