GLM-5 is a 78-layer (3 dense + 75 MoE), 744B-parameter sparse MoE with 40B active parameters and 8 of 256 experts (1 shared) active per token. Its defining move is replacing the dense attention of GLM-4.5 with DeepSeek Sparse Attention on every layer: a lightning indexer (32 heads, head dim 128) scores all cached KV entries per query and attention is computed only over the top-k = 2048 retrieved tokens, cutting long-sequence attention cost ~1.5-2×. The attention itself remains DeepSeek-style MLA, re-tuned as MLA-256 (64 heads, QK head dim 192, V head dim 256, 576-dim latent KV) with an added Q LoRA, and with Muon Split — per-head orthogonalization of the attention up-projections — stabilizing logit scale without clipping. Decoding is sped by sharing parameters across 3 MTP layers, which also serve as the speculative draft. Context grows 4K → 200K through a staged mid-training curriculum (32K/128K/200K) with long-context agentic data, plus a DSA warm-up and 20B-token sparse-adaptation stage; SFT then runs at 202,752-token windows. Post-training is a sequential Reasoning → Agentic → General RL pipeline (GRPO-based with IcePop-style mismatch control; asynchronous, decoupled rollout engines with a TITO gateway and direct double-sided importance sampling for agentic work) closed by on-policy cross-stage distillation. The report's framing is 'vibe coding to agentic engineering': long-horizon SWE, terminal, search and slide-generation environments, CC-Bench-V2 evaluation, and a full-stack adaptation to seven Chinese GPU platforms with W4A8 quantization. Note: the gallery's GLM-5.1 and GLM-5.2 cards cite this report, but the report itself discloses only GLM-5.
What the report claims is novel
Adoption of DeepSeek Sparse Attention (DSA): content-based top-k sparse attention (lightning indexer, k=2048) applied to all layers, cutting attention compute ~1.5-2× at long sequences with no quality degradation, enabling scaling to 744B parameters (40B active) and a 28.5T-token budget [§1, §2.1.1]
Muon Split: per-head matrix orthogonalization for the Muon optimizer that makes MLA match GQA-8 performance and keeps attention-logit scale stable without clipping; plus MLA-256 (QK head dim 192 → V head dim 256, heads reduced by 1/3) to cut decoding cost [§2.1]
Multi-Token Prediction with parameter sharing: 3 MTP layers share parameters during training, keeping draft-model memory at DeepSeek-V3 level while raising acceptance length to 2.76 (vs DeepSeek-V3.2's 2.55) [§2.1]
Fully asynchronous, decoupled RL infrastructure (slime-based): Multi-Task Rollout Orchestrator with server-based rollouts, TITO (Token-in-Token-out) gateway, direct double-sided importance sampling with token-level clipping, DP-aware routing for KV-cache locality, heartbeat-driven fault tolerance and PD disaggregation [§3.3, §3.6, §4.1]
Novel asynchronous agent RL algorithms that learn from diverse long-horizon interactions, with deterministic indexer top-k (torch.topk) and frozen indexer during RL to resolve DSA training-inference mismatch [§1, §3.2, §4.1]
On-policy cross-stage distillation to prevent catastrophic forgetting across the sequential Reasoning → Agentic → General RL pipeline [§3.5]
CC-Bench-V2: an automated internal evaluation suite (frontend/backend/long-horizon) with Agent-as-a-Judge, removing human labeling entirely [§6.2]
Full-stack adaptation to seven Chinese GPU ecosystems (Huawei Ascend, Moore Threads, Hygon, Cambricon, Kunlunxin, MetaX, Enflame) from kernels to inference frameworks, with W4A8 quantization fitting the model on a single Atlas 800T A3 machine [§1, §5]
First open-weights model to score 50 on the Artificial Analysis Intelligence Index v4.0, and #1 open model on LMArena Text and Code arenas; ~20% average improvement over GLM-4.7 on the eight ARC benchmarks [§1]
Atlas assessment: The block is a conservative evolution of the GLM-4.5/DeepSeek-V3 MLA-MoE lineage: the same 256-total/8-active/1-shared expert organization as DeepSeek-V3, MLA retained but re-tuned (MLA-256 with a Q LoRA added, Muon Split for head-wise optimizer scaling), and the dense-attention backbone replaced by DeepSeek-V3.2's DSA sparse attention on every layer. Against atlas peers: vs deepseek-v3 the differences are DSA, Muon (not AdamW), MTP parameter sharing, and the 3-dense-layer structure; vs zai-glm-4-5 it roughly doubles parameters (355B→744B) while shrinking depth (92→78 layers) and swapping in DSA. None of the attention-side ingredients is original to this report (MLA and DSA are both credited to DeepSeek), and the sparse-attention RL-stability fix (deterministic top-k, frozen indexer) is a pragmatic engineering contribution rather than a new mechanism. The genuinely distinctive contributions are in the post-training and serving systems: fully asynchronous agentic RL with TITO gateways and direct double-sided importance sampling, on-policy cross-stage distillation, and the multi-vendor Chinese-GPU inference stack. Reported gains ('~20% over GLM-4.7', '50 on AA Index v4.0') are internal or third-party leaderboard claims and not independently verifiable from the report.
Relation to the atlas
Lineage in
Multi-head Latent Attention (MLA) — latent KV compression retained in the MLA-256 variant from DeepSeek-V2
DeepSeek Sparse Attention (DSA) — content-based top-k sparse attention via a lightning indexer from DeepSeek-V3.2
Multi-Token Prediction (MTP) with a single (here parameter-shared) MTP layer as speculative draft from DeepSeek-V3
Muon optimizer recipe (per-head 'Muon Split' orthogonalization is the GLM-5 adaptation of the GLM-4.5 recipe) from GLM-4.5
slime framework (decoupled rollout engines initialized in GLM-4.5) reused as the unified RL infrastructure from GLM-4.5
IcePop technique for mitigating training-inference mismatch in RL (removing its KL term) from IcePop (not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
GLM-5.1 and GLM-5.2 are never mentioned in this report (the gallery lists cards for them citing this same report)
Positional encoding never named (no RoPE/NoPE statement; the 576-dim MLA latent hints at a 64-dim RoPE component but it is unstated)
Tokenizer algorithm not described (only the 154,880 vocab size in Table 10)
License for the weight release not named
Router function (gating) and MoE load balancing not described
Normalization type/placement and activation function not stated
Master training precision not stated (only INT4 QAT at SFT and FP8 rollouts)
Batch sizes and peak pre-training learning-rate values not printed (deferred to GLM-4.5 settings)
Training hardware and total compute (FLOPs/GPU-hours) not disclosed
No pretraining data-mixture percentages disclosed
Input/output embedding tying not disclosed
The prose claim of 'layer count 80' (§2.1) is inconsistent with Table 10's 3 dense + 75 MoE = 78 backbone layers and is never reconciled
uniform — DeepSeek Sparse Attention (DSA) applied to all layers: a lightning indexer (32 heads, head dim 128, top-k = 2048 KV entries per query) selects relevant keys/values and attention is computed sparsely over the retrieved subset; no dense-attention or sliding-window alternation[14]
Web: refined GLM-4.5 pipeline with an additional DCLM sentence-embedding classifier and a World Knowledge classifier (Wikipedia/LLM-labeled) for long-tail knowledge. Code: refreshed snapshots from code hosts plus code-containing web pages (+28% fuzzily deduplicated unique tokens), Software Heritage metadata fixes, and dedicated classifiers for low-resource programming languages (Scala, Swift, Lua, etc.). Math & science: LLM-scored documents from webpages/books/papers with chunk-and-aggregate scoring for long documents; synthetic/AI-generated/template data strictly filtered out[22]
Curriculum / staging
two-stage base training (pre-training prioritizing code and reasoning early on, then mid-training for agentic and long-context capacity) with progressive context extension 32K (1T tokens) → 128K (500B) → 200K (50B); long-context data (natural + synthetic interleaved packing inspired by NextLong/EntropyLong, MRCR-like data at 200K) upsampled at later stages; DSA warm-up (1,000 steps, 14 sequences of 202,752 tokens, lr 5e-3 → 2e-4) then sparse adaptation on 20B tokens (constant lr 1e-5)[23]
Optimizer
Muon with Muon Split (per-head Newton-Schulz-style matrix orthogonalization applied to independent per-head up-projection matrices); zero-redundant distributed communication variant[24]
LR schedule
pre-training: warmup from 0 to 2e-4 then decay to 4e-5 (cosine per GLM-4.5 setting); mid-training: linear decrease 4e-5 → 1e-5; DSA warm-up: 5e-3 → 2e-4; DSA sparse adaptation: constant 1e-5[25]
Batch schedule
not disclosed
Precision
not disclosed
Parallelism
interleaved pipeline parallelism with flexible MTP placement (MTP output layer co-located with main output layer for parameter sharing), Pipeline ZeRO2 gradient sharding with double-buffered accumulation buffers, zero-redundant all-gather for the Muon optimizer, layer-granular pipeline activation offloading to host memory, sequence-chunked output projection, workload-aware sequence reordering with dynamic redistribution and flexible context-parallel groups using hierarchical all-to-all[26]
sequential RL pipeline: Reasoning RL (mixed domains: mathematics, science, code, tool-integrated reasoning with domain-specific judges), asynchronous Agentic RL (decoupled rollout engines via a Multi-Task Rollout Orchestrator, TITO gateway, direct double-sided importance sampling, DP-aware routing, >10K verifiable SWE/terminal/search environments), and General RL (foundational correctness, emotional intelligence, task-specific quality with hybrid rule-based/ORM/GRM rewards and human-authored anchors); on-policy cross-stage distillation as final stage[29]
Distillation
on-policy cross-stage distillation: final checkpoints of preceding SFT/RL stages serve as teachers, with the advantage term replaced by the log-ratio between teacher and student policies (group size 1, batch 1024)[30]
MLA low-rank latent KV cache (576-dim per token: KV LoRA 512 + 64); DSA lightning indexer retrieves top-k = 2048 KV entries per query so attention is computed sparsely over cached entries; multi-node DP-attention used in RL rollouts to avoid copying KV across ranks[32]
Quantisation shipped
INT4 QAT from the SFT stage with bitwise-identical training/inference kernel (§2.4.3); on Ascend: W4A8 mixed-precision (MoE experts W4A8/INT4, attention and MLP blocks W8A8/INT8, QuaRot + Flex_AWQ_SSZ calibration) fitting the 750B model on one Atlas 800T A3 (§5); FP8 for RL rollout inference (§3.6.2)[33]
Speculative decoding
Multi-Token Prediction with parameter sharing: 3 MTP layers share parameters during training (memory cost of a single MTP layer, matching DeepSeek-V3) while increasing acceptance rate; accept length 2.76 vs DeepSeek-V3.2's 2.55 at 4 speculative steps; MTP also accelerates RL rollouts[34]
Serving optimisations
slime RL infrastructure (mixed-precision rollouts, MTP, Prefill-Decode disaggregation, no-queue multi-node serving with DP-attention, heartbeat-driven fault tolerance); DP-aware routing with consistent hashing for KV-cache locality; on Ascend: fused Lightning Indexer / Sparse Flash Attention / MLAPO kernels, vLLM-Ascend and SGLang with asynchronous scheduling, RadixCache and prefix cache, FlashComm[35]
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
HLE
30.5
Table 7
HLE (w/ Tools)
50.4
Table 7
AIME 2026 I
92.7
Table 7
HMMT Feb. 2025
97.9
Table 7
IMO-AnswerBench
82.5
Table 7
GPQA-Diamond
86.0
Table 7
LongBench v2
64.5
Table 7
SWE-bench Verified
77.8
Table 7
SWE-bench Multilingual
73.3
Table 7
Terminal-Bench 2.0 (Terminus-2)
56.2 / 60.7
Table 7
CyberGym
43.2
Table 7
BrowseComp (w/ Context Manage)
75.9
Table 7
BrowseComp-ZH
72.7
Table 7
τ2-Bench
89.7
Table 7
MCP-Atlas (Public Set)
67.8
Table 7
Tool-Decathlon
39.2
Table 7
Vending-Bench 2
$4,432
Table 7
GDPval-AA Elo
1409
Table 7
Artificial Analysis Intelligence Index v4.0
50
§1
SWE-rebench (Resolved Rate)
42.1
Table 9
CC-Bench-V2 Backend Engineering (Pass@1)
25.8
Table 8
CC-Bench-V2 Long-horizon Chained Tasks (Pass@1)
52.3
Table 8
Provenance references
stated · Title
stated · Title page byline
stated · pipeline
stated · Abstract ('Code, models, and more information are available at https://github.com/zai-org/GLM-5'); §1 ('We release GLM-5 to the community'); §1 (first open weights model to score 50 on the AA Intelligence Index v4.0); §7 ('By open-sourcing the model')
stated · §2.1 (Multi-latent Attention), Table 10 (Q LoRA Dim 2048, KV LoRA Dim 512)
stated · §2.1.1, §3.2 ('DSA is lossless by construction ... enabling application to all layers'), Table 10 (# Indexer Attn Heads 32, # Indexer Head Dim 128)
stated · Table 10 (# Routed Experts 8)
stated · Table 10 (# Shared Experts 1)
stated · §2.3, §2.1.1
stated · §2.3 (200K stage), §3.1 (SFT extends max context to 202,752 tokens)
stated · §2.3 (mid-training extends the context window across 32K/128K/200K stages), §3.1 (SFT extends maximum context length to 202,752 tokens), §2.1.1 (DSA warm-up trains 202,752-token sequences)
stated · Appendix B.2 (SWE-bench evaluated 'with a 200K context window'), §6.1.1 (max generation 202,752 tokens for HLE-with-tools)
stated · §2 ('totaling 28.5 trillion tokens for the base model'), §1 ('we extend the training token budget to 28.5T tokens')
stated · §2.2
stated · §2, §2.3, §2.1.1
stated · §2.1, §2.4.1, Appendix A ('we follow the setting of GLM-4.5, including the Muon optimizer')
stated · Appendix A
stated · §2.4.1, §2.4.2
stated · §3.1 (multi-task SFT over General Chat, Reasoning, and Coding & Agent categories; INT4 QAT from the SFT stage; max context 202,752 tokens; interleaved/preserved/turn-level thinking modes)
stated · §3.2 ('Our RL algorithm builds upon GRPO [40] and incorporates the IcePop technique', group-normalized advantages, PPO-style clipped importance ratios, β=2, ϵlow=0.2, ϵhigh=0.28); §4.1 (agentic RL uses a group-wise policy optimization objective over K sampled traces)