INTELLECT-3 is a 106B-parameter sparse MoE (12B active, 46 layers, hidden 4096, per-expert MoE dim 1408) that Prime Intellect built entirely by post-training Z.ai's GLM-4.5-Air base — the report discloses no block-level detail of its own, since every architectural attribute is inherited. The contribution is the training system around that fixed block. Two SFT stages establish behavior: a general chat-and-reasoning stage (~33M tokens/step at 65K context, Muon, warmup to 5e-5) and a smaller agentic stage at 98K context via context parallelism, using a Qwen3/GLM-style chat template that always prepends a <|think|> token and preserves reasoning across turns. Then a large-scale RLVR stage (batch 256 × 16 rollouts, 65,536 max context) optimizes the model across math, code, science, logic, deep-research (Serper search tools) and software-engineering environments with verifiable rewards — rule-based math-verify plus a CompassVerifier-7B LLM judge, sandbox test execution, and binary search rewards. The RL loop is the engineering centerpiece: prime-rl runs trainer (FSDP2) and inference (vLLM) disaggregated, with continuous batching, in-flight weight updates (max_off_policy_steps 8), a multi-client orchestrator for linear inference scaling, online difficulty filtering, and an IcePop-style double-sided masked importance-sampling objective (α=0.5, β=5) that the authors found critical for stable long-horizon training. Execution happens in Prime Sandboxes (gVisor containers, Rust gateway, image streaming, up to 256 sandboxes per node) at 4,000+ concurrent rollouts, on a 512-H200 cluster with distributed all-to-all Muon. The report frames the model as evidence for an open, reproducible RL stack as much as for the model itself.
What the report claims is novel
INTELLECT-3: state-of-the-art performance in its weight class, achieving 90.8% / 88.0% on AIME 2024/2025 (outperforming DeepSeek's frontier models and matching GLM-4.6, which has over 3× the parameters) and 69.3% on LiveCodeBench v6 (+8% over GLM-4.5-Air post-train) [Abstract, §1, Table 2]
prime-rl: an open, production-scale asynchronous RL framework (FSDP2 trainer + vLLM inference, disaggregated and individually multi-node deployable) with continuous batching and in-flight weight updates, a multi-client orchestrator with round-robin distribution, online difficulty filtering, and efficient MoE support [§1, §2.1]
verifiers and the Environments Hub: a unified, versionable environment interface and open registry that standardize agentic RL training and evaluation, decoupling environments from the trainer [§1, §2.2]
Prime Sandboxes: high-throughput secure code execution for RL (Rust-based gateway bypassing the Kubernetes control plane, gVisor runtime, image streaming + warm pools, 256 sandboxes per node) supporting over 4,000 concurrent sandboxes during training [§2.3, §3.1.2]
Distributed Muon for FSDP-sharded gradients via an all-to-all-based shard reshuffle, avoiding InfiniBand congestion from overlapping gathers [§2.1.7]
RL recipe: masked token-level importance sampling (IcePop-style) with double-sided masking and a 1e-5 token-ratio floor, found critical for stable long-context agentic RL; scaling RL to 512 H200s with high training efficiency [§3.3, Abstract]
Open-sourcing the complete stack — model, RL framework, environments, and recipe — so the same code spans single-node experiments to production-scale training [Abstract, §5]
Atlas assessment: INTELLECT-3 contributes no new model architecture: the report states plainly that it is an SFT+RL post-train of Z.ai's GLM-4.5-Air base, and every block-level attribute (attention, norm, activation, tokenizer, expert routing) is inherited and undisclosed. Its genuine novelty is the open RL infrastructure — prime-rl's disaggregated async trainer/inference with in-flight weight updates and multi-client orchestration, the verifiers/Environments Hub ecosystem, and Prime Sandboxes' control-plane-bypassing execution path — plus the RL algorithm detail of double-sided masked token-level importance sampling (α=0.5, β=5, 1e-5 floor) with ablations against GSPO/CISPO. vs deepseek-r1 (the atlas's other open RLVR post-train): INTELLECT-3 is built on an MoE base instead of R1's dense V3-base, uses Muon rather than AdamW, and emphasizes multi-turn agentic environments (SWE, deep research) alongside math/science. vs openai-gpt-oss: both are open post-trained reasoning MoEs, but INTELLECT-3's base (GLM-4.5-Air, ~355B-class MoE family) and full infrastructure disclosure differ. The headline claims (SOTA in weight class, matching GLM-4.6) rest on the report's own harness (Appendix A) and are not independently verifiable here; the report itself flags that its AIME numbers are conservative (no LLM judge).
Relation to the atlas
Lineage in
Base model and full architecture: SFT and RL run on top of the GLM-4.5-Air base (all block-level design inherited) from GLM-4.5-Air (Z.ai)
Masked token-level importance sampling objective (J_IcePop) with double-sided masking and group-relative advantages from IcePop (not in atlas)
Muon optimizer reuse in post-training (justified by the base model having been pretrained with Muon) from Muon (not in atlas)
Chat template with <|system|>/<|user|>/<|assistant|> and <|im_start|>/<|im_end|> control tokens, XML-style tool calls, <|think|> reasoning token from Qwen3 / GLM chat templates
Influence out
No descendants recorded yet.
Notable omissions
No expert count or experts-per-token disclosed (Figure 5's expert sweep is a kernel benchmark, not the model config)
No attention configuration (heads, KV heads, MLA vs GQA), position encoding, normalization, activation, or tokenizer — the entire block is inherited from GLM-4.5-Air and never restated
No vocab size, no embedding-tying statement
No license named and no release date printed (undated PDF, non-arXiv)
Deployed context length not stated (training max 65,536 RL / 98K agentic SFT)
No pretraining (post-train only) and no compute (FLOPs/GPU-hours) disclosure
Training precision not stated
No quantization, KV-cache, or serving-optimization discussion
No load-balancing method stated (MaxViolation is only a logged diagnostic)
MMLU-Pro 83.5 for GLM-4.5 is marked 'reported by AA Index' rather than measured in-house (Table 2 footnote)
SFT (Table 1): OpenReasoning-Math (2M/78.1B), OpenReasoning-Code (1.9M/94.3B), OpenReasoning-Science (310K/32B), OpenReasoning-Tool (800K/3.8B), AM General Chat (952K/8.4B), AM Instruction Following (54K/400M), SWE Swiss (10.3K/700M), Toucan Tool (116K/700M), Environments Mix (38.4K/1.9B); main sources are NVIDIA's Nemotron-Post-Training-Dataset-v1 and AM-DeepSeek-R1-0528-Distilled (synthetic reasoning traces from DeepSeek-R1-0528). RL environments: math (21.2K problems from Skywork-OR1, Acereason-Math, DAPO, ORZ-Hard), code (8.6K Python problems, DeepCoder-inspired, SYNTHETIC-2), science (29.3K from MegaScience), logic (11.6K from SynLogic), deep research (DeepDive: 1K SFT + 2.2K RL samples, Serper web search), software engineering (R2E-Gym and mini-swe-agent-plus scaffolds)[11]
Curriculum / staging
two-stage SFT (general chat-and-reasoning at 65K context, then agentic SFT at 98K context) followed by RL with online difficulty filtering (problems sorted into easy/normal/hard pools by observed solve rate, trivial rollouts discarded, easy-pool prompts with pass rate 1 removed); max_off_policy_steps = 8[12]
Optimizer
Muon (distributed via an all-to-all-based shard reshuffle for FSDP; Newton-Schulz update computed per rank on gathered gradient shards); weight decay 0.01 in SFT stage 1[13]
LR schedule
SFT stage 1: linear warmup from 1e-8 to 5e-5 over 300 steps; SFT stage 2: 5e-8 with linear decay over 800 steps; RL: Muon learning rate 1e-6[14]
Batch schedule
SFT: ~33M tokens per step, FSDP world size 64 with DP replicate size 8 across 512 GPUs; RL: batch of 256 prompts with 16 rollouts per prompt[14]
Precision
not disclosed
Parallelism
FSDP2 trainer (with tensor/context/expert parallelism support in torchtitan; expert parallelism evaluated and NOT enabled because grouped-GEMM kernels already saturate at the training's sequence length/hidden dim); vLLM inference backend; context parallelism used to reach 98K in agentic SFT; FSDP degree 32 at 48K sequence length; activation offloading to CPU for long sequences[15]
Hardware
512 NVIDIA H200 GPUs across 64 nodes (400Gbps NDR InfiniBand, ConnectX-7; Lustre + NVMe-NFS storage; Slurm with Cgroup v2); RL used 60 nodes (16 training / 44 inference at ~1:3)[16]
large-scale RLVR across math, code, science, logic, deep-research (web search) and software-engineering environments with verifiable rewards (rule-based math-verify plus CompassVerifier-7B LLM-judge; sandbox test execution; binary search rewards); long chain-of-thought with an always-reasoning chat template (<|think|> token, reasoning retained across turns); multi-turn agentic tool use (up to 200 turns in SWE scaffolds); online difficulty filtering curriculum[19]
vLLM OpenAI-compatible inference backend used for rollout generation, with custom /update_weights and /reload_weights endpoints for policy updates (§2.1.1); no other serving optimizations disclosed[23]
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
AIME 2024
90.8
Table 2
AIME 2025
88.0
Table 2
LiveCodeBench v6
69.3
Table 2
GPQA (Diamond)
74.4
Table 2
HLE
14.6
Table 2
MMLU-Pro
81.9
Table 2
Provenance references
stated · Title
stated · Title page (Prime Intellect Team; correspondence to [email protected])
stated · pipeline
stated · Abstract ('We open-source the model together with the full infrastructure stack'); §1 (footnote 2: huggingface.co/PrimeIntellect/INTELLECT-3); §5 ('By open sourcing INTELLECT-3, the environments, and the complete training framework')
stated · Abstract ('a 106B-parameter Mixture-of-Experts model (12B active)')
stated · Abstract ('12B active')
stated · §2.1.6 ('the activation memory for a sequence length of 48k, hidden size 4096, and 46 decoder layers is Memact = 46 × (48,000 × 4,096) × 2 bytes ≈ 18 GB')
stated · Figure 5 caption ('Execution time and TFLOPS of torch._grouped_mm with hidden dim 4096 and MoE dim 1408 on H200 SXM')
stated · §3.3 ('a maximum context length of 65536' during RL), §3.2 (general SFT at 65K context; agentic SFT scaled to 98K context via context parallelism), §2.1.6 (sequence lengths up to 72k with activation offloading)
stated · §3.1, Table 1, §3.2
stated · §3.2, §3.3, §2.1.5
stated · §3.2, §2.1.7
stated · §3.2, §3.3
stated · §2.1.1, §2.1.6, §2.1.8, §3.2
stated · §2.4, §3.3
stated · §3.2 (general chat-and-reasoning SFT stage plus agentic SFT stage; chat template with <|think|> token and Qwen3/GLM-style control tokens; always-reasoning model)
stated · §3.3 (RLVR objective J_IcePop: masked token-level importance sampling with double-sided masking M(k) for k∈[α,β], α=0.5, β=5, group-relative advantages Âi,t = Si − mean({Si}G); early ablations compared GSPO vs CISPO)
stated · §3.1, §3.2, §3.3
n/a · §1, §4 (text-only model)
stated · §A.2 (the report only notes that evaluations avoid quantization loss by using creator APIs where available)