Nemotron 3 Ultra is a 108-layer, 550B-parameter sparse MoE (55B active per token) built as a scaled-up Nemotron 3 Super: a hybrid Mamba-Attention backbone in which Mamba-2 state-space layers dominate and sparse global-attention anchors (GQA, 64 query heads / 2 KV heads, head dim 128) provide periodic exact-attention interaction, with every layer carrying a LatentMoE FFN (512 experts per layer, top-22 activated, experts operating in a 2,048-wide latent space instead of the full 8,192 width, plus a shared expert with 10,240 intermediate size). Pre-training runs 20T tokens in NVFP4 — E2M1 weights with 2D block quantization and stochastic rounding, with the final 16 layers and all attention/latent/embedding/MTP projections held in higher precision — under a Warmup-Stable-Decay schedule (200B warmup to 2.5e-4, minus-sqrt decay over the final 5T), split into a 15T diversity phase and a 5T quality phase. Two training divergences are dissected in detail: a BF16 gradient-accumulation regression for the output layer (fixed by returning to FP32 reduction) and an undetermined divergence at ~16T (mitigated by early annealing to a 20T horizon). A 33B-token LC-Phase continuous-pretraining stage (92% of iterations at 1,048,576 tokens, 8% at 4K) delivers the 1M context. Post-training is agent-focused: two-stage SFT, unified RLVR, then two iterations of asynchronous Multi-teacher On-Policy Distillation from more than ten specialized teachers (SWE, office, search, terminal-use, chat, safety, STEM, coding…), finished with MTP Boosting that aligns the shared-weight two-block draft head to backbone logits for speculative decoding. Reasoning-effort control (off/regular/medium) tunes the accuracy-compute trade-off at inference.
What the report claims is novel
An open, efficient 550B-total / 55B-active MoE Hybrid Mamba-Attention model achieving up to ~6× higher inference throughput than state-of-the-art open LLMs (5.9× vs GLM-5.1-754B-A40B, 4.8× vs Kimi-K2.6-1T-A32B, 1.6× vs Qwen-3.5-397B-17B at 8K/64K) at on-par accuracy, with a 1M-token context for long-running agentic tasks [Abstract, §1, Figure 1]
LatentMoE for better accuracy per parameter than standard Granular MoEs, buying more routed experts at fixed inference cost by trading away hidden-dimension width [§1, §2.1, §5]
Native Multi-Token Prediction with two shared-weight heads used for speculative decoding throughout training, plus MTP Boosting (head-only forward-KL distillation) that closes the train-inference mismatch of the draft head [§2.1, §3.4]
The largest-scale demonstration of stable and accurate NVFP4 pre-training to date (20T tokens, E2M1 with 2D block quantization, <0.4% relative train-loss gap vs BF16 segments) [§2.2]
Agent-focused post-training: unified multi-environment RLVR followed by two iterations of asynchronous Multi-teacher On-Policy Distillation from more than ten domain-specialized teachers, with reasoning-effort control (reasoning-off / regular / medium-effort) and inference-time budget control [§3.2, §3.3, §3.5]
1M-token context via an LC-Phase continuous-pretraining recipe (92% 1M / 8% 4K iterations, 33B tokens) that preserves short-benchmark accuracy [§2.5]
Inference infrastructure for hybrid Mamba-Attention serving at scale: prefill-decode disaggregation with hybrid cache-state transfer upstreamed to vLLM, FlashInfer one-sided all-to-all, DWDP, EPLB hot-expert replication, MoE-side prefill chunking, and SSM-state snapshotting for draft rollback and prefix caching [§5.2, §5.3]
Full openness: base, post-trained, and NVFP4-quantized checkpoints plus training data, recipes, and RL environments released on HuggingFace [Abstract, §1, §6]
Atlas assessment: The block is a scaled-up Nemotron 3 Super: hybrid Mamba-2 + sparse GQA attention with LatentMoE FFNs, so within the atlas its closest relatives are ai21-jamba (hybrid SSM-attention, dense) and deepseek-v3 (frontier MoE + MTP) rather than any all-attention MoE. Biggest design differences vs deepseek-v3: the Mamba-2 majority backbone (attention only as sparse anchors, 2 KV heads) instead of all-MLA layers, NVFP4 pre-training instead of BF16/FP8, and the two-iteration MOPD pipeline instead of single-model RLVR. MTP-as-native-drafter and NVFP4 5.03-BPE mixed quantization are shared with the Nemotron 3 family rather than novel vs the atlas; MOPD (on-policy distillation of multiple teachers) mirrors the kimi-k3 and nemotron-cascade-2 direction and is not new to the atlas. The 550B/55B scale with a 512-expert LatentMoE (22 active) and 1M context is the largest hybrid-SSM record in the atlas. Claims of ~6× throughput are measured in-house on GB200 (TRT-LLM vs vLLM baselines) and are not independently verifiable.
Relation to the atlas
Lineage in
LatentMoE — routed experts operating in a compact latent space from LatentMoE (not in atlas)
Hybrid Mamba-Attention backbone — same architecture as Nemotron 3 Super (Mamba-2 layers + sparse global attention anchors) from Nemotron 3 Super (not in atlas)
Multi-Token Prediction (MTP) — shared-weight auxiliary heads doubling as a speculative draft model from Gloeckle et al. (not in atlas)
On-policy distillation of specialized teachers into one student (MOPD builds on Nemotron-Cascade-2 and Thinking Machines' on-policy distillation) from Nemotron-Cascade-2 (not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
Vocabulary size and tokenizer never disclosed
Normalization type/placement and activation function never named (LatentMoE / Mamba-2 internals deferred to the LatentMoE and Super reports)
MoE router function and load-balancing mechanism not described in this report (only MaxVio monitoring)
Shared-expert count not stated (Table 1 prints 'Total Experts per Layer: 512' without clarifying inclusion of the shared expert)
Pretraining optimizer and batch sizes not disclosed
No training compute (FLOPs/GPU-hours) disclosed
No license named
Input/output embedding tying not disclosed
Exact attention-vs-Mamba layer counts appear only in Figure 2 and are not stated in prose (PDF extraction of the figure is garbled)
RoPE / positional-encoding configuration of the attention layers not disclosed
Internal inconsistency: GLM-5.1 is '754B-A40B' in §1 but '744B-A40B' in Table 10
Total parameters (550B) vs active (55B) are stated but no per-component breakdown (attention + Mamba + routed/shared experts + embeddings) is given to verify the active count
sparse global Attention layers anchored among Mamba-2 layers: repeating groups of Mamba-2 + LatentMoE blocks with periodic Attention + LatentMoE anchors (Figure 2); exact per-layer counts are not stated in the prose[15]
fine-grained LatentMoE: routed experts operate in a compact latent space (MoE Latent Size 2048) rather than at full hidden width; 512 experts per layer with top-22 active
LC-Phase continuous pretraining (CPT): 1M context for 92% of iterations and 4K for 8% (math/code SFT-style data only), constant LR 2.5e-6, 33B tokens; long-context data 46% + Phase-2 data 54% in the blend; no RULER-style data[21]
Two-phase mixture (adaptation of the Nemotron 3 Super/Nano recipe): phase 1 (~15T tokens, ~75%) biases diversity, phase 2 (~5T) biases quality; 19 high-level categories, largest being quality-filtered and synthetic web crawl (~49% phase 1 / ~38% phase 2: crawl-medium/high, syn-crawl-medium/high), code 14%, math 6.4%, multilingual 5% (11 languages), nemotron-cc-code 2.1%, finepdfs (upweighted in phase 2), wiki, academic, legal, Crawl++, and synthetic SFT-style data (sft-code/stem/general); new datasets added since Super: Nemotron-Pretraining-Code-v3 (173B fresh GitHub tokens, cut-off Sep 30 2025), benchmark-seeded synthetic Multiple-Choice and Generative QA, Fact-Seeking (from Finewiki), Moral-Scenarios (CoT via Qwen3-235B), and a Legal family (regulations, case-law summaries, CaseHOLD/ContractNLI reformats, synthetic outcome/definition/diversity questions)[23]
Curriculum / staging
Two-phase pretraining (15T diversity-biased then 5T quality-biased, transition at ~75% of pretraining) under a Warmup-Stable-Decay LR schedule; offline checkpoint merging (sliding 500B-token merge window, 25B checkpoint interval) for evaluation; final checkpoint selected over a large merge sweep; followed by the 1M LC-Phase CPT (33B tokens)[24]
Optimizer
not disclosed
LR schedule
Warmup-Stable-Decay (WSD): 200B-token warmup to peak 2.5e-4, minus-sqrt decay over the final 5T tokens to 2.5e-6; MTP loss scaling factor 0.1 (0.05 per MTP block)[25]
Batch schedule
not disclosed
Precision
NVFP4 pre-training (E2M1 datatype with two-dimensional block quantization on weights, Random Hadamard Transforms on wgrad inputs, stochastic rounding on gradients); the final 15% of the network (16 layers), Mamba output projections, latent projections, QKV and attention projections, MTP layers, and embedding layers kept in higher precision; claimed largest-scale stable NVFP4 training to date[26]
Parallelism
LC-Phase: 32-way context parallelism, 8-way tensor parallelism, 128-way expert parallelism, 2-way pipeline parallelism; post-training RL on GB200 with Megatron EP groups co-located in one NVLink domain (NVLink-aware rank placement)[27]
Hardware
GB200 GPUs (LC-Phase 'train on GB200 GPUs'; production RL cluster on GB200 NVL72 nodes)[27]
hybrid cache: Mamba-2 layers keep a constant-size SSM state per sequence (FP32 at train, FP16 with stochastic rounding at deploy; block-scaled INT8 explored); global-attention layers keep a small GQA KV cache (2 KV heads, FP8 at deploy); the 32-bit Mamba cache is larger than the FP8 KV cache at sequence lengths up to 64K[33]
Quantisation shipped
Single NVFP4 checkpoint at 5.03 bits-per-element: NVFP4 routed-expert GEMMs (E2M1, dynamic max-based activation scaling, Four-Over-Six 4/6 weight scaling), FP8 per-tensor shared experts and Mamba linear layers, BF16 attention linear layers / MoE latent projections / embeddings / MTP layers, FP8 KV cache, FP16-SR Mamba cache; runs native W4A4 on Blackwell and W4A16 on Hopper[34]
Speculative decoding
Native Multi-Token Prediction (MTP): two shared-weight MTP blocks (each a single attention layer + single MoE layer) trained throughout pretraining/SFT/RL as an internal drafter; MTP Boosting (head-only forward-KL distillation, temperature 2, 7 MTP steps) fixes the train-inference mismatch; draft length is a deployment knob (peak 2.89× decode speedup at DL=6); SPEED-Bench average acceptance length 4.584 (greedy, boosted)[35]
Serving optimisations
TRT-LLM for the headline throughput numbers (vLLM for baselines); wide EP for high-throughput / wide TP for low-latency serving on GB200 NVL72 (single NVLink domain); prefill-decode disaggregation for hybrid Mamba-Attention landed in vLLM (~10% throughput gain); FlashInfer NVLinkOneSided all-to-all backend (~5% gain) and DWDP weight-prefetch alternative; EPLB hot-expert replication; MoE-side prefill chunking; SSM-state snapshotting at every draft step for speculative rollback and coarse-grained prefix caching[36]
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
Terminal Bench 2.1
56.4
Table 10
SWE-Bench Verified
70.7
Table 10
GDPVal
46.7
Table 10
TauBench V3 (Average)
70.9
Table 10
BrowseComp
44.4
Table 10
PinchBench
90.0
Table 10
ProfBench (Search)
56.0
Table 10
IOI 2025 (Score / 600)
570.0
Table 10
LiveCodeBench (v6)
89.0
Table 10
IMOAnswerBench (with tools)
92.3
Table 10
GPQA (no tools)
87.0
Table 10
MMLU-Pro
86.8
Table 10
HLE (no tools)
26.7
Table 10
AA-LCR
65.4
Table 10
RULER (1M)
94.7
Table 10
MMLU-ProX (avg 10 langs)
83.0
Table 10
IFBench (prompt loose)
81.7
Table 10
MMLU (base, 5-shot acc)
89.08
Table 2
RULER 64K (base)
95.3
Table 2
IMO-ProofBench Advanced (test-time scaling)
82.3% (173/210)
Table 11
Provenance references
stated · Title, Abstract
stated · Title page byline
stated · Title page (first line of the PDF: '2026-6-9')
stated · pipeline
stated · Abstract: 'We open-source the base, post-trained, and quantized checkpoints, along with the training data and recipe on HuggingFace'; §1 (checkpoint list: Base BF16, Post-Trained BF16, NVFP4, GenRM)
stated · Abstract, Table 1, §6 (550B total)
stated · Abstract, Table 1, §6 (55B active per token)
stated · Table 1 (Total Layers)
stated · Table 1 (Model Dimension)
stated · Table 1 (Expert Hidden Dimension); note: per-expert hidden dim for LatentMoE routed experts; shared expert intermediate size is 10240; MoE latent size is 2048
stated · Table 1 (Q-Heads n_q)
stated · Table 1 (KV-Heads n_kv) — GQA with 2 KV heads on the global attention layers
stated · Table 1 (Head Dimension)
stated · §2.1, §5 ('hybrid Mamba-2 stack with sparse global Attention anchors'); Table 1 (Q-Heads 64, KV-Heads 2, Head Dim 128 — the attention layers are GQA)
stated · Figure 2, §2.1, §5; note: the PDF text extraction of Figure 2 is partially garbled ('x3/x4/x2' multipliers), so exact attention-layer count is not recoverable from the stored text
stated · Table 1 (Total Experts per Layer)
stated · Table 1 (Top-k / Activated Experts)
stated · §2.5 (CPT on 1,048,576 (1M) context length for 92% of iterations)
stated · §2.5 (LC-Phase CPT at 1,048,576 (1M) context for 92% of iterations, 4,096 for the remaining 8%; 33B tokens total)
stated · Abstract, §3.7.2 ('supports contexts of up to 1M tokens')
stated · §2.5
stated · Abstract, §2.4 ('total horizon of 20 trillion tokens')
stated · §2.3, Figure 4
stated · §2.3.6, §2.4, §2.5
stated · §2.4
stated · §2.2
stated · §2.5, §3.6.2
stated · §3.1 (two-stage SFT: stage 1 packed 294,912-token sequences, global batch 64, 204,800 samples, cosine LR peak 1.5e-5; stage 2 packed 515,000-token sequences with long-context data up to 512K, 19,200 samples; shared-weight MTP objective retained)
stated · §3.2 (unified RLVR 'largely follows the asynchronous GRPO algorithm with the stability optimizations proposed in NVIDIA (2026)'; global batch 8192, 16 rollouts/sample, max generation 48K→64K tokens); §3.3.2 (Chat teacher additionally uses RLHF with an Ultra-based generative reward model / GenRM)