Nemotron 3 Super

NVIDIA · 2026-04-03 · analysed 2026-08-10 · skill v0.3.5

Technical report ↗

Design overview

Nemotron 3 Super 120B-A12B scales the Nemotron 3 hybrid recipe to 88 layers, 120.6B total / 12.7B active parameters (12.1B excluding embeddings), pairing a predominantly Mamba-2 backbone with periodically inserted GQA attention 'anchors' (32 query / 2 KV heads, head dimension 128) for full-token interaction — per Figure 2 a 22-layer period with seven attention-containing blocks repeats across the stack. Its headline innovation is LatentMoE: tokens are down-projected from 4096 into a 1024-wide latent space where 512 experts (top-22 per token, expert hidden dimension 2688) do all routed computation and all-to-all traffic, with expert count and top-k scaled up by d/ℓ = 4; the router, shared experts (intermediate size 5376), and non-expert layers stay in full width. Two shared-weight MTP layers add a multi-token-prediction objective (loss scale 0.3) and double as a native draft model for speculative decoding, reaching an average acceptance length of 3.45 on SPEED-Bench. The model is NoPE with untied embeddings and RMSNorm and — a family first — was pre-trained entirely in NVFP4 (final 15% of layers, latent projections, MTP, QKV/attention projections and embeddings in BF16; Mamba output projection MXFP8) stably across 25T tokens. Post-training is agentic-heavy: two-stage SFT (7M samples, 80B tokens) with a per-conversation normalized second stage, multi-environment RLVR over 21 environments, a separate end-to-end SWE-RL stage, PivotRL for long-horizon agentic domains, RLHF with a principle-following GenRM, and MTP healing. Deployment ships FP8 (Hopper) and NVFP4 (Blackwell) checkpoints with 99.8% median accuracy.

What the report claims is novel

• First model in the Nemotron 3 family to be pre-trained in NVFP4: stable and accurate low-precision pretraining over 25T tokens (NVFP4 GEMM kernels from Transformer Engine, final 15% of network kept BF16) [Abstract, §2.2]
• LatentMoE — a new Mixture-of-Experts architecture optimizing accuracy per FLOP and per parameter: tokens projected from hidden dim d into a smaller latent dimension ℓ for routing and expert computation, with expert count and top-k scaled up by d/ℓ for higher accuracy at approximately constant inference cost [§2.1.1]
• MTP layers with shared weights for native speculative decoding: a unified prediction head exposed to multiple offsets that can be applied recursively at inference for longer drafts without an external draft model (highest average acceptance length 3.45 on SPEED-Bench at draft length 7) [§2.1.2, Table 2]
• Hybrid interleaved architecture with strategic GQA attention anchors over a predominantly Mamba-2 backbone: 88 layers, 120.6B total / 12.7B active parameters, 1M-token context, combining linear-time Mamba sequence modeling with full-token interaction [§2.1.3]
• Checkpoint merging (offline minus-sqrt decay emulation) as a compute-saving evaluation technique during the stable LR phase: saves ~4T tokens of compute (~16% of the total pretraining FLOP budget); the final base model selected for alignment was itself a 500B merge [§2.5]
• Scaled agentic post-training: 21 RL environments / 37 datasets, large-scale asynchronous RL on thousands of GPUs with in-flight weight updates and masked importance sampling, a dedicated end-to-end SWE-RL stage, and PivotRL for efficient long-horizon agentic RL [§3.2]
• FP8 and NVFP4 deployment checkpoints: NVFP4 PTQ combining MSE-calibrated weight scales, dynamic max-based activation scales, and AutoQuantize mixed-precision search (99.8% median accuracy vs BF16), plus an FP16-with-stochastic-rounding SSM cache recipe that eliminates recurrent quantization drift [§4]
• Up to 2.2× and 7.5× higher inference throughput than GPT-OSS-120B and Qwen3.5-122B respectively, at comparable or better benchmark accuracy (8k input / 64k output) [Abstract, §1]

Atlas assessment: LatentMoE is the first atlas entry that computes routed experts in a compressed latent space — the same design family as Kimi K3's Stable LatentMoE (K3 cites the LatentMoE line, §2.3), so this is a shared new mechanism rather than a unique one, and Super's variant is the simpler un-stabilized version (no RMSNorm/SiTU-GLU stabilization described; activation function not even named). NVFP4 from-scratch pretraining over 25T tokens is a genuine first for the atlas (peers train BF16/FP8). The shared-weight MTP with recursive drafting is a real refinement over DeepSeek-V3's per-layer MTP heads. The rest is Nemotron lineage: Mamba-2 + GQA anchors (Jamba/Nemotron-H), aux-loss-free balancing (DeepSeek-V3), WSD + checkpoint merging (MiniCPM/WSM), GenRM RLHF (RLBFF), and the two-stage SFT loss is a small but sensible fix for long-input-short-output degradation. The 99.8%-accuracy NVFP4 PTQ plus the stochastic-rounding SSM cache analysis (recurrent quantization error accumulation) is a strong systems contribution.

Relation to the atlas

Lineage in

• Hybrid Mamba-2 + GQA backbone with MoE FFNs from Nemotron 3 Nano / Nemotron-H / Jamba (Mamba-2); GQA (Ainslie et al., 2023)
• LatentMoE — experts operating in a compressed latent space from LatentMoE (Elango et al., 2026, not in atlas)
• Multi-Token Prediction (MTP) for quality and speculative decoding from DeepSeek-V3 / Gloeckle et al.
• Aux-loss-free bias-based load balancing from DeepSeek-V3
• Warmup-Stable-Decay LR schedule and checkpoint merging (weighted sliding-window averaging) from MiniCPM (WSD) / Model Soups + WSM
• Principle-following generative reward model (GenRM) for RLHF from RLBFF (Wang et al., 2025b, not in atlas)
• PivotRL — assistant-turn-level RL reusing offline SFT expert trajectories at uncertain 'pivot' turns from PivotRL (Yi et al., 2026, not in atlas)

Influence out

• nvidia-nemotron-3-ultra-550b-a55b: Hybrid Mamba-Attention backbone — same architecture as Nemotron 3 Super (Mamba-2 layers + sparse global attention anchors)

Notable omissions

• No vocabulary size or tokenizer algorithm disclosed
• No license named for the released weights
• Expert activation function never stated (Nemotron 3 Nano disclosed squared ReLU; Super is silent)
• Number of shared experts not disclosed (only the shared-expert intermediate size, 5376)
• Attention layer count not stated in the text — only the Figure 2 periodic pattern (7 attention-containing blocks per 22-layer period), whose exact repetition is left to the reader
• Main 25T-token pretraining hardware and parallelism not disclosed (only the LC-Phase: GB200, 64-way context / 2-way tensor / 64-way expert parallelism)
• No total training compute (FLOPs or GPU-hours) disclosed — only relative savings from checkpoint merging (~4T tokens ≈ 16% of the pretraining FLOP budget)
• Normalization placement (pre/post) and QK-norm not described
• Mamba SSM parameterization (discretization, decay initialization) not described despite Table 1 giving state dimension/groups/heads
• RLHF hyperparameters (prompts per step, generations per prompt, reward formula) deferred to the Nemotron 3 Nano recipe rather than restated

Closest relatives in the atlas

• Nemotron 3 Nano — 6 shared comparable fields
• Solar Open 2 — 5 shared comparable fields
• Nemotron 3 Ultra (Nemotron 3 family) — 5 shared comparable fields

Spec sheet

Identity

Scale

Core block

Context

Tokenizer

Training

Post-training

Modality

Inference efficiency

Evaluation

Benchmarks (report's own numbers only)

Provenance references

1. stated · Title
2. stated · Title page byline
3. stated · Title page date line (printed '2026-4-3')
4. stated · pipeline
5. stated · Abstract ('Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace'); §5
6. stated · §2.1 ('We extend this foundation to 120.6B total parameters'); Abstract ('120 billion (active 12 billion) parameter')
7. stated · §2.1 ('maintaining a constrained active budget of 12.7B parameters (12.1B excluding embeddings) per forward pass')
8. stated · Table 1 (Total Layers)
9. stated · Table 1 (Model Dimension)
10. stated · Table 1 (Expert Hidden Dimension — per routed expert; shared experts have intermediate size 5376)
11. stated · Table 1 (Q-Heads); §2.1.3
12. stated · Table 1 (KV-Heads); §2.1.3 ('Grouped-Query Attention (GQA) with 32 query heads and 2 KV heads')
13. stated · Table 1 (Head Dimension); §2.1.3
14. stated · §2.1.3 ('maintain un-tied embedding and output weights')
15. stated · §2.1.3 ('The attention layers employ Grouped-Query Attention (GQA) with 32 query heads and 2 KV heads (head dimension 128)')
16. stated · §2.1.3, Figure 2, Table 1
17. stated · Table 1 (Total Experts per Layer); §2.4 ('MoE layers with 512 total experts')
18. stated · Table 1 (Top-k (Activated Experts) 22); §2.1.3 ('top-22 routing')
19. stated · §2.4; §2.1.1
20. stated · §2.4 (cites Wang et al., 2024; DeepSeek-AI, 2025c; Lepikhin et al., 2020)
21. stated · §2.1.1, Table 1
22. stated · §2.6
23. n/a · §2.6 (no rescaling/interpolation factor; no RoPE)
24. stated · §2.1.3 ('This configuration supports context lengths of up to 1M tokens'); RULER 1M evaluated (Table 4, Table 5)
25. stated · §2.1.3 ('we omit positional embeddings')
26. n/a · §2.1.3 (no positional embeddings of any kind; no RoPE anywhere)
27. n/a · §2.1.3 (no RoPE)
28. stated · §2.1.3 ('use RMSNorm for normalization')
29. stated · §2.6 (LC-Phase CPT on 1,048,576 (1m) context length for 34B tokens, then a stage alternating 1m and 4k sequences for 17B tokens)
30. stated · §2.1.3 ('supports context lengths of up to 1M tokens'); RULER 1M (Table 4, Table 5)
31. stated · Abstract ('pre-trained Nemotron 3 Super on 25 trillion tokens'); §1 (20T phase 1 + 5T phase 2)
32. stated · §2.3
33. stated · §2.3.7, §2.5, §2.6, §3.2.1
34. stated · §2.4
35. stated · §2.4, §2.6
36. stated · §2.4, §2.6, §3.1, §3.2.4
37. stated · §2.2, Table 3
38. stated · §2.6, §3.2.4, §3.2.5
39. stated · §2.6, §4.2, Figure 1 caption, Figure 5 caption
40. stated · §3.1 (two-stage SFT over 7M samples / 80B tokens: Stage 1 token-level average loss with 256k packing, global batch 64, constant lr 1e−5 with 30k warmup samples; Stage 2 per-conversation normalized loss with 512k packing, global batch 32, lr 1e−5, long-context data up to 512K; MTP objective continued with 0.3-scaled auxiliary loss; low-effort reasoning mode added, 2% of SFT data; reasoning-off via 3% trace stripping; 350-step semi-on-policy budget-control stage truncating 12% of traces)
41. stated · §3.2.3 (separate RLHF-only stage at the end of post-training, plus GenRM used throughout multi-environment RL; principle-following GenRM initialized from Qwen3-235B-A22B-Thinking-2507, trained on Helpsteer 3, commercially friendly lmarena-140k subsets, and recent human preference data)
42. stated · §3.2
43. n/a · modality.type (text-only)
44. stated · §4.1, §4.2, §4.3
45. stated · §4
46. stated · §2.1.2, Table 2, Figure 5
47. stated · Figure 1 caption, §4.2, Figure 5 caption