Nemotron 3 Nano 30B-A3B is an open, efficient sparse MoE hybrid Mamba-Transformer: 52 layers, 31.6B total parameters with only 3.2B active per forward pass (3.6B including embeddings). The backbone follows the Nemotron-H / Nemotron 2 Nano lineage — a Jamba-style interleaving of Mamba-2 state-space layers and Grouped-Query-Attention layers, 6 of 52 layers being GQA (32 query heads, 2 KV heads, head dimension 128) and the remaining 46 being Mamba-2 (state dimension 128, 8 groups, 64 heads) — with every standard FFN replaced by a granular MoE: 128 routable experts of dimension 1856, 6 activated per token plus 2 shared experts, a learnt MLP sigmoid router, squared ReLU activation, and DeepSeek's aux-loss-free bias-based load balancing combined with a small auxiliary loss. The model is fully NoPE: no positional embeddings, no dropout, no linear-layer bias, RMSNorm, untied embeddings. Context reaches 1M tokens through a final LC-Phase of continuous pretraining on a 512k/4k sequence mixture, with no positional-encoding extension required. Post-training runs SFT over chat, agentic, and reasoning traces with reasoning on/off and token-budget control, a unified multi-environment RLVR stage (synchronous GRPO with masked importance sampling and frozen router weights), and RLHF with a GRPO-trained generative reward model plus Group Relative Length Control that cut verbosity 30% without accuracy loss. Deployment ships BF16 and a selective FP8 PTQ checkpoint that keeps the 6 attention layers and their feeding Mamba layers in BF16, recovering ~99% median accuracy.
What the report claims is novel
Open, efficient MoE hybrid Mamba-Transformer for agentic reasoning: 31.6B total / 3.2B active parameters (3.6B including embeddings), better or on-par accuracy vs GPT-OSS-20B and Qwen3-30B-A3B-Thinking-2507 with up to 3.3× higher inference throughput (8K input / 16K output on a single H200) [Abstract, §1]
Granular MoE with shared experts: 128 routable experts (6 active per token) with squared ReLU and a learnt MLP sigmoid router, achieving 'better accuracy at a fraction of the active parameter count' [§2.1]
25T-token pretraining with new open datasets: Nemotron-CC-v2.1 (2.5T new English tokens incl. translation from 9 languages), Nemotron-CC-Code-v1 (427.92B code tokens), Nemotron-Pretraining-Code-v2, and Nemotron-Pretraining-Specialized-v1 (incl. InfiniByte cross-domain problem synthesis and 4.3M RQA STEM reasoning demonstrations) [§2.2]
First Nemotron effort to scale RL in post-training: unified multi-environment RLVR training on all environments simultaneously, followed by RLHF with a GRPO-trained generative reward model and a novel Group Relative Length Control mechanism that reduced verbosity 30% without sacrificing accuracy [§3]
Selective FP8 PTQ keeping the 6 self-attention layers and their 6 feeding Mamba layers in BF16: ~99% median accuracy recovery with FP8 KV cache and significantly higher throughput [§4]
Open release: base and post-trained checkpoints on HuggingFace, plus training recipe, code, and 'most of the data' used to train the model [Abstract, §5]
Atlas assessment: Architecturally an incremental combination of established mechanisms: the Mamba-2 + GQA hybrid follows the Jamba / Nemotron-H / Nemotron 2 Nano lineage, the granular MoE with shared experts follows DeepSeekMoE, and the aux-loss-free bias-based load balancing follows DeepSeek-V3. The genuinely distinctive contributions are in post-training and efficiency: unified multi-environment RLVR (all environments simultaneously), RLHF with a generative reward model plus Group Relative Length Control (a new, simple reward shaping that cut verbosity 30% with no accuracy loss), and a selective FP8 PTQ scheme (6 attention + 6 feeding Mamba layers kept BF16). Squared ReLU MoE at 6/128 routing is a notable but minor variant. Nothing here is a new mixing mechanism; the report's headline is the accuracy/throughput trade-off for an open 30B-class agentic model.
No vocabulary size or tokenizer algorithm disclosed
No license named for the released weights
Main 25T-token pretraining hardware and parallelism not disclosed (only the LC-Phase: H100, 8-way context/tensor/expert parallelism, 4-way pipeline)
No total training compute (FLOPs or GPU-hours) disclosed
Normalization placement (pre/post) and QK-norm not described
Exact attention-layer interleaving ambiguous: text states 6 of 52 layers are self-attention (§4.2), but Figure 2's repeat counts (x5/x3/x1/x4) do not obviously reconcile with that count in the extracted figure text
No speculative decoding or multi-token prediction (unlike Nemotron 3 Super)
Mamba SSM parameterization (discretization, decay initialization) not described despite Table 1 giving state dimension/groups/heads
Base-model comparison to GPT-OSS-20B omitted because no GPT-OSS base model was released (report states this explicitly)
Peak learning rate and batch size are disclosed, but no loss-curve or scaling-law data
6 of 52 layers are GQA self-attention layers, the remaining 46 are Mamba-2; Figure 2 depicts a periodic interleaving of Mamba-2+MoE and Mamba-2+Attention+MoE blocks (repeat counts x5/x3/x1/x4 per the figure) — the extracted figure text does not fully resolve the exact interleaving, but §4.2 states 6 self-attention layers and that the 6 Mamba layers feeding them are kept in BF16[16]
standard learnt MLP router with sigmoid gating; router weights frozen during RLVR to stabilize training[21]
Load balancing
DeepSeek aux-loss-free load balancing (expert-bias based) with update rate 10−3, used in conjunction with the standard load balancing loss with coefficient 10−4; sequence-level MoE load balancing regularizer (coefficient 10−4) during SFT[22]
Expert granularity
granular MoE (DeepSeekMoE-style fine-grained experts) with shared experts: 128 routable experts of dimension 1856, 6 active per token, plus 2 shared experts[23]
LC-Phase continuous pretraining (CPT): 512k and 4k sequence mixture, 121B tokens, with 20% long-context document QA and 1% synthetic retrieval-focused data (up to 256k); no positional-encoding modification needed (NoPE)[24]
15 data categories: web crawl subdivided into five quality groups (crawl-medium, crawl-medium-high, syn-crawl-medium-high, crawl-high, syn-crawl-high) plus math, Wikipedia, code, nemotron-cc-code, academic text, Crawl++ (OpenWebText, BigScience, Reddit), multilingual (19 languages), and synthetic SFT-style datasets (general-sft, stem-sft, code-sft); new released datasets: Nemotron-CC-v2.1 (~2.5T new English tokens: 3 fresh Common Crawl snapshots, 5-prompt rephrasing of Medium-High data from 110 snapshots, translation to English from 9 languages), Nemotron-CC-Code-v1 (427.92B-token code corpus via Lynx + Phi-4 cleaning), Nemotron-Pretraining-Code-v2 (GitHub refresh + Qwen3-32B synthetic dialogues, SGCR/SCOR rephrasing, Python-to-C++ transpilation), Nemotron-Pretraining-Specialized-v1 (synthetic Wikipedia, math textbooks, scientific coding, InfiniByte cross-domain code, RQA 4.3M demonstrations / ~31.7B tokens, DQA ~8B tokens)[34]
Curriculum / staging
two-phase curriculum: Phase 1 promotes diversity, Phase 2 primarily high-quality datasets (e.g., Wikipedia), switch at the 94% point of training; LC-Phase CPT at the end (512k/4k mixture); RLVR uses a difficulty curriculum with Gaussian pass-rate sampling that shifts from easy to hard tasks across training steps[35]
Warmup-Stable-Decay: warmed up over 8.4B tokens to a maximum of 10−3, maintained for 80% of training (20T tokens), decayed to a minimum of 10−5 over the final 5T tokens; LC-Phase used a constant LR of 10−5[37]
Batch schedule
pretraining: sequence length 8192, batch size 3072 (~25M tokens per batch); LC-Phase: global batch size 48; SFT: batch size 64 with sequence packing to 256K; RLVR: batch size 2048 (128 prompts/step, 16 generations/prompt)[38]
multi-environment RLVR from verifiable rewards: all environments trained simultaneously (competition math DAPO/SkyWorks, competition coding, STEM MCQ QA, structured outputs/JSON schema, instruction following, long context, agentic tool use incl. Workplace Assistant and multi-turn conversational agent); synchronous GRPO with masked importance sampling, 128 prompts/step, 16 generations/prompt, batch 2048, max generation length 49K, overlong filtering, difficulty curriculum; two RLVR stages (after SFT and after RLHF); reasoning on/off and token-budget control via the chat template[43]
GQA with 2 KV heads (KV cache ∝ 2 heads); KV cache quantized to FP8 via PTQ — FP8 KV cache significantly improves throughput by enabling larger batch sizes[45]
Quantisation shipped
FP8 post-training quantization (PTQ) via ModelOpt and Megatron-LM with selective BF16 retention: the 6 self-attention layers, the 6 Mamba layers feeding them, and all Mamba Conv1D layers kept in BF16; weights, activations, and KV cache FP8; ~99% median accuracy recovery[46]
Speculative decoding
none disclosed
Serving optimisations
inference throughput measured with vLLM and TRT-LLM (best of the two per model) on a single H200 GPU with FP8 weights and activations; no dedicated serving-optimization stack described[47]
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
MMLU-Pro
78.3
Table 3
AIME25 (no tools)
89.06
Table 3
AIME25 (with tools)
99.17
Table 3
GPQA (no tools)
73.04
Table 3
GPQA (with tools)
75.0
Table 3
LiveCodeBench (v6 2024-08↔2025-05)
68.25
Table 3
SciCode (subtask)
33.28
Table 3
HLE (no tools)
10.57
Table 3
HLE (with tools)
15.48
Table 3
MiniF2F pass@1
50.03
Table 3
MiniF2F pass@32
79.92
Table 3
Terminal Bench (hard subset)
8.51
Table 3
SWE-Bench (OpenHands)
38.76
Table 3
TauBench V2 Airline
48.0
Table 3
TauBench V2 Retail
56.91
Table 3
TauBench V2 Telecom
42.21
Table 3
TauBench V2 Average
49.04
Table 3
BFCL v4
53.76
Table 3
IFBench (prompt)
71.51
Table 3
Scale AI Multi Challenge
38.45
Table 3
Arena-Hard-V2 (Average)
67.65
Table 3
AA-LCR
35.85
Table 3
RULER-100 @ 256k
92.92
Table 3
RULER-100 @ 512K
91.25
Table 3
RULER-100 @ 1M
86.34
Table 3
MMLU-ProX (avg over langs)
59.5
Table 3
WMT24++ (en→xx)
86.2
Table 3
Provenance references
stated · Title
stated · Title page byline
stated · Title page date line (2025-12-23)
stated · pipeline
stated · Abstract: 'We release both our pretrained Nemotron 3 Nano 30B-A3B Base and post-trained Nemotron 3 Nano 30B-A3B checkpoints on Hugging Face'
stated · §2.1 ('31.6B total parameters'); Abstract
stated · §2.1 ('3.2B are active (3.6B including embeddings) per forward pass'); Abstract ('activating less than half of the parameters per forward pass')
stated · Table 1 (Num Layers)
stated · Table 1 (Model Dimension)
stated · Table 1 (Expert Dimension)
stated · Table 1 (Q-heads)
stated · Table 1 (KV-heads); §1 (GQA)
stated · Table 1 (Head Dimension)
stated · §2.1 ('un-tie embedding and projection weights')
stated · §1 ('combination of Mamba-2 (Dao & Gu, 2024) and Grouped-Query-Attention (GQA) (Ainslie et al., 2023)')
stated · §4.2 ('self-attention layers (6 out of 52 layers for Nemotron 3 Nano)'); Figure 2
stated · §2.1 ('For the MoE layers, we use squared ReLU activation')
stated · Table 1 (Total Routable Experts)
stated · Table 1 (Number of Activated Experts); §1 ('activates 6 out of 128 experts')
stated · Table 1 (Number of Shared Experts)
stated · §2.1; §3.2.5
stated · §2.4; §3.1.6; §3.2.5
stated · §2.1 ('granular MoE architecture along with shared experts (Dai et al., 2024)'); Table 1
stated · §2.5
n/a · §2.5 (no rescaling/interpolation factor; no RoPE)
stated · §1, §5 ('supports context lengths up to 1M tokens'); RULER-100 @ 1M evaluated (Table 3)
stated · §2.1 ('We do not use any positional embeddings')
n/a · §2.1 (no positional embeddings of any kind; no RoPE anywhere)
n/a · §2.1 (no RoPE)
stated · §2.1 ('We use RMSNorm for normalization')
stated · §2.5 (LC-Phase CPT on batches of 524,288 (512k) token sequences, mixed with 4k sequences; base pretraining at 8192)
stated · §1 ('supports context lengths up to 1M tokens'); RULER-100 @ 1M (Table 3)
stated · §4 ('After post-training the model in BF16, we applied Post-Training Quantization (PTQ) using ModelOpt and Megatron-LM to quantize the model to FP8')
stated · §2.5 ('to train on H100 GPUs'); throughput measured on a single H200 GPU (§1)
stated · §3.1 (SFT over 18M samples; 13000 steps, batch size 64, sequence packing to 256K, LR 5·10−5 with 800 warmup steps; chat/agentic/reasoning traces; reasoning on/off control by stripping traces from 10% of samples, budget control by truncating 3%)
stated · §3.3 (RLHF with a generative reward model (GenRM) built by GRPO-training Qwen3-235B-A22B-Thinking-2507; Group Relative Length Control with length-normalized reward adjustment and quality-gated conciseness bonus; verbosity reduced 30% without accuracy loss)
stated · §3.2
n/a · modality.type (text-only)
stated · §4.2 ('The model weights, activations, and KV cache are quantized to FP8'), §4.3