Gemma 3 is Google DeepMind's third-generation open decoder-only transformer family (1B–27B), and its headline architectural change targets long-context inference cost: the uniform attention stack of Gemma 2 gives way to a 5:1 interleaving of local and global layers, starting with a local layer, where each local layer is a sliding-window self-attention with a span of only 1024 tokens and each global layer keeps full-length attention. Only the global layers therefore attend over the full 128K context, cutting KV-cache memory overhead from roughly 60% of model memory (global-only) to under 15% at 32K prefill. Attention remains grouped-query (GQA) with RMSNorm pre- and post-norm, and Gemma 2's logit soft-capping is replaced by QK-norm, borrowed from the ViT and Chameleon lines. Long context is reached by pre-training at 32K and rescaling RoPE at the end of pre-training (positional-interpolation-style factor 8), with the global layers' RoPE base raised from 10k to 1M while local layers stay at 10k. Multimodality is an adapter: a frozen 400M SigLIP encoder (896×896, shared across 4B/12B/27B) converts images into 256 soft tokens with 4×4 average pooling, plus an inference-time Pan & Scan windowing scheme for arbitrary aspect ratios; the 1B variant is text-only. All models are trained with knowledge distillation (teacher unnamed) on 2T–14T tokens, and the IT models use an improved post-training recipe built on distillation and BOND/WARM/WARP-style RL with math and code-execution rewards. The report is unusually light on architecture numbers: it discloses no layer counts, hidden dims, FFN widths, or head counts for any variant.
What the report claims is novel
Multimodal addition to the Gemma family: a tailored 400M SigLIP vision encoder shared across 4B/12B/27B, images condensed to a fixed 256 soft tokens, with a Pan & Scan adaptive windowing algorithm for native-aspect-ratio, high-resolution images [§1, §2.1]
KV-cache reduction for long context: 5:1 interleaving of local (sliding-window, 1024-token span) and global attention layers, so only global layers attend the 128K context [§1, §2]
Replacing Gemma 2's soft-capping with QK-norm (inspired by Dehghani et al. 2023, Wortsman et al. 2023, Chameleon Team 2024) [§2]
A novel post-training recipe (improved knowledge distillation from a large IT teacher plus RL fine-tuning based on BOND, WARM, and WARP) that significantly improves math, chat, instruction-following, and multilingual abilities [§1, §3]
Multilinguality: wider language coverage via increased monolingual and parallel data with Unimax-style imbalance handling; Global MMLU-Lite 75.1 and WMT24++ 53.4 on the 27B IT model [§1, §2.2, Table 18]
Scale-relative performance: Gemma3-4B-IT competitive with Gemma2-27B-IT, and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks [Abstract, §4]
Atlas assessment: The block is a conservative evolution of the dense GQA decoder-only skeleton shared with meta-llama-3.1/meta-llama-3 (RMSNorm pre/post-norm, RoPE, SwiGLU-class FFN by lineage). The genuinely new axis for the atlas is the per-layer attention pattern: a 5:1 local/global interleaving with a 1024-token sliding-window span (the first sliding-window record in the atlas; phi-3-small's blocksparse pattern is the nearest peer but is a different mechanism) and the dual RoPE bases (10k local / 1M global). QK-norm replacing soft-capping follows deepseek-v3's QK-norm practice but is applied to a dense GQA model. Long context reuses positional interpolation (factor 8) rather than a new mechanism. Vision is an adapter (frozen SigLIP + P&S), not native. The 'novel post-training recipe' is a training contribution (distillation from an unnamed teacher + BOND/WARM/WARP-based RL), and the report's flagship claim (4B-IT ≈ Gemma2-27B-IT; 27B-IT ≈ Gemini-1.5-Pro) is internal to the report and not independently verifiable. The report is unusually sparse on architecture disclosure: no layer counts, hidden dims, FFN widths, or head counts for any variant.
Relation to the atlas
Lineage in
Grouped-Query Attention (GQA) — KV sharing retained for all attention layers from GQA (Ainslie et al., 2023)
Sliding-window local attention — 1024-token span on local layers from Longformer (Beltagy et al., 2020)
QK-norm replacing soft-capping from Dehghani et al. 2023 / Wortsman et al. 2023 / Chameleon (not in atlas)
Positional interpolation (RoPE rescaling, factor 8) for context extension from Chen et al. 2023 (positional interpolation)
SigLIP vision encoder (400M, frozen) with LLaVA-inspired flexible-resolution Pan & Scan from SigLIP (Zhai et al., 2023) / LLaVA (Liu et al., 2024) (not in atlas)
Knowledge distillation for pre-training and instruction tuning (256 sampled logits per token) from Hinton et al. 2015 (knowledge distillation)
Influence out
jetbrains-mellum2-thinking-12b-a2-5b: Layer-selective context scaling — extension applied only to global-attention layers ('first reported in the Gemma 3 technical report… subsequently adopted by OLMo 3'); Mellum 2 uses YaRN as the scaling method
Notable omissions
No layer counts, hidden dimensions, FFN widths, or attention head counts for any variant (no configuration table anywhere in the report; Table 1 gives parameter counts only)
FFN activation function never named (Gemma 2 lineage implies GeGLU but the report does not state it)
Vocab size printed inconsistently: 262k entries in §2.2 vs 256k in the Table 1 caption
No training optimizer, learning-rate schedule, batch sizes, or training precision disclosed
No training compute (FLOPs or TPU-hours) disclosed
No license named for the model weights (arXiv page's CC BY 4.0 covers the paper only)
The distillation teacher (pre-training and IT) is never identified
No per-variant architecture table (e.g. which layers are local/global per size, or per-size RoPE bases)
Input/output embedding tying not disclosed
Only the 1B–27B sizes appear in this report; a sub-1B (270M-class) variant is not covered
RoPE rescaling (positional-interpolation-style, factor 8) applied at the end of pre-training to extend the 32K-trained 4B/12B/27B models to 128K; global-layer RoPE base raised 10k → 1M while local layers stay at 10k[22]
Same tokenizer as Gemini 2.0; ~262k entries, more balanced for non-English languages; PT and IT models share the tokenizer, with control tokens reserved for IT formatting (<start_of_turn>/<end_of_turn>); a [BOS] token must be added explicitly (add_bos=True); PT models end generation with <eos>, IT models with <end_of_turn>[24]
Slightly larger token budget than Gemma 2; mixture of images and text; increased multilingual data (both monolingual and parallel) with language-imbalance handling inspired by Unimax (Chung et al. 2023); filtering to reduce unsafe/unwanted utterances and personal information, decontamination of evaluation sets, recitation-risk reduction, and a quality reweighting step (Sachdeva et al. 2024)[26]
Curriculum / staging
Pre-train at 32K sequences, then rescale the 4B/12B/27B models to 128K at the end of pre-training via RoPE rescaling (factor 8); knowledge distillation during pre-training; per-size token budgets (2T/1B, 4T/4B, 12T/12B, 14T/27B)[27]
Optimizer
not disclosed
LR schedule
not disclosed
Batch schedule
not disclosed
Precision
not disclosed
Parallelism
ZeRO-3 optimizer-state sharding; data-replica reduction over the data center network (Pathways) for multi-pod training; sharding by data, sequence, and replica (Table 2); JAX/Pathways single-controller paradigm with the GSPMD partitioner and the MegaScale XLA compiler[28]
Hardware
TPUv4, TPUv5e, and TPUv5p (1B: 512 TPUv5e chips; 4B: 2048 TPUv5e; 12B: 6144 TPUv4; 27B: 6144 TPUv5p); vision embeddings pre-computed so the vision encoder adds no training cost[28]
RL objectives cover helpfulness, math (ground-truth rewards for solving math problems), coding (code execution feedback), reasoning, instruction-following, and multilingual abilities; no long-CoT / RLVR pipeline is described[31]
Distillation
All Gemma 3 models are trained with knowledge distillation (256 logits per token sampled, weighted by teacher probabilities, cross-entropy on the renormalized teacher distribution); IT models use an improved distillation from a large IT teacher; the teacher model is never named[32]
5:1 local:global interleaving with a 1024-token local span bounds each local layer's KV cache at 1024 tokens while only global layers attend the full context; report measures KV-cache memory overhead falling from ~60% (global-only) to <15% (5:1, sw=1024) of model memory at 32K prefill[35]
stated · Header: 'arXiv:2503.19786v1 [cs.CL] 25 Mar 2025'
stated · pipeline
stated · Abstract: 'We release all our models to the community.'
stated · Abstract, Table 1
stated · §2.2 Tokenizer
stated · §1, §2
stated · §2 ('We alternate between a local sliding window self-attention and global self-attention, with a pattern of 5 local layers for every global layer, starting with a local layer as the first layer of the model'); §1
n/a · block_type dense
stated · §5.3 ('we pre-train our models with 32K sequences and then scale the 4B, 12B, and 27B models up to 128K tokens at the end of pre-training while rescaling RoPE [Chen et al., 2023]'); §2 ('We follow a process similar to the positional interpolation of Chen et al. [2023] to extend the span of the global self-attention layers')
stated · §5.3 ('We find a scaling factor of 8 to work well in practice')
stated · §2 Long context ('Gemma 3 models support context length of 128K tokens, with the exception of the 1B model that has 32K')
stated · §2 Long context ('We increase RoPE base frequency from 10k to 1M on global self-attention layers')
stated · §2, §5.3
stated · §2 ('Grouped-Query Attention (GQA) with post-norm and pre-norm with RMSNorm')
stated · §2 ('with post-norm and pre-norm with RMSNorm' — both pre- and post-norm applied)
stated · §2 ('Inspired by Dehghani et al. [2023], Wortsman et al. [2023] and Chameleon Team [2024], we replace the soft-capping of Gemma 2 with QK-norm.')
stated · §2 ('we replace the soft-capping of Gemma 2 with QK-norm')
stated · §5.3 ('pre-train our models with 32K sequences and then scale the 4B, 12B, and 27B models up to 128K tokens at the end of pre-training')
stated · §2 Long context ('Gemma 3 models support context length of 128K tokens')
stated · §5.3, §2
stated · §2.2 Tokenizer ('a SentencePiece tokenizer with split digits, preserved whitespace, and byte-level encodings')
stated · §2.2, §3, Table 4
stated · §2.2 Pre-training ('we train on 14T tokens for Gemma 3 27B, 12T for the 12B version, 4T for the 4B, and 2T tokens for the 1B')
stated · §2.2
stated · §5.3, §2.2
stated · §2.4, Table 2
stated · §3 ('Pre-trained models are turned into instruction-tuned models with an improved post-training approach'); §7.2 ('we also use both SFT and RLHF')
stated · §7.2 ('both SFT and RLHF'); §3
stated · §3
stated · §2.2 Distillation, §3, Abstract ('The Gemma 3 models are trained with distillation')
stated · §1 ('most Gemma 3 models are compatible with a tailored version of the SigLIP vision encoder')