Gemma 4 is the fourth generation of Google DeepMind's open Gemma family, a suite of five natively multimodal decoder-only Transformers spanning dense 31B, 12B, and the per-layer-embedding E4B/E2B models plus one sparse MoE (26B total, 3.8B active), all sharing one SentencePiece tokenizer with 262k entries. Architecturally it is a direct refinement of Gemma 3: the backbone keeps the 5:1 local sliding-window to global attention interleaving (4:1 on E2B) with dual RoPE bases (10k local, 1M global), but adds pp-RoPE with p=0.25 on global layers, reuses keys as values in global attention (values=keys, except E2B/E4B), and shares KV caches across layers at 20/35 or 18/42 ratios — together cutting the global KV cache by up to 37.5%. Blocks keep RMSNorm pre- and post-norm with QK-norm. Vision attaches through frozen ViT encoders (550M for 31B/26B-A4B, 150M for E2B/E4B) with axial 2D-RoPE and aspect-ratio-preserving resizing; audio exists only on E2B/E4B via a slimmed 305M USM-style Conformer encoder, while the flagship novelty is the 12B's encoder-free design, which feeds raw 40ms audio chunks and 48×48×3 image patches through a 35M matmul directly into the LLM. Every model ships with an MTP drafter for speculative decoding and QAT-quantized checkpoints (mobile int2/int4, Q4_0). A thinking mode — reasoning traces toggled by a <|think|> token — is the family's headline behavioral feature, and Arena results put the 31B at the top of open dense models.
What the report claims is novel
Thinking mode: models output a reasoning trace before responding, improving reasoning-heavy domains such as mathematics and coding [§1, §3]
Long-context efficiency: 5:1 local sliding-window to global attention (4:1 for E2B), pp-RoPE positional encoding, KV cache sharing, and keys-as-values reuse in global layers, reducing the global KV cache footprint by up to 37.5% [§1, §2]
An autoregressive MTP drafter head for speculative decoding that eliminates MTP prefill and supports any draft length [§1, §2.6]
Quantized checkpoints trained with quantization-aware training (mobile int2/int4 and Q4_0 formats) with minimal quality impact [§1, §2.5]
A unified, encoder-free architecture for the 12B model that projects raw 40ms audio chunks and image patches into the LLM embedding space, removing separate encoders and reducing memory fragmentation [§1, §2.3]
Gemma 4 31B is the leading dense open model on the Arena Text leaderboard, rivaling much larger open models in human-rated tasks [§4.1, Table 4]
Atlas assessment: A conservative, efficiency-focused evolution of the google-gemma-3 skeleton: same dense GQA-class decoder with 5:1 local/global attention (4:1 on E2B), dual RoPE bases, RMSNorm pre/post-norm + QK-norm, and the Gemini/SentencePiece tokenizer. The genuinely new mechanisms for the atlas are incremental rather than structural: keys-as-values reuse on global layers (values=keys, from Kayyam et al. 2026), cross-layer KV cache sharing (Shazeer 2019 MQA lineage), pp-RoPE with p=0.25 partial rotation on global layers (Barbero et al. 2025), per-layer embeddings on the small dense models (Gemma 3n lineage, not an atlas record), an EAGLE-style MTP drafter (deepseek-v3's MTP is the nearest atlas peer), QAT-shipped quantized checkpoints, and o1-style thinking mode. The encoder-free 12B (raw audio chunks + image patches through a 35M matmul) is the most distinctive claim — no atlas peer ingests raw patches without an encoder. Notably, the report does not name GQA, the FFN activation, or any MoE routing detail for 26B-A4B, and audio input is absent from the 31B and 26B-A4B variants despite the 'natively multimodal' framing. No new attention or mixing family is introduced.
Relation to the atlas
Lineage in
Local sliding-window / global attention interleaving (5:1, 4:1 for E2B) with dual RoPE bases (10k local / 1M global) from Gemma 3
Per-layer embeddings on the small dense models (E2B/E4B) from Gemma 3n
KV cache sharing across attention layers from Shazeer 2019 (MQA)
Keys-as-values reuse in global attention layers (values=keys) from Kayyam et al. 2026
pp-RoPE (partial rotary) positional encoding with p=0.25 on global layers from Barbero et al. 2025 (pp-RoPE)
Autoregressive MTP drafter head for speculative decoding from EAGLE (Li et al. 2024)
USM-based audio encoder (two downsampling convolutions + twelve Conformer layers), slimmed from 680M to 305M from Universal Speech Model (USM)
SentencePiece tokenizer with split digits, preserved whitespace, and byte-level encodings from Gemini 2.5 (Gemini Team 2025)
Influence out
No descendants recorded yet.
Notable omissions
No layer count, hidden dim, FFN dim, head count, or head dim for any LLM backbone (only the vision encoders are dimensioned, Table 10: 550M d_model 1152/d_MLP 4304/16 heads/27 layers; 150M 768/3072/12/16)
FFN activation function never named
MoE internals of 26B-A4B entirely undisclosed: no expert count, top-k, router, load balancing, or expert granularity
Inconsistent active-parameter figures for 26B-A4B: §2 text says 3.8B activated, Table 1 prints 2,800M active einsums (≈4.1B with encoders+embedder), Table 4 prints '26B / 4B'
No pre-training token count, FLOPs, or compute hours
No optimizer, learning rate, or batch schedule disclosed (pre-training 'similar to Gemma 3', §2.4)
Deployed context window never stated (Table 9 evaluates up to ~256k; Table 3 footprints at 32k); no context-extension method described
Post-training recipe deferred to Gemma 3; no preference-optimization method named; thinking-mode training recipe not disclosed
Distillation never mentioned (Gemma 3's report described distillation in detail)
GQA/MQA never named — KV-head structure of the attention is undisclosed despite cross-layer KV sharing being a headline feature
Input/output embedding tying not disclosed
Audio input absent from the 31B and 26B-A4B variants, so 'natively multimodal' applies unevenly across the family (Table 1)
5:1 interleaving of local sliding-window attention layers per global self-attention layer for all models except E2B (4:1); local window span not disclosed; global layers reuse keys as values (values=keys) except in E2B and E4B; KV cache shared across layers at ratios 20/35 (E2B) and 18/42 (E4B)[9]
Same tokenizer as Gemini Team [2025]; 262k vocabulary; IT formatting adds control tokens (<|think|>, <|turn|>, <|tool_call|>, <|channel>) for thinking mode and function calling (Table 11)[18]
Training
Training tokens
not disclosed
Data composition
Large-scale, diverse collection from a wide range of domains and modalities, including web documents, code, images, and audio (audio for E2B, E4B and 12B); data cutoff date January 2025; filtered to decontaminate benchmarks and reduce unwanted or unsafe utterances and recitation risk[19]
ZeRO-3 optimizer-state sharding; Pathways data-replica reduction over the data-center network; GSPMD partitioner with the MegaScale XLA compiler; Slice-Granularity Elasticity for failure recovery on the larger models; sharding by data, sequence, and replica (Table 2)[21]
Thinking mode: models generate a reasoning trace before responding (o1-style, cited OpenAI [2024]); activated via the <|think|> token in a leading system turn (Table 11); the training recipe behind the thinking mode is not disclosed[23]
Local sliding-window layers bound KV growth to the window span; global layers reuse keys as values (values=keys, except E2B/E4B) and use pp-RoPE p=0.25, reducing the global KV cache footprint by up to 37.5%; KV caches shared across layers at ratios 20/35 (E2B) and 18/42 (E4B); int8 KV caching offered (Table 3, '+KV' at 32k context)[26]
Quantisation shipped
QAT-trained quantized checkpoints: mobile quantization (per-channel low-bitwidth weights, int2/int4 mix, with int8 activation quantization) and Q4_0 blockwise quantization; raw checkpoints bf16; QAT also applied to encoders — 150M vision encoder W8A8 (forward memory 400→200 MB), audio encoder int8 activations with {2,4,8}-bit weights by layer cluster (on-disk 390→87 MB)[27]
Speculative decoding
Autoregressive MTP drafter head for speculative decoding: a 4-layer Transformer block (d_model 256 for E2B/E4B, 1024 for 26B-A4B/31B; 3 local + 1 global attention layers) with a separate embedder, cross-attending to the main model's KVs — no MTP prefill, any draft length; E2B/E4B drafters use cluster top-k decoding (d×262,000 → d×4,096)[28]