OLMo 3

Allen Institute for AI (Olmo Team) · 2025-12-15 · analysed 2026-08-10 · skill v0.3.5

dense hybrid text-only open-weights-open-data large frontier-generalist

Technical report ↗

Design overview

OLMo 3 is AI2's fully-open dense model flow at 7B and 32B. The 32B flagship keeps the OLMo 2 stability package — RMSNorm on sub-layer outputs, QK-norm, RoPE base 500,000, z-loss, no embedding weight decay — and adds two structural changes: GQA with 8 KV heads at 32B (7B stays MHA), and a 3:1 hybrid attention pattern where three of every four layers use sliding-window attention (window 4,096) and the fourth plus the final layer use full attention. Context grows in three stages — 8K pretrain, 8K midtrain with checkpoint souping, then a 65K long-context stage using YaRN on full-attention layers only, best-fit document packing, intra-document masking, and 8-way context parallelism. Post-training is equally staged: Dolci Think SFT, DPO, then OlmoRL — a fully open GRPO-family RLVR algorithm. The report's distinctive claim is completeness of disclosure: data, checkpoints, and recipes all released.

What the report claims is novel

Atlas assessment: Directly extends the OLMo 2 record's design package (RMSNorm on outputs, QK-norm, RoPE 5e5, z-loss) with GQA at 32B and a 3:1 sliding-window/full-attention hybrid — the first SWA hybrid in the dense line of the atlas. The genuinely notable novelty is the openness of the RLVR pipeline (OlmoRL) and the RL-Zero decontamination methodology, which the atlas records under post-training and contributions rather than block design.

Relation to the atlas

Lineage in

Influence out

No descendants recorded yet.

Notable omissions

Closest relatives in the atlas

Spec sheet

Identity

FamilyOLMo 3[1]
OrganisationAllen Institute for AI (Olmo Team)[2]
Release date2025-12-15
Report URLhttps://arxiv.org/abs/2512.13961[3]
Open weightsyes[4]

Variants

Scale

Total parameters32B[5]
Active parameters / token32B
Layers64[6]
Hidden dimension5120[6]
Query heads40[6]
KV heads8[7]
Head dimension128

Per-variant configs

VariantTotalLayersHiddenFFNQ headsKV headsContext
7B7B324096not disclosed323265,536
32B32B645120not disclosed40865,536

Core block

Block typedense
Attention varianthybrid[8]
Attention layer patternSliding window on 3 of every 4 layers (window 4,096), full attention on the 4th; last layer full attention[8]
Depth mixingsequential-residual
ActivationSwiGLU[9]
MethodRoPE[9]
RoPE base frequency500000[6]
TypeRMSNorm[11]
Placementpost[8]
QK-normyes[12]
Attention sinksno
Softcappingno
Otherz-loss 1e-5; no weight decay on embeddings[9]

Context

Trained context length8,192[13]
Deployed context length65,536[14]
Extension methodYaRN applied to full-attention layers only; best-fit document packing; intra-document masking; 8-way context parallelism[8]

Tokenizer

AlgorithmBPE (cl100k-derived)[15]
NotesVocabulary size not stated[3]

Training

Training tokens5.5T[16]
Data compositionDolma 3 pretraining mix; midtrain stages (2×100B soup candidates); long-context stage 100B (32B); Dolci Think/Instruct/RL-Zero data suites for post-training (Table 35 / §3–6)[17]
Curriculum / stagingPretrain → midtrain (2×100B, checkpoint-souped) → long-context stage (100B at 65K)[10]
LR scheduleCosine truncated at 5.5T tokens; peak LR 6.0e-4 (32B)[18]
Batch schedule1,024 instances per batch (8,388,608 tokens)[18]
PrecisionBF16[8]
ParallelismHSDP + DP-replication + Llama 3-style context parallelism[8]
HardwareH100 80GB × 1,024 (pretraining)[19]
Disclosed compute~56 days end-to-end; $2.75M at $2/H100-hr; 1,960 TPS/GPU (32B)[20]

Post-training

SFTyes[21]
Preference optimisationDPO[22]
Reasoning trainingThink: SFT → DPO → OlmoRL (GRPO + DAPO/Dr.GRPO-style RLVR: zero-gradient filtering, active sampling, token-level loss, no KL, clip-higher, truncated importance sampling, no std-dev normalization); RL-Zero: RLVR directly from Base across 4–5 domains; Instruct: SFT + function calling + length-normalized DPO + RLVR[23]
Distillationnone

Modality

Typetext-only

Inference efficiency

KV-cache designGQA at 32B (8 KV heads) plus sliding-window layers (window 4,096) shrink KV cache vs full attention; YaRN only on full-attention layers
Quantisation shippednone disclosed[3]
Speculative decodingnone disclosed[3]
Serving optimisationsnone disclosed[3]

Evaluation

Benchmarks (report's own numbers only)

BenchmarkValueRef
MATH96.2Table 14 (OLMo 3.1 32B Think column)
AIME 202480.6Table 14 (OLMo 3.1 32B Think column)
AIME 202578.1Table 14 (OLMo 3.1 32B Think column)
OMEGA53.4Table 14 (OLMo 3.1 32B Think column)
BigBenchHard88.6Table 14 (OLMo 3.1 32B Think column)
ZebraLogic80.1Table 14 (OLMo 3.1 32B Think column)
AGI Eval English89.2Table 14 (OLMo 3.1 32B Think column)
HumanEvalPlus91.5Table 14 (OLMo 3.1 32B Think column)
MBPP+68.3Table 14 (OLMo 3.1 32B Think column)
LiveCodeBench v383.3Table 14 (OLMo 3.1 32B Think column)

Provenance references

  1. stated · Title
  2. stated · Title page byline
  3. stated · pipeline
  4. stated · Abstract / §1 (whole model flow released: checkpoints, code, Dolma 3 data mixes)
  5. stated · Abstract (flagship 32B); exact parameter counts not disclosed
  6. stated · Table 33
  7. stated · Table 33 (GQA at 32B; 7B uses 32/32 MHA)
  8. stated · §3.2
  9. stated · Table 33 / §3.2
  10. stated · §3.2 / Table 35
  11. stated · Table 33 / §3.2 (on outputs)
  12. stated · Table 33 / §3.2 (QK-Norm)
  13. stated · Table 35 (pretrain/midtrain sequence length)
  14. stated · §3.2 / Table 35 (long-context stage trains directly at 65,536)
  15. stated · §2 (same tokenizer as OLMo 2)
  16. stated · Table 35 (32B pretraining; 7B uses 5.93T)
  17. stated · Table 35 / §3
  18. stated · Table 35
  19. stated · Table 35 / §2.4
  20. stated · §2.4 / Table 35
  21. stated · §4.2 (Dolci Think SFT); §5 (Instruct SFT)
  22. stated · §4.3 / §5 (Think: DPO then OlmoRL; Instruct: length-normalized DPO)
  23. stated · §4–6