OLMo 2 is a dense decoder-only transformer family (7B/32 layers, 13B/40 layers) that deliberately keeps the OLMo 1 block structure and changes only what training-stability experiments demanded. Each layer is x + RMSNorm(Attention(x)) then x + RMSNorm(MLP(x)): RMSNorm sits on the sublayer outputs ('reordered norm', from Liu et al. 2021), queries and keys get their own RMSNorm QK-norm before attention, z-loss (10^-5) tames logit growth, and the RoPE base is raised to 500,000. There is no KV-head sharing — plain MHA with head count equal to layer width/128 — and the context window stays at 4,096 tokens with no extension. Training is the more distinctive half: two stages, ~3.9T web-heavy tokens (OLMo 2 Mix 1124) then a 50B-token anneal on Dolmino Mix 1124 (quality-filtered web plus synthetic math like TinyGSM-MIND and TuluMath), repeated with three different data orders and averaged into a checkpoint soup; AdamW runs with epsilon 1e-8, no weight decay on embeddings, and gradient clipping 1.0. The 13B doubles the batch and anneals on 100B/300B mixes. Post-training follows Tülu 3: SFT on 939k prompts, DPO on GPT-4o-judged on-policy preference data, then multi-stage RLVR (PPO against verifiable GSM8K/MATH rewards, value function seeded from reward models). The significance is not the block — it is that every artifact, data point, and log is released, making it the first fully-open record in the atlas at this scale.
What the report claims is novel
A fully open family of 7B and 13B models (base + Instruct) with all training data, code, recipes, logs and thousands of intermediate checkpoints released; base models sit on the performance-to-compute Pareto frontier, often matching Llama 3.1 and Qwen 2.5 with fewer FLOPs [Abstract / Section 1]
A training-stability package: repeated n-gram filtering and loss masking, truncated-normal initialization (mean 0, std 0.02), RMSNorm with reordered (output) placement, QK-norm, z-loss, excluding embeddings from weight decay, and AdamW epsilon 1e-8 [Section 3]
A mid-training recipe on the Dolmino Mix 1124 with micro-annealing as a low-cost technique to evaluate individual data sources, plus checkpoint souping over multiple anneal runs with different data orders [Section 4]
OLMo 2-Instruct built on the Tülu 3 recipe (SFT, DPO, RLVR) with permissively-licensed data and a multi-stage RLVR protocol that consistently improves both reward and downstream evaluations at both scales [Section 5]
Infrastructure as a research catalyst: Beaker workload portability, torch.compile-based throughput, asynchronous bookkeeping, explicit garbage collection, and GPU health checks that improved cluster utilization and reduced failure rates [Section 6]
Atlas assessment: The block is a conservative evolution of OLMo 1 / OLMo-0424: the only structural changes are QK-norm, reordered output-side RMSNorm, z-loss, RoPE base 500K, and no embedding weight decay — all stability techniques borrowed from PaLM/Chameleon/Dehghani et al. lineage, with no new mixing or attention mechanism. Against atlas peers, the architecture is closest to Llama 3.1 (dense, RoPE, RMSNorm, SwiGLU) but keeps plain MHA at 4K context (no GQA, no 128K extension), and to Phi-3 in scale class; its genuinely distinctive contribution is the fully-open data/recipe release (open-weights-open-data, the first in the atlas at this scale) plus the micro-annealing data-curation methodology. The 2-point average gains over OLMo-0424 are attributed by the report to the mid-training mix and souping rather than to the architecture changes.
Relation to the atlas
Lineage in
RoPE base frequency 500,000 (raised from 10,000) from Llama 3 (Grattafiori et al. 2024)
QK-norm (RMSNorm on query and key projections before attention) from Dehghani et al. 2023 (Scaling ViT to 22B)
Reordered layer norm (norm on sublayer outputs instead of inputs) from Liu et al. 2021 (not in atlas)
z-loss regularization from PaLM (Chowdhery et al. 2022, not in atlas)
cl100k tokenizer (pretokenizer and vocabulary) from OpenAI GPT-3.5 / GPT-4 (not in atlas)
Post-training recipe: SFT, on-policy DPO, RLVR with value-initialized PPO from Tülu 3 (Lambert et al. 2024, not in atlas)
Influence out
allenai-olmo-3: RMSNorm on outputs, QK-norm, RoPE 5e5, z-loss stability package
Notable omissions
No exact parameter counts (variants named only as '7B' and '13B')
No vocabulary-size integer (cl100k borrowed; never quantified)
Training precision (BF16/FP8) not stated; parallelism configuration not disclosed
No context extension; deployed inference context length not stated
Input/output embedding tying not disclosed; partial-RoPE dimension split not disclosed
z-loss weight discrepancy inside the report: Table 3 prints 10^-5, Section 3.3.3 text writes 10^-4 · log^2 Z
13B FFN inner dim not printed (derived from the stated 8/3d-to-multiple-of-128 rule)
No KV-cache sizing, inference quantisation, speculative decoding, or serving details
Extraction note: ar5iv HTML conversion garbles LaTeX math and table structures; headline numbers verified against the text (Tables 4, 6, 7, 33 survive intact)
z-loss regularization (Table 3 gives weight 10^-5; Section 3.3.3 text writes the term as 10^-4 · log^2 Z — the report's two figures disagree); No bias terms anywhere in the architecture (Section 2.2); Trainer-side loss masking of input sequences containing 32+ repeated n-grams, in addition to curation-time filtering (Section 3.1); Truncated normal initialization, mean 0, std 0.02 (Sections 2.3 / 3.2)[19]
cl100k vocabulary plus the PII-masking special tokens (|||PHONE_NUMBER|||, |||EMAIL_ADDRESS|||, |||IP_ADDRESS|||) kept from earlier OLMo tokenizers for backwards compatibility with early Dolma sources; base models use the GPT2Tokenizer class with custom pre-tokenization logic; a tokenizer mismatch between the initial Instruct Preview and the base models led to retraining the Instruct models (Appendix B.3)[22]
Two-stage mix. Pretraining: OLMo 2 Mix 1124, ~3.9T tokens, >95% web (DCLM-Baseline web 3.71T, StarCoder filtered code 83.0B, peS2o papers 58.6B, arXiv 20.8B, OpenWebMath 12.2B, Algebraic Stack 11.8B, Wikipedia/Wikibooks 3.7B). Mid-training: Dolmino Mix 1124 (50B for 7B), high-quality web (DCLM FastText top-7% + FineWeb≥2, ~47% of mix) plus FLAN, StackExchange Q&A, peS2o, Wikipedia, and a math mix (TuluMath synthetic 230M, DolminoSynthMath 28.7M, TinyGSM-MIND 6.48B, MathCoder2-synthetic 3.87B, Metamath, CodeSearchNet, GSM8K train). Exact 50B/100B/300B mix percentages in Table 13.[24]
Curriculum / staging
Two-stage: pretraining on web data (cosine LR decay calibrated over 5T tokens, truncated at 4T for the 7B), then mid-training ('annealing') on Dolmino Mix 1124 with linear LR decay to zero, run multiple times with different data-order seeds and averaged (checkpoint souping: 3×50B runs for 7B; 3×100B + 1×300B for 13B)[25]
Optimizer
AdamW with epsilon lowered from 1e-5 to 1e-8; weight decay 0.1 applied multiplicatively (1 − 0.1·lr per step), excluded for embeddings; gradient clipping 1.0[26]
LR schedule
Linear warmup from 0 over 2000 steps to peak 3.0e-4 (7B; 13B peak 9.0e-4), then cosine decay calibrated to reach 10% of peak after 5T tokens (7B truncated at 4T before mid-training); mid-training stage linearly decays LR to zero[27]
Batch schedule
Batch size 1024 (7B) / 2048 (13B) sequences at sequence length 4096; mid-training uses the same batch size (Table 13 note: 13B runs 100B-token anneals to keep the same number of steps)[28]
Precision
not disclosed
Parallelism
not disclosed
Hardware
Two Ai2 clusters: Jupiter (Cirrascale, Austin TX) — 1,024 NVIDIA H100 80GB HBM3 GPUs across 128 nodes, InfiniBand 400 Gbps/card, RDMA; Augusta (Google Cloud, Council Bluffs IA) — 160 A3 Mega VMs each with 8 NVIDIA H100 (1,280 GPUs), GPUDirect-TCPXO. Bulk of 7B training on Jupiter, bulk of 13B on Augusta.[29]
Disclosed compute
OLMo 2 7B ≈ 1.8 × 10^23 FLOPs; OLMo 2 13B ≈ 4.6 × 10^23 FLOPs (Table 6, Kaplan 6×tokens×params approximation); pretraining energy ≈ 391 MWh, ≈ 154 tCO2eq, ≈ 1.1M liters water for both models (Table 19)[30]
RLVR: Reinforcement Learning with Verifiable Rewards — PPO (Schulman et al. 2017) against rule-verifiable rewards (GSM8K and MATH training sets plus constraint prompts from Lambert et al. 2024); value function initialized from reward models trained on the on-policy preference data; multi-stage RLVR for the 13B (GSM8K pass, then MATH pass); RLVR hyperparameters in Table 18 (β sweep 0.03–0.1, 100k–200k episodes, max 2,048 response tokens)[33]
stated · Title page byline ('OLMo Team'; '1] Allen Institute for AI')
stated · pipeline
stated · Section 2 ('we release all training code, data, and recipes openly under the Apache 2.0 license wherever possible')
stated · Abstract ('We release all OLMo 2 artifacts openly—models at 7B and 13B scales, both pretrained and post-trained, including their full training data, training code and recipes')
stated · Abstract ('models at 7B and 13B scales'); Section 2.2 ('deliver 7B and 13B parameter variants')
stated · Table 4 (OLMo 2 7B: Layers 32)
stated · Table 4 (Hidden Size (d_model) 4096)
stated · Section 2.2 ('set the corresponding hidden size to approximately 8/3 d, but increased to the closest multiple of 128 (11,008 for our 7B model)')
stated · Table 4 (Attention Heads 32)
stated · Abstract ('OLMo 2 includes dense autoregressive models')
stated · Table 3 (RoPE θ: 5·10^5); Section 2.2 ('We increase the RoPE θ to 500,000 from 10,000')
stated · Section 2.2 / 3.3.1 ('We use the RMSNorm variant of LayerNorm')
stated · Section 2.2 Eqs. 1-2 / 3.3.2 ('We normalize the outputs to the attention and feedforward (MLP) layers within each transformer block, instead of the inputs')
stated · Section 2.2 / 3.3.2 ('we normalize the key and query projections with RMSNorm before calculating attention')