Llama 3

Llama Team, AI @ Meta · 2024-07-23 · analysed 2026-08-10 · skill v0.3.5

dense full-attention text-only open-weights frontier frontier-generalist

Technical report ↗

Design overview

Llama 3 is a dense decoder-only transformer: GQA with 8 KV heads, RoPE at base 500,000, SwiGLU, a 128K-token vocabulary built from tiktoken plus 28K added tokens, and staged continued pretraining from 8K to 128K context. The report leads with data scale (15.6T tokens, ~50/25/17/8 knowledge/reasoning/code/multilingual mix) and a six-round post-training pipeline of SFT, rejection sampling, and DPO with formatting-token masking. The 405B flagship is the largest dense model in the atlas and the reference variant; the 8B/70B siblings share the block with shallower, narrower stacks. Architecturally it is the same family as the existing meta-llama-3.1 record — the report's results in fact describe the Llama 3.1 models — so this record's value is as the base-herd source: the pretraining recipe, staged context extension, and the explicit release of all three sizes under the Llama 3 Community License.

What the report claims is novel

Atlas assessment: Structurally refines the Llama 3.1 record's architecture (same family, this record covers the base April-2024 releases; the report's results refer to the 3.1 models). GQA + RoPE-500K + 128K context were already in the atlas via Llama 3.1; nothing structurally novel within atlas scope, but this is the canonical source for the base herd.

Relation to the atlas

Lineage in

Influence out

Notable omissions

Closest relatives in the atlas

Spec sheet

Identity

FamilyLlama 3[1]
OrganisationLlama Team, AI @ Meta[2]
Release date2024-07-23[3]
Report URLhttps://arxiv.org/abs/2407.21783[4]
LicenseLlama 3 Community License (updated version)[5]
Open weightsyes[6]

Variants

Scale

Total parameters405B[7]
Active parameters / token405B
Layers126[8]
Hidden dimension16384[8]
FFN inner dimension53248[8]
FFN ratio3.25
Query heads128[8]
KV heads8[9]
Head dimension128
Vocabulary size128000[10]

Per-variant configs

VariantTotalLayersHiddenFFNQ headsKV headsContext
8B8B32409614336328131,072
70B70B80819228672648131,072
405B405B12616384532481288131,072

Core block

Block typedense
Attention variantGQA[11]
Attention layer patternuniform
Depth mixingsequential-residual
ActivationSwiGLU[12]
MethodRoPE[12]
RoPE base frequency500000[12]
QK-normno
Attention sinksno
Softcappingno
Otherdocument-separation attention mask (§3.2)[11]

Context

Trained context length131,072[15]
Deployed context length131,072[16]
Extension methodstaged continued pretraining 8K→128K, no interpolation[13]

Tokenizer

Algorithmtiktoken-based BPE[11]
NotesVocabulary combines 100K tokens from the tiktoken tokenizer with 28K additional tokens for non-English support; improved compression vs Llama 2 tokenizer (§3.2)[11]

Training

Training tokens15.6T[17]
Data composition~50% general knowledge, ~25% math/reasoning, ~17% code, ~8% multilingual (§3.1.2); 4.7T tokens of synthetic data for post-training (§4)[18]
Curriculum / stagingThree-stage: general pretraining; annealing (40M tokens); continued pretraining 8K→128K (~800B tokens) (§3.4)[19]
OptimizerAdamW[8]
LR scheduleCosine schedule with peak LR 8e-5 (8B), 1.5e-4 (70B), 3e-4 (405B), decaying to 10% of peak[8]
Batch scheduleFixed batch per compute scale, 4M→8M→16M tokens across stages[20]
PrecisionBF16 with FP32 gradient accumulation[21]
Parallelism4D parallelism: TP/CP/PP/DP with FSDP[21]
HardwareUp to 16,384 H100 GPUs (Grand Teton cluster)[21]
Disclosed compute3.8e25 FLOPs (405B pretraining)[8]

Post-training

SFTyes[22]
Preference optimisationDPO[23]
Reasoning trainingRejection-sampled SFT data for reasoning; no RLVR disclosed
Distillationnone disclosed

Modality

Typetext-only

Inference efficiency

KV-cache designGQA: KV cache ∝ 8 KV heads (vs 128 query heads)
Quantisation shippedFP8 inference for 405B: FFN matmuls with row-wise dynamic scaling, max 1200, first/last layers unquantized (§6.2)[24]
Speculative decodingnone disclosed[4]
Serving optimisationsBF16 pipeline parallelism across 16 GPUs on two machines with micro-batching (§6.1)[25]

Evaluation

Benchmarks (report's own numbers only)

BenchmarkValueRef
MMLU (5-shot)87.3Table 2
MMLU-Pro (5-shot, CoT)73.3Table 2
IFEval88.6Table 2
HumanEval89.0Table 2
MBPP EvalPlus88.6Table 2
GSM8K (8-shot, CoT)96.8Table 2
MATH (0-shot CoT)73.8Table 2
ARC-C96.9Table 2
GPQA51.1Table 2
MGSM91.6Table 2

Provenance references

  1. stated · Title / §1
  2. stated · Title page byline
  3. stated · Title block date line; note: arXiv header shows v3 dated 2024-11-23; the atlas's meta-llama-3.1 record uses arXiv v1 2024-07-31 as inferred
  4. stated · pipeline
  5. stated · §1.2 / p.2 release paragraph
  6. stated · §1.2: 'We are publicly releasing all three Llama 3 models…'; release paragraph
  7. stated · Abstract / §1 / Table 3
  8. stated · Table 3
  9. stated · Table 3 / §3.2 (GQA with 8 KV heads)
  10. stated · §3.2 / Table 3 (report prints 128,000; tokenizer combines 100K tiktoken tokens + 28K additional)
  11. stated · §3.2
  12. stated · §3.2 / Table 3
  13. stated · §3.4.2
  14. stated · Table 4
  15. stated · §3.4.2 / Table 4 (staged continued pretraining to 128K)
  16. stated · §3.4.2 (128K context reported for the models)
  17. stated · §1 / §3.4
  18. stated · §3.1.2 / §4
  19. stated · §3.4
  20. stated · §3.4 / Table 4
  21. stated · §3.3
  22. stated · §4 (SFT data; six rounds of post-training)
  23. stated · §4 (DPO with β=0.1, formatting-token masking, NLL term 0.2)
  24. stated · §6.2
  25. stated · §6.1