Llama 3.1 is a deliberately conservative dense Transformer: the report describes a standard architecture (Vaswani et al. 2017) that 'does not deviate significantly' from Llama 2, with four enumerated changes — grouped-query attention with 8 KV heads (shrinking KV caches during decoding), a document-level attention mask that blocks cross-document mixing within a sequence, a 128K-token vocabulary (100K tiktoken tokens plus 28K added for non-English), and a RoPE base frequency raised to 500,000 for longer contexts. The 405B reference variant stacks 126 layers of hidden size 16,384 with 128 query heads, an FFN of 53,248, and SwiGLU activations; head dimension is 128. Context grows from 8K to 128K tokens not by interpolation but by staged continued pre-training, followed by an annealing phase with checkpoint averaging. The pre-training recipe is plain but extreme in scale: AdamW, cosine schedule, batch size doubling from 4M to 16M tokens, 15.6T tokens and 3.8e25 FLOPs, run with 4D parallelism (TP/CP/PP/DP) on up to 16K H100s with a tuned interleaved pipeline schedule. Post-training runs six iterative rounds of SFT and DPO, leaning heavily on rejection sampling, synthetic data, and reward-model filtering, with dedicated capability pushes for code, multilinguality, math, long context, tool use, and factuality. The report also sketches compositional image/video/speech extensions via cross-attention adapters, but these are explicitly unreleased; the shipped herd is text-only.
What the report claims is novel
Data as a primary lever: more careful pre-processing and curation pipelines for pre-training data and rigorous quality assurance/filtering for post-training data, yielding a ~15T-token corpus vs 1.8T for Llama 2. [§1 / §3.1]
Scale: a 405B dense flagship pre-trained on 15.6T tokens with 3.8e25 FLOPs (~50x the largest Llama 2), with smaller models over-trained past compute-optimality to be stronger at the same inference budget. [§1]
Managing complexity: a deliberately standard dense Transformer (rather than MoE) and a simple, scalable post-training recipe of supervised finetuning, rejection sampling, and DPO rather than heavier RL algorithms. [§1 / §4]
128K-token context support achieved by staged continued pre-training with an increased RoPE base (500,000) and a document-separation attention mask important for long-sequence training. [§3.2 / §3.4.2]
A compositional approach to multimodal extension (image, video, speech) via cross-attention adapters over the frozen LLM, competitive with state of the art without joint multimodal pre-training (experimental, not released). [Abstract / §7]
Open release of pre-trained and post-trained 405B models plus the Llama Guard 3 system-level safety suite (Prompt Guard, Code Shield), intended to spur research. [§1 / §5.4.7]
Atlas assessment: Architecturally Llama 3.1 is deliberately incremental: the report itself frames the block as a standard dense Transformer barely diverging from Llama 2, with GQA, a raised RoPE base, a tiktoken-derived vocabulary, and a document mask as the only enumerated changes — no new attention, normalization, or stability mechanisms relative to peers. Its genuine novelty lies outside the block: an order-of-magnitude jump in data and compute (15.6T tokens, 3.8e25 FLOPs), context extension by staged continued pre-training rather than post-hoc interpolation, and an iterated SFT + rejection-sampling + DPO pipeline that scaled to a 405B open release with a system-level safety suite. The multimodal adapters follow Flamingo-style cross-attention (cited in-report) and are explicitly unreleased, so they do not count as a shipped contribution.
Relation to the atlas
Lineage in
standard dense Transformer (self-attention + FFN blocks) from Transformer (Vaswani et al. 2017)
overall architecture continuity with the Llama line from Llama / Llama 2
grouped-query attention with 8 KV heads from GQA (Ainslie et al. 2023)
RoPE rotary position embeddings (base raised to 500,000) from Llama lineage (RoPE)
tiktoken tokenizer (100K tokens) extended with 28K tokens from OpenAI tiktoken
Influence out
No descendants recorded yet.
Notable omissions
Normalization type and placement never named (no RMSNorm/pre-norm disclosure).
Embedding tying not disclosed.
RoPE dimensional scope (full vs partial) not stated.
QK-normalisation and logit softcapping never mentioned.
Training corpus not released; only coarse mix percentages (roughly 50/25/17/8) disclosed.
No speculative decoding or multi-token prediction disclosed.
Per-stage token budgets of the six long-context stages not itemised (only ~800B total for the stage).
Multimodal models described in detail but explicitly not released.
Vocabulary combines 100K tokens from the tiktoken tokenizer with 28K additional tokens for better non-English support; improves English compression from 3.17 to 3.94 characters per token vs the Llama 2 tokenizer. Vocab size (128,000) lives in scale.vocab_size.[10]
Final mix roughly 50% general knowledge, 25% mathematical/reasoning, 17% code, 8% multilingual tokens; heavy curation of web data (PII/safety filtering, URL/doc MinHash and line-level dedup, heuristic and model-based quality filtering, code/math pipelines, 176-language LID); knowledge cutoff end of 2023.[15]
Curriculum / staging
Three stages: (1) initial pre-training at 8K context, (2) long-context pre-training to 128K in six increments (~800B tokens), (3) annealing on the final 40M tokens with LR linearly annealed to 0 and checkpoint averaging.[16]
Peak LR 8e-5 (405B; 3e-4 for 8B and 1.5e-4 for 70B per Table 3), linear warmup of 8,000 steps, cosine decay to 8e-7 over 1,200,000 steps.[18]
Batch schedule
Starts at 4M tokens (seq len 4,096); doubled to 8M tokens at seq len 8,192 after 252M tokens; doubled again to 16M after 2.87T tokens.[17]
Precision
BF16 (38-43% BF16 MFU); FP32 gradient accumulation and FP32 reduce-scatter for numerical stability[19]
Parallelism
4D parallelism combining tensor, pipeline, context, and data parallelism (FSDP), ordered [TP, CP, PP, DP]; all-gather-based CP; interleaved PP schedule with tunable micro-batch count N; network-aware configuration.[19]
Hardware
Up to 16K H100 GPUs (700W TDP, 80GB HBM3) on Meta Grand Teton servers (8 GPUs/server, NVLink); RoCE fabric (Arista 7800, Minipack2) for 405B, Nvidia Quantum2 InfiniBand for smaller models; 400 Gbps interconnects; MAST scheduler; Tectonic storage fabric.[20]
Disclosed compute
3.8e25 FLOPs (405B pre-training), almost 50x the largest Llama 2 model[4]
SFT on step-wise reasoning traces filtered by answer correctness, self-verification, and outcome/stepwise reward models (MCTS used for challenging prompts); interleaved text+Python-code reasoning with execution feedback; no RLVR or test-time compute disclosed.[23]
GQA with 8 KV heads: KV-cache size and decoding memory reduced vs MHA; all-gather CP overhead small because KV tensors are far smaller than Q under GQA[25]
Quantisation shipped
FP8 inference for 405B (FFN-layer matmuls only, row-wise dynamic scaling, scaling factors upper-bounded at 1200, first/last layers unquantized; kernels released); Llama Guard 3 ships an int8-quantized variant (size reduced >40%)[26]
Speculative decoding
none disclosed
Serving optimisations
405B inference: BF16 pipeline parallelism across 16 GPUs on two machines (TP within node over NVLink, PP across nodes) with micro-batching for better throughput-latency trade-off; FP8 inference raises pre-fill throughput up to 50% vs two-machine BF16[27]
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
MMLU (5-shot)
87.3
Table 2
MMLU-Pro (5-shot, CoT)
73.3
Table 2
IFEval
88.6
Table 2
HumanEval (0-shot)
89.0
Table 2
MBPP EvalPlus (0-shot)
88.6
Table 2
GSM8K (8-shot, CoT)
96.8
Table 2
MATH (0-shot, CoT)
73.8
Table 2
ARC Challenge (0-shot)
96.9
Table 2
GPQA (0-shot, CoT)
51.1
Table 2
BFCL
88.5
Table 2
Nexus
58.7
Table 2
ZeroSCROLLS/QuALITY
95.2
Table 2
InfiniteBench/En.MC
83.4
Table 2
NIH/Multi-needle
98.1
Table 2
MGSM (0-shot, CoT)
91.6
Table 2
Provenance references
stated · §1 (report title is 'The Llama 3 Herd of Models'; the released family is named Llama 3.1)