Phi-4

Microsoft Research · 2024-12-12 · analysed 2026-08-10 · skill v0.3.5

dense full-attention text-only undisclosed medium reasoning-specialised

Design overview

Phi-4 is a 14B dense decoder-only transformer that 'closely follows phi-3-medium' with three stated deltas: a tiktoken tokenizer with a padded 100,352-token vocabulary, full attention over the 4K default context instead of phi-3-medium's 2K sliding window, and a midtraining stage that extends the context to 16K. The report discloses almost nothing about the block internals — no layer count, hidden dimension, head counts, normalization, or activation — so the record is deliberately sparse on scale beyond the headline 14B. The report's contribution is the recipe: ~10T tokens with a synthetic-first mixture (~400B unweighted synthetic tokens across 50 dataset types), curated organic data, then one round of SFT and two rounds of DPO, the first built on the novel Pivotal Token Search technique. The framing is explicitly about matching much larger models (Llama-3.1-405B) on reasoning benchmarks with a 14B parameter budget.

What the report claims is novel

Midtraining: an intermediate training stage between pretraining and post-training [§1 / §2.2 / §3]
Pivotal Token Search (PTS): a technique to create DPO pairs from pivotal tokens [§2.4 / §4.3]
Synthetic-first data: 50 broad types of synthetic datasets with novel generation methodologies [§1 / §2.2]
Going 'beyond distillation': matches or exceeds much larger models (e.g. Llama-3.1-405B) on reasoning benchmarks, surpassing its teacher GPT-4o on STEM QA [Abstract / §1]

Atlas assessment: Within the atlas, phi-4 is a direct refinement of the existing microsoft-phi-3 line: same 'small model, dense, data-centric' intent, with the 2K sliding window dropped for full attention and a tiktoken vocabulary. The architectural novelty is modest; the claimed novelty is in the data and post-training recipes (midtraining, PTS), which the schema records under training/post_training rather than block design.

Relation to the atlas

Lineage in

full attention, tiktoken tokenizer, midtraining from Phi-3-medium (report: 'closely follows phi-3-medium, except…')

Influence out

No descendants recorded yet.

Notable omissions

No layer count, hidden dimension, FFN dimension, or attention head counts disclosed
No normalization, activation, or positional-encoding disclosure
No optimizer name or numeric precision for pretraining
Silent on weight release and license
Benchmarks shown only as Table 1 comparisons without harness details

Closest relatives in the atlas

Phi-3 — 6 shared comparable fields
OLMo 2 — 5 shared comparable fields
GPT-2 — 5 shared comparable fields

Spec sheet

Identity

Family	Phi-4^[1]
Organisation	Microsoft Research^[2]
Release date	2024-12-12
Report URL	https://arxiv.org/abs/2412.08905^[3]

Scale

Total parameters	14B^[4]
Active parameters / token	14B
Vocabulary size	100352^[5]

Core block

Block type	dense
Attention variant	MHA
Attention layer pattern	uniform
Depth mixing	sequential-residual

Context

Trained context length	4,096^[6]
Deployed context length	16,384^[7]
Extension method	extended to 16K during midtraining^[6]

Tokenizer

Algorithm	tiktoken-based BPE^[6]
Notes	Padded vocabulary of 100,352 tokens including unused tokens (§3)^[6]

Training

Training tokens	10T^[8]
Data composition	Synthetic-first: ~50 broad types of synthetic datasets (~400B unweighted tokens) for pretraining/midtraining; curated and filtered organic data (arXiv, PubMed Central, GitHub, licensed books, filtered web); multilingual coverage (DE/ES/FR/PT/IT/HI/JA) (§2)^[9]
Curriculum / staging	Pretraining → midtraining (16K context stage)^[6]
LR schedule	Linear warm-up and decay; peak learning rate 0.0003; constant weight decay 0.1^[6]
Batch schedule	Global batch size 5760^[6]

Post-training

SFT	yes^[10]
Preference optimisation	DPO^[11]
Reasoning training	none disclosed
Distillation	none disclosed

Modality

Type	text-only

Inference efficiency

KV-cache design	Full attention with no grouping disclosed: KV cache per token scales with all attention heads; 16K deployed context multiplies cache
Quantisation shipped	none disclosed^[3]
Speculative decoding	none disclosed^[3]
Serving optimisations	none disclosed^[3]

Evaluation

Benchmarks (report's own numbers only)

Benchmark	Value	Ref
MMLU	84.8	Table 1
GPQA	56.1	Table 1
MATH	80.4	Table 1
HumanEval	82.6	Table 1
MGSM	80.6	Table 1
SimpleQA	3.0	Table 1
DROP	75.5	Table 1
MMLU-Pro	70.4	Table 1
HumanEval+	82.8	Table 1
ArenaHard	75.4	Table 1
IFEval	63.0	Table 1
PhiBench	56.2	Table 1

Provenance references

stated · Title
stated · Title page byline
stated · pipeline
stated · Abstract / §1 / Table 1
stated · §3 (padded vocabulary size of 100,352 including unused tokens)
stated · §3
stated · §3 (extended to a 16K context length during midtraining)
stated · §3 (pretrained for approximately 10T tokens)
stated · §2
stated · §4.1 (one round of SFT)
stated · §4.2 (two rounds of DPO: first with pivotal-token-search pairs, second with full-length preference pairs)