Phi-3-mini is a compact dense decoder-only transformer built, per the report, on a block structure similar to Llama-2: 32 layers, hidden size 3072, 32 attention heads, and a 32,064-token vocabulary shared with the Llama-2 tokenizer (BoS tokens removed, chat tokens added). The report never names its positional encoding, normalization, or activation — only dimensions and head counts — though the stated Llama-2 lineage and the LongRope extension point to RoPE. Its defining contribution is not the block but the data: a 'data-optimal regime' in which heavily filtered web data (selected for 'educational level') and LLM-generated synthetic data are calibrated to a small model's capacity, trained in two sequential phases (web-heavy, then filtered-web plus synthetic) for 3.3T tokens in bfloat16. Default context is 4K, extended to 128K by LongRope for the phi-3-mini-128K variant. Post-training is a two-stage SFT + DPO pipeline with red-teaming and RAI safety evaluation. The family scales the recipe: phi-3-small (7B) swaps in the tiktoken tokenizer, GEGLU, muP-tuned hyperparameters, GQA (4 queries per key), and a novel blocksparse attention that alternates dense and sparse layers with custom Triton/vLLM kernels; phi-3-medium (14B) reuses mini's architecture at 40 layers and hidden size 5120. Later phi-3.5 variants add a top-2-of-16 MoE (6.6B active of 42B), long-context mid-training, and a CLIP-based vision variant. The result is a 3.8B model that rivals far larger models on the report's own benchmarks and runs fully offline on a phone.
What the report claims is novel
The 'data optimal regime': calibrating data quality (heavily filtered web plus synthetic data) for a given model scale rather than chasing compute-optimal scaling, so a 3.8B model rivals Mixtral 8x7B and GPT-3.5. [Abstract / §1 / Training Methodology]
A long-context version, phi-3-mini-128K, that extends the default 4K context to 128K via LongRope. [§2 / §4]
Blocksparse attention in phi-3-small: per-head sparsity patterns over the KV cache with dense and blocksparse layers alternated, backed by custom Triton (Flash Attention based) training kernels and a vLLM paged-attention inference kernel for real deployment speed-ups. [§2]
phi-3-small's recipe: grouped-query attention (4 queries per key), GEGLU activation, and Maximal Update Parametrization (muP) to tune hyperparameters on a proxy model and transfer them, for better performance and training stability. [§2]
On-device capability: 4-bit quantized phi-3-mini (about 1.8 GB) running natively and fully offline at over 12 tokens per second on an iPhone 14 with A16 Bionic. [§2 / Figure 2]
Two-stage post-training (SFT + DPO) with responsible-AI alignment: red-teaming, automated testing, and safety evaluation across dozens of RAI harm categories, substantially reducing harmful response rates. [Post-training / §5]
Atlas assessment: The genuinely novel pieces are the 'data-optimal regime' training recipe — heavily filtered web plus synthetic data at small scale, explicitly deviating from compute-optimal scaling laws — and phi-3-small's blocksparse attention with custom kernels; nearly everything else (Llama-2-style decoder block, MHA, GQA, GEGLU, muP, LongRope extension, SFT + DPO) is adopted from prior architectures the report itself names and cites. The headline claim that a 3.8B model rivals Mixtral 8x7B and GPT-3.5 rests on the report's own evaluation pipeline, which the report discloses as unoptimised for phi-3 — a credible but self-run comparison. The phi-3.5-MoE variant is a conventional top-2-of-16 routed MoE whose SparseMixer router training is cited from prior work, adding little architectural novelty. Relative to other atlas entries, phi-3 is architecturally conservative (like Llama 3.1) but methodologically distinctive in making data quality, not compute, the primary lever — and it is the atlas taxonomy's anchor for on-device design intent.
Relation to the atlas
Lineage in
Llama-2-style decoder block structure from Llama-2
same tokenizer as Llama-2 (vocab 32064; BoS removed, chat-template tokens added) from Llama-2
LongRope context extension (4K to 128K) from LongRoPE [DZZ+24a]
data-centric training recipe (filtered web + synthetic data) for small models from 'Textbooks Are All You Need' / phi-1.5–phi-2 line [GZA+23, LBE+23, JBA+23]
grouped-query attention (4 queries per key) — phi-3-small from GQA (prior literature; not cited in report)
Maximal Update Parametrization (muP) hyperparameter transfer — phi-3-small from Tensor Programs V / muP [YHB+22]
Flash Attention based Triton training kernel and vLLM paged-attention inference kernel — phi-3-small from Flash Attention [DFE+22] and vLLM [KLZ+23]
SparseMixer sparse-backpropagation router training — phi-3.5-MoE from SparseMixer [LGC23, LDL+23]
Influence out
microsoft-phi-4: full attention, tiktoken tokenizer, midtraining
Notable omissions
No release statement: the report never explicitly says the weights are released (openness classified undisclosed) and names no license.
Base positional encoding, normalization type/placement, and activation are never named for phi-3-mini (implied only by the stated Llama-2-like block).
FFN hidden dimension not disclosed for any variant, so ffn_ratio is not computable.
Training details peers disclose are absent: optimizer, learning-rate and batch schedules, hardware, and FLOPs/GPU-hours.
Data mixture proportions not disclosed — composition is qualitative, and the two-phase split carries no token budgets.
No long-context evaluation of phi-3-mini-128K itself: RULER and RepoQA results cover only the phi-3.5 models (§4), which the report admits drop sharply at the 128K window.
The 'mixed context window approach' used for the phi-3.5 series' long-context mid-training is not described.
Multilingual coverage acknowledged as a weakness for phi-3-mini (mostly English); multilingual results exist only for phi-3-small and the phi-3.5 series.
LongRope (LongRoPE, [DZZ+24a]); the phi-3.5 series additionally uses 'a mixed context window approach' during mid-training[8]
Tokenizer
Algorithm
SentencePiece BPE (same tokenizer as Llama-2)
Notes
Same tokenizer as Llama-2: BoS tokens removed and additional tokens added for the chat template (footnote 1, §2); vocab size 32064 lives in scale.vocab_size. phi-3-small instead 'leverages the tiktoken tokenizer (for better multilingual tokenization) with a vocabulary size of 100352' with unused tokens removed (§2).[5]
Heavily filtered publicly available web data (filtered to an 'educational level' from various open internet sources) plus synthetic LLM-generated data; mixture proportions not disclosed. phi-3-small adds 'an additional 10% multilingual data' (§2). phi-3-medium and phi-3-small trained on 4.8T tokens (Abstract).[9]
Curriculum / staging
Two disjoint sequential pre-training phases: phase-1 mostly web sources for general knowledge and language understanding; phase-2 a more heavily filtered subset of the web data plus synthetic data for logical reasoning and niche skills. Framed as the 'data optimal regime' — calibrating data quality for a given scale — rather than the compute-optimal regime.[10]
MHA with 32 heads: full per-head KV cache with no sharing, so KV-cache size scales with all 32 heads
Quantisation shipped
4-bit quantization: phi-3-mini occupies approximately 1.8 GB and runs natively on an iPhone 14 (A16 Bionic), fully offline, at over 12 tokens per second[13]