Mixtral

Mistral AI · 2024-01-08 · analysed 2026-08-08 · skill v0.3.0

Technical report ↗

Design overview

Mixtral 8x7B is a decoder-only sparse mixture-of-experts transformer in which every one of the 32 layers replaces its feedforward sub-block with an MoE layer holding 8 expert FFNs. For each token, a linear router produces logits x·Wg, keeps the top 2, and softmaxes them; the layer output is the weighted sum of the two selected experts' outputs, so the selected experts can differ per token and per layer. Each expert is a standard vanilla-transformer FFN using the SwiGLU activation, with inner dimension 14336. The rest of the block follows the Mistral 7B design the report explicitly inherits — grouped-query attention (32 query heads, 8 KV heads, head dimension 128) — with the stated exceptions that attention is fully dense over the 32768-token context (no sliding window) and the FFNs are routed. Total parameters are 47B, of which only about 13B are active per token, roughly 5x fewer than Llama 2 70B, which the model matches or beats across the report's benchmarks, especially in math, code, and multilingual tasks. The report is candid that the MoE formulation is close to GShard, differing by replacing every FFN block (GShard alternates) and using simpler gating for the second expert. Serving efficiency relies on Megablocks-style sparse matmul kernels integrated into vLLM, and expert parallelism for multi-GPU placement. A routing analysis finds experts do not specialize by domain; assignments track syntax and show temporal locality, relevant to expert-parallel load balancing.

What the report claims is novel

• First sparse mixture-of-experts network to reach state-of-the-art performance among open-source models. [§6 Conclusion / §1]
• Matches or outperforms Llama 2 70B and GPT-3.5 on most benchmarks while activating only 13B parameters per token (5x fewer active parameters than Llama 2 70B). [§1 Abstract / Table 2]
• Vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. [§1 / §3 / Table 4]
• Mixtral 8x7B – Instruct (SFT + DPO) surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B chat on human evaluation; MT-Bench 8.30, best open-weights model as of December 2023. [§4 / Table 3]
• Fully dense 32k-token context with 100% passkey retrieval regardless of sequence length and passkey position. [§1 / §3.2 / Figure 4]
• Open release under Apache 2.0 with open-source inference-stack contributions (vLLM + Megablocks kernels, TensorRT-LLM support). [§1 / Acknowledgements]

Atlas assessment: The architectural novelty is modest: the report itself frames Mixtral as Mistral 7B's architecture with every FFN replaced by a GShard-style top-2 routed MoE layer (both GShard [21] and Mistral 7B [18] are named as sources), with sliding-window attention dropped for fully dense 32k attention. Its contribution is primarily demonstrative and empirical — the first open-weights MoE to match a dense 70B-class model with roughly 5x fewer active parameters, plus an unusually candid routing analysis showing experts do not specialize by domain and that expert choice exhibits syntactic structure and temporal locality. Relative to other atlas entries it introduces no new attention or positional-encoding mechanism; its influence rests on popularising sparse-MoE cost-performance at the 7B-expert scale rather than on a novel mechanism.

Relation to the atlas

Lineage in

• Sparse MoE layer with softmax top-k gating replacing FFN sub-blocks (all layers, vs GShard's alternating pattern) from GShard [21]
• Base decoder-only transformer architecture and its modifications (report: "same architecture as Mistral 7B", "same modifications as described in [18]") from Mistral 7B [18]
• Softmax over Top-K logits of a linear layer as the expert gating function from Shazeer et al., sparsely-gated MoE [28]

Influence out

No descendants recorded yet.

Notable omissions

• No pretraining token count or detailed data mixture (only "multilingual data" with multilingual proportion upsampled relative to Mistral 7B).
• No training hyperparameters: optimizer, LR schedule, batch size, precision, hardware, or compute (FLOPs/GPU-hours) are not disclosed.
• No tokenizer details beyond vocab size 32000 (algorithm not named).
• No router load-balancing loss or mechanism disclosed (load balancing discussed only as an expert-parallelism serving concern).
• Core architectural modifications (positional encoding, normalization, placement) deferred to Mistral 7B [18] without being named in this report.
• Attention variant (GQA) not named; inferable only from the n_heads/n_kv_heads counts in Table 1.
• Instruct fine-tuning details (SFT and DPO dataset composition/size) not disclosed.
• No quantization formats or speculative-decoding support disclosed.

Closest relatives in the atlas

• Qwen3 — 6 shared comparable fields
• Qwen3 — 6 shared comparable fields
• DeepSeek-V3 — 5 shared comparable fields

Spec sheet

Identity

Variants

• Mixtral 8x7B
• Mixtral 8x7B – Instruct

Scale

Core block

Context

Tokenizer

Training

Post-training

Modality

Inference efficiency

Evaluation

Benchmarks (report's own numbers only)

Provenance references

1. stated · Title / §1
2. stated · Title page (Code: github.com/mistralai/mistral-src; Webpage: mistral.ai)
3. stated · pipeline
4. stated · §1 Abstract / §1
5. stated · §1 ("with open weights"; "We release both Mixtral 8x7B and Mixtral 8x7B – Instruct")
6. stated · §1 Abstract ("each token has access to 47B parameters")
7. stated · §1 Abstract ("only uses 13B active parameters during inference")
8. stated · Table 1 (n_layers)
9. stated · Table 1 (dim)
10. stated · Table 1 (hidden_dim; per-expert FFN inner dimension)
11. stated · Table 1 (n_heads)
12. stated · Table 1 (n_kv_heads)
13. stated · Table 1 (head_dim)
14. stated · Table 1 (vocab_size)
15. stated · §1 Abstract / §2.1 (Sparse Mixture of Experts; "each layer is composed of 8 feedforward blocks (i.e. experts)")
16. stated · §1 Abstract ("each layer is composed of 8 feedforward blocks") / §2 (fully dense context length of 32k)
17. stated · §2.1 ("we use the same SwiGLU architecture as the expert function E_i(x)")
18. stated · Table 1 (num_experts) / §2.1
19. stated · Table 1 (top_k_experts) / §2.1 (K=2)
20. stated · §2.1 (layer output = weighted sum of the two routed experts' outputs; the 8 experts exhaustively compose the layer, no always-on expert described)
21. stated · §2.1
22. stated · §2.1 ("an expert is a standard feedforward block as in a vanilla transformer architecture")
23. stated · §1 / §2 (trained at the fully dense 32k length; no extension disclosed)
24. n/a · §1 / §2
25. stated · Table 1 (context_len) / §1 Abstract ("trained with a context size of 32k tokens")
26. stated · §1 ("context window of 32k tokens") / Table 1
27. stated · §1 / §2 (trained at the full deployed 32k length; no extension disclosed)
28. stated · §1 / §3.1
29. stated · §4 ("supervised fine-tuning (SFT) on an instruction dataset")
30. stated · §4 ("Direct Preference Optimization (DPO) [25] on a paired feedback dataset")
31. n/a · §1 (decoder-only text language model)
32. stated · §1 / Acknowledgements