sparse-MoE

taxonomy axis: compute_structure

Decision rule: MoE layers dominate (≥80% of transformer layers are routed FFNs).

Anchor: Mixtral 8x7B

Models in this category (27)

Model	Org	Date	Block	Attention	Context	Total	Active
Laguna	poolside	not disclosed	sparse-MoE	hybrid	131,072	33.4B	3B
INTELLECT-3	Prime Intellect (Prime Intellect, Inc.)	not disclosed	sparse-MoE	not disclosed	65,536	106B	12B
Mixtral	Mistral AI	2024-01-08	sparse-MoE	GQA	32,768	47B	13B
DeepSeek-V3	DeepSeek-AI	2024-12-27	sparse-MoE	MLA	4,096	671B	37B
DeepSeek-R1	DeepSeek-AI	2025-01-22	sparse-MoE			671B	37B
Qwen3	Qwen Team	2025-05-15	sparse-MoE	GQA	32,768	235B	22B
Kimi K2	Kimi Team (Moonshot AI)	2025-07-28	sparse-MoE	MLA	4,096	1.043T	32.6B
gpt-oss	OpenAI	2025-08-05	sparse-MoE	hybrid	not disclosed	116.83B	5.13B
GLM-4.5 (ARC series)	Zhipu AI & Tsinghua University (GLM-4.5 Team)	2025-08-08	sparse-MoE	GQA	131,072	106B	12B
GLM-4.5 (ARC series)	Zhipu AI & Tsinghua University (GLM-4.5 Team)	2025-08-08	sparse-MoE	GQA	131,072	355B	32B
Kimi Linear	Kimi Team (Moonshot AI)	2025-11-01	sparse-MoE	hybrid	4,096	48B	3B
DeepSeek-V3.2	DeepSeek-AI	2025-12-02	sparse-MoE	MLA	131,072	not disclosed	not disclosed
Nemotron 3 Nano	NVIDIA	2025-12-23	sparse-MoE	GQA	524,288	31.6B	3.2B
MiMo-V2	Xiaomi (LLM-Core)	2026-01-06	sparse-MoE	hybrid	262,144	309B	15B
LongCat-Flash	Meituan LongCat Team	2026-01-29	sparse-MoE	not disclosed	131,072	68.5B	2.9B
GLM-5	Zhipu AI & Tsinghua University (GLM-5 Team)	2026-02-17	sparse-MoE	MLA	200K	744B	40B
Trinity	Arcee AI	2026-02-19	sparse-MoE	hybrid	262,144	400B	13B
Nemotron 3 Super	NVIDIA	2026-04-03	sparse-MoE	GQA	1,048,576	120.6B	12.7B
DeepSeek-V4	DeepSeek-AI	2026-04-26	sparse-MoE	hybrid	1M	1.6T	49B
ZAYA1	Zyphra	2026-05-06	sparse-MoE	GQA	131,072	8.4B	760M
MiniMax-M2 series	MiniMax	2026-05-26	sparse-MoE	GQA	192K	229.9B	9.8B
Mellum 2	JetBrains (with Constructor University, Bremen)	2026-05-29	sparse-MoE	hybrid	131,072	12B	2.5B
Nemotron 3 Ultra (Nemotron 3 family)	NVIDIA	2026-06-09	sparse-MoE	hybrid	1,048,576	550B	55B
MiniMax M3	MiniMax (with authors from Peking University, NVIDIA, Zhejiang University, HUST, Nanjing University, Hangzhou Dianzi University)	2026-06-11	sparse-MoE	GQA	not disclosed	109B	6B
Ling 2.6	Ling Team, Inclusion AI	2026-06-13	sparse-MoE	hybrid	262,144	1T	not disclosed
Solar Open 2	Upstage (Upstage Solar Team)	2026-07-22	sparse-MoE	hybrid	1,048,576	250B	15B
Kimi K3	Kimi Team (Moonshot AI)	2026-07-27	sparse-MoE	hybrid	1M	2.78T	104.2B

What varies within this category

attention variant varies — GQA, MLA, hybrid, not disclosed. trained context varies — 1000000, 1048576, 131072, 192000, 200000, 262144, 32768, 4096, 524288, 65536, not disclosed. openness varies — open-weights, open-weights-open-data, undisclosed. scale class varies — frontier, large, medium, not disclosed.