dense
taxonomy axis: compute_structure
Decision rule: Every transformer layer's FFN is fully active per token; no routing anywhere.
Anchor: Llama 3 (report describes no MoE; absence rule → inferred)
Models in this category (16)
| Model | Org | Date | Block | Attention | Context | Total | Active |
|---|---|---|---|---|---|---|---|
| Antares | Cisco Foundation AI | not disclosed | dense | GQA | 131,072 | 1B | 1B |
| Nanbeige4.2 | Nanbeige LLM Lab, Boss Zhipin | not disclosed | dense | not disclosed | 262,144 | 3B | 3B |
| GPT-2 | OpenAI | not disclosed | dense | 1,024 | 1.542B | 1.542B | |
| Phi-3 | Microsoft | 2024-04-22 | dense | MHA | 4,096 | 3.8B | 3.8B |
| Llama 3 | Llama Team, AI @ Meta | 2024-07-23 | dense | GQA | 131,072 | 405B | 405B |
| Llama 3.1 | Meta (Llama Team, AI @ Meta) | 2024-07-31 | dense | GQA | 131,072 | 405B | 405B |
| Phi-4 | Microsoft Research | 2024-12-12 | dense | MHA | 4,096 | 14B | 14B |
| OLMo 2 | OLMo Team, Allen Institute for AI (Ai2) | 2024-12-31 | dense | MHA | 4,096 | 7B | 7B |
| xLSTM-7B | NXAI (NX-AI) | 2025-03-17 | dense | linear/state-space | 8,192 | 6.87B | 6.87B |
| Gemma 3 | Google DeepMind (Gemma Team) | 2025-03-25 | dense | sliding-window | 131,072 | 27B | 27B |
| Qwen3 | Qwen Team | 2025-05-15 | dense | GQA | 32,768 | 32B | 32B |
| OLMo 3 | Allen Institute for AI (Olmo Team) | 2025-12-15 | dense | hybrid | 8,192 | 32B | 32B |
| Nanbeige4.1 | Nanbeige LLM Lab, Boss Zhipin | 2026-02-13 | dense | not disclosed | 262,144 | 3B | 3B |
| Tiny Aya | Cohere Labs / Cohere | 2026-03-12 | dense | hybrid | 8,192 | 3.35B | 3.35B |
| VibeThinker | 2026-06-15 | dense | not disclosed | 65,536 | 3B | 3B | |
| Gemma 4 | Google DeepMind (Gemma Team) | 2026-07-24 | dense | hybrid | not disclosed | 31.25B | 31.25B |
What varies within this category
attention variant varies — GQA, MHA, hybrid, linear/state-space, not disclosed, sliding-window. trained context varies — 1024, 131072, 262144, 32768, 4096, 65536, 8192, not disclosed. openness varies — open-weights, open-weights-open-data, undisclosed. scale class varies — frontier, large, medium.