MiniMax Sparse Attention (MSA) is the attention mechanism behind MiniMax M3, described here on the report's 109B/6B-active, 41-layer MoE testbed (3 dense + 38 MoE layers, 128 routed experts with top-4, 1 shared expert, 64 Q heads / 4 KV heads, head dim 128, 200K vocab, native text+image/video training). MSA keeps standard scaled dot-product softmax attention but makes its support learned and bounded: a lightweight Index Branch (one index query head per GQA group plus a single shared index key head) scores key-value blocks by max-pooled dot products and picks the top-k=16 of 128-token blocks for each query and group — the local block is always forced in — giving a fixed 2,048-token budget per query; the Main Branch then runs exact block-sparse attention over only those blocks. Because top-k selection is non-differentiable, the indexer is trained by a KL alignment loss against the group-averaged Main Branch distribution, confined to the index projections by a stop-gradient on its input, and stabilized by a two-stage warmup (full attention first, 40B tokens at this scale) plus the forced local block; ablations showed a learnable attention sink and an index value head are unnecessary, though a first-token sink emerges naturally. Efficiency comes from co-designed kernels: exp-free TopK in the small-k regime and KV-outer sparse attention with pre-scheduled chunking and a two-phase LSE combine for tensor-core utilization, delivering 14.2× prefill / 7.6× decode speedups on H800 at 1M context and a 28.4× attention-FLOPs cut. Two training routes are validated: from-scratch sparse pretraining (MSA-PT) and near-lossless conversion of a GQA full-attention checkpoint via 400B-token sparse CPT (MSA-CPT), plus a ~140B-token long-context extension evaluated at 128K. The production MiniMax-M3 model itself is released on HuggingFace but its configuration is not described in this report.
What the report claims is novel
MSA, a minimal, scalable, and accelerated blockwise sparse attention mechanism built on GQA: a lightweight Index Branch (one index query head per GQA group + one shared index key head, max-pooled block scores) selects a top-k subset of key blocks independently for each GQA group, and the Main Branch runs exact block-sparse softmax attention over the selected blocks [Abstract, §3.1]
A stable training recipe for the non-differentiable indexer: KL alignment loss against the group-averaged Main Branch distribution, gradient detach confining the auxiliary loss to the index projections, indexer warmup (full-attention first), and a forced local block [§3.2]
Co-designed GPU kernels — exp-free TopK for the small-k regime, KV-outer sparse attention with pre-scheduled chunking and a two-phase combine (no atomics), LSE fusion, and dynamic load balancing — translating sparsity into 14.2× prefill and 7.6× decode wall-clock speedups on H800 at 1M context [§4, Abstract]
On a 109B-parameter native-multimodal MoE model under a 3T-token budget, MSA matches the GQA full-attention baseline on pretraining and agentic benchmarks while reducing per-token attention compute by 28.4× at 1M context [Abstract, §5.3, §7]
Two training routes validated: from-scratch sparse pretraining (MSA-PT) and near-lossless conversion of pretrained GQA full-attention checkpoints via sparse continued pretraining (MSA-CPT, 2.6T full-attention + 400B sparse with a 40B warmup) [§5.1, §5.2]
Open release of the inference kernel (github.com/MiniMax-AI/MSA) and of MiniMax-M3, a production-grade natively multimodal model powered by MSA (huggingface.co/MiniMaxAI/MiniMax-M3) [Abstract]
Atlas assessment: MSA is the first learned blockwise-sparse attention record in the atlas: unlike linear-attention hybrids (kimi-k3's KDA, zyphra-zaya1, ai21-jamba) it keeps exact softmax attention and instead learns a per-GQA-group block support, and unlike the atlas' full-attention MoEs (deepseek-v3) it is sub-quadratic. Its closest published relatives — DeepSeek's DSA, NSA, MoBA, InfLLM-V2 — are not in the atlas, so MSA fills a genuinely new cell (learned sparse softmax attention on GQA). The per-group independent Top-k plus block-level selection and the KL+detach+warmup training recipe are the cleanest novel elements; the kernel work is engineering but substantial and open-sourced. Two caveats: all experiments are on the 109B model (the production 428B M3 config is not disclosed in this report, so scale-up behaviour is unverified here), and long-context quality still trails full attention slightly (HELMET-128K −0.60), with the paper itself noting a residual long-context retrieval gap.
Relation to the atlas
Lineage in
Grouped-Query Attention backbone — MSA is built on GQA (group-shared index sets, shared KV heads) from GQA (Ainslie et al., 2023)
FlashAttention algorithmic skeleton — the MSA kernels 'reuse the FlashAttention algorithmic skeleton with a loop ordering tuned to the GQA-native, block-granular access pattern' from FlashAttention (not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
The production MiniMax-M3 configuration is not disclosed: no 428B figure, no layer count, no context length, no benchmark scores for M3 appear anywhere in the report — it is referenced only by a HuggingFace link in the abstract (the atlas slug's '428b' cannot be sourced from this report)
All experiments and evaluations are on the 109B MSA testbed, not on the released production model
Expert inner dimension and MoE router/load-balancing details for the 109B model are not stated (only expert count 128, top-4, 1 shared expert)
Index head dimension d_idx is used in the equations but its value is never given
Normalization type, activation function, optimizer, LR schedule, batch size, precision, parallelism, training hardware, and compute are all undisclosed
Tokenizer algorithm not described (only the 200K vocab size)
Trained context length of the main pretraining run not stated
No post-training pipeline described (SFT/RL mentioned only as future work)
Long-context quality still lags full attention (HELMET-128K Overall −0.60 vs baseline); the paper acknowledges a residual long-context retrieval gap
long-context continued training: ~140B tokens of long-context training on top of the sparse-CPT checkpoint, evaluated on HELMET-128K and RULER-128K with the fixed 2,048-token attention budget preserved[22]
native multimodal mixture of text and image/video data; detailed composition not disclosed[24]
Curriculum / staging
MSA-PT route: 40B-token indexer-warmup (full attention + KL) then sparse training for the remainder of the 3T budget; MSA-CPT route: starts from a GQA full-attention checkpoint trained on 2.6T tokens, replaces dense attention with MSA, and continues pretraining for 400B tokens (first 40B warmup, then sparse); plus a ~140B-token long-context extension stage[25]
GQA KV cache proportional to 4 KV heads; MSA sparsifies attention compute (top-16 of 128-token blocks = 2,048-token budget per query) rather than the cache itself — the report does not discuss cache-side savings
MSA co-designed GPU kernels (open-sourced at github.com/MiniMax-AI/MSA): exp-free TopK selection (per-thread register top-k with k-element min-heaps and shuffle merge; Bk=128, k=16), KV-outer sparse attention with query gathering and query concatenation for tensor-core utilization, pre-scheduled tile chunking for hot KV blocks, two-phase combine with per-partial LSE buffers (no atomics), LSE fusion into the forward pass, and dynamic load balancing via a persistent grid with atomic work claiming; measured 14.2× prefill and 7.6× decode wall-clock speedups on H800 at 1M context[32]
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
MMLU (MSA-PT)
67.2
Table 2
MMLU-Pro (MSA-PT)
38.8
Table 2
BBH (MSA-PT)
66.6
Table 2
GPQA Hard (MSA-PT)
26.3
Table 2
GSM8K (MSA-PT)
77.7
Table 2
HumanEval (MSA-PT)
64.0
Table 2
EvalPlus (MSA-PT)
61.8
Table 2
BigCodeBench (MSA-PT)
44.0
Table 2
RULER-8K (MSA-PT)
84.2
Table 2
RULER-32K (MSA-PT)
77.5
Table 2
MMMU (MSA-PT)
45.9
Table 2
ChartQA (MSA-PT)
75.4
Table 2
VideoMME (MSA-PT)
45.48
Table 2
MLVU (MSA-PT)
46.94
Table 2
HELMET-128K Overall (MSA-CPT, after long-context extension)
45.93
Table 3
RULER-128K Overall (MSA-CPT, after long-context extension)
72.12
Table 3
Provenance references
stated · Abstract ('A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3'); note: the report itself is titled 'MiniMax Sparse Attention' and its experiments use a 109B MSA model, not M3
stated · Title page author affiliations
stated · pipeline
stated · Abstract ('A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3'); inference kernel open-sourced at github.com/MiniMax-AI/MSA
stated · §5.1 ('approximately 109B total parameters')
stated · §5.1 ('6B activated parameters per token')
stated · §5.1 ('41-layer MoE backbone')
stated · §5.1 ('hidden size d_model=3072')
stated · §5.1 ('64 query heads')
stated · §5.1 ('4 KV heads')
stated · §5.1 ('head dimension 128')
stated · §5.1 ('a 200K-token vocabulary'); also Appendix B.1 (pilot)
stated · §2.3, §3.1 ('MSA, a GQA-based sparse attention mechanism'; 64 query heads, 4 KV heads, head dim 128)
stated · §5.1 ('Each attention module uses MSA')
stated · §5.1 ('128 routed experts')
stated · §5.1 ('top-4 routed expert selection')
stated · §5.1 ('1 shared expert')
stated · §5.1 ('RoPE dimension 64')
stated · §5.1 ('RoPE dimension 64' with head dimension 128 — RoPE applied to half of the head dims)
stated · Appendix B.5 ('we do not include the learnable attention sink in the final recipe'); Appendix A (an attention sink on the first token emerges naturally in trained MSA models — emergent, not designed)
stated · Abstract ('reducing per-token attention compute by 28.4× at 1M context'; '14.2× prefill and 7.6× decoding wall-clock speedups at 1M context length')
stated · §5.3
stated · §5.1 ('total budget of 3T tokens')
stated · §5 ('a native multimodal model trained on a mixture of text and image/video data')
stated · §5.1, §5.3
stated · §7 (RL post-training is mentioned only as future work: 'extending the same selector-only design to settings beyond pretraining, including reinforcement-learning post-training')
stated · §7 (mentioned only as future work)
stated · §5 ('native multimodal model trained on a mixture of text and image/video data'; evaluation includes image (AI2D, ChartQA, MMMU, OCRBench v2, CharXiv, VisualWebBench, CVBench) and video (EgoSchema, LongVideoBench, MLVU, MMVU, VideoMME, TemporalBench) benchmarks)
stated · Abstract, §5 ('native multimodal training', 'natively multimodal model'); encoder architecture not described in this report
stated · §4.2 (kernels 'also support fp8' but no shipped quantization format is described)