xLSTM-7B

NXAI (NX-AI) · 2025-03-17 · analysed 2026-08-10 · skill v0.3.5

dense SSM text-only open-weights medium frontier-generalist

Design overview

xLSTM-7B is a 32-block, 6.87B-parameter fully recurrent LLM with no attention anywhere: every block pairs an output-gated multi-head mLSTM sequence-mix layer (8 heads, head dimension 512, embedding dimension 4096) with a SwiGLU channel-mix MLP (projection factor 2.66), both under a pre-norm RMSNorm. The mLSTM cell — inherited from Beck et al. (2024) — keeps a fixed-size matrix memory (C_t, n_t, m_t) per head: 134.2 MB total for the 8-head configuration, equivalent to a 128-token KV cache in a similar Transformer, with linear compute in sequence length and a fully parallel training mode via chunkwise kernels. The report's architectural contribution is the post-up projection block: running the mLSTM at the embedding dimension instead of an up-projected space, and dropping the channel-wise convolutions and learnable skip connections of the original xLSTM, which yields 2×–4× higher token throughput. Training stability at 7B scale comes from a four-part recipe: RMSNorm pre-norms, soft-capping of the input/forget gates at a=15, input-gate bias initialized to −10, and final-logit soft-capping at a=30 (Gemma-style). There is no positional encoding at all — order is carried implicitly by the recurrence — and no context extension: the model is trained and served at 8K, with a separate experimental LCTX variant reaching 32K via a long-context cool-down. Training ran on 128 H100s over 2.3T tokens (DCLM, then a math/coding/Q&A stage with public SFT datasets), using FSDP, batch ramp-up to 512, and EOD-token memory resets for sequence packing. Only the pre-trained base model is released.

What the report claims is novel

xLSTM-7B: the first xLSTM-based LLM scaled to modern data budgets (7B parameters, 2.3T tokens), showing the architecture matches similar-sized Transformers and Mamba models on downstream tasks [§1, §5.1]
Post-up projection block redesign: mLSTM operated in the embedding dimension (instead of a higher-dimensional up-projected space) with position-wise SwiGLU MLPs after each mLSTM layer, discarding channel-wise convolutions and learnable skip connections — 2×–4× higher token throughput than the previous xLSTM architecture with similar language modeling performance [§1, §3.1]
Stability optimizations for the mLSTM gating mechanism at scale: RMSNorm pre-norm, soft-capping of input/forget gates (a=15), negative input-gate bias initialization, and final-logit soft-capping (a=30) [§1, §3.2]
Fastest and most efficient 7B LLM in their benchmarks: about 50% faster text generation than Mamba, highest prefill throughput (about 70% higher than Codestral Mamba), lowest GPU memory footprint, and fastest time-to-first-token at batch size 1 [§1, §5.2]
Long-context cool-down training enables strong RULER performance (20% average accuracy at 131K for the LCTX variant) despite training at only 32K context during the cool-down, outperforming RWKV-5/6 and matching state-space baselines [§5.1]

Atlas assessment: The first pure-recurrent (zero softmax-attention) record in the atlas. Against atlas peers: the mLSTM cell itself is inherited unchanged from Beck et al. (2024) xLSTM; the genuine architectural contribution is the post-up projection block — mLSTM run at the embedding dimension with a separate SwiGLU MLP, and removal of convolutions/learnable skips — which is a real efficiency refinement, not a new mixing family. The stability toolkit (RMSNorm pre-norm, gate soft-capping, negative gate-bias init, logit soft-capping a=30) adapts Transformer/Mamba practice (Gemma-style softcapping explicitly cited). Notably the scalar sLSTM branch of the original xLSTM is dropped entirely and never mentioned. The reported efficiency gains (≈50% faster generation than Mamba, ≈70% higher prefill throughput than Codestral Mamba) are internal single-GPU benchmarks with torch.compile/CUDA-graph setups and are not independently verified here. Downstream performance is mid-range among 7B models (Table 1), which the authors attribute to dataset quality.

Relation to the atlas

Lineage in

mLSTM cell with matrix memory (recurrent + parallel modes) from xLSTM (Beck et al., 2024)
SwiGLU gated MLP as channel-mix layer (projection factor 2.66 'matching common Transformers') from Transformer LLM practice
RMSNorm pre-norm at block entry (replacing LayerNorm; practice from Transformer and Mamba training) from Transformer / Mamba training practice
Final-layer logit soft-capping with a=30 (a·tanh(x/a)) from Gemma (Gemma Team, 2024b)
Chunkwise-parallel training kernels for the mLSTM cell from Anonymous (2025) mLSTM kernels (not in atlas)
GPT-NeoX-20B byte-level BPE tokenizer (vocab 50,257) from GPT-NeoX-20B

Influence out

No descendants recorded yet.

Notable omissions

sLSTM — the scalar-LSTM branch of the original xLSTM architecture (Beck et al., 2024) — is entirely absent; xLSTM-7B relies fully on mLSTM cells and the report never mentions sLSTM
No positional encoding of any kind is discussed (order carried implicitly by recurrence)
SwiGLU inner (FFN) dimension not printed; only the 2.66 projection factor is given
Training precision (BF16/FP32/FP8) not disclosed
No total compute (FLOPs/GPU-hours) or MFU disclosed
No license named for the open-source release
No quantisation formats, speculative decoding, or serving-system optimisations disclosed (speed benchmarks use torch.compile + CUDA graphs)
No chat/instruction-tuned variant released — only the pre-trained base model
Long-context capability is shown only for the experimental LCTX variant (Fig. 3); no long-context model is released
RULER results reported only as figures, not tables of numbers

Closest relatives in the atlas

Antares — 6 shared comparable fields
Tiny Aya — 5 shared comparable fields
OLMo 2 — 5 shared comparable fields

Spec sheet

Identity

Family	xLSTM-7B^[1]
Organisation	NXAI (NX-AI)^[2]
Release date	2025-03-17
Report URL	https://arxiv.org/abs/2503.13427^[3]
License	not disclosed
Open weights	yes^[4]

Scale

Total parameters	6.87B^[5]
Active parameters / token	6.87B
Layers	32^[6]
Hidden dimension	4096^[7]
FFN inner dimension	not disclosed
FFN ratio	not disclosed
Query heads	8^[8]
KV heads	n/a^[9]
Head dimension	512
Vocabulary size	50257^[10]
Embedding tying	no^[11]

Core block

Block type	dense
Attention variant	linear/state-space^[12]
Attention layer pattern	uniform
Depth mixing	sequential-residual
Activation	SwiGLU^[13]

Expert count	n/a^[14]
Experts per token	n/a^[14]
Shared experts	n/a^[14]
Routing	n/a^[14]
Load balancing	n/a^[14]
Expert granularity	n/a^[14]

Method	NoPE
RoPE base frequency	n/a^[16]
Partial RoPE	n/a^[17]

Type	RMSNorm^[18]
Placement	pre^[19]
QK-norm	not disclosed

Attention sinks	not disclosed
Softcapping	30^[20]

Context

Trained context length	8,192^[21]
Deployed context length	8,192
Extension method	none

Tokenizer

Algorithm	BPE (GPT-NeoX-20B tokenizer)^[22]
Notes	GPT-NeoX-20B BPE tokenizer, vocabulary 50,257; input (embedding) and output (logits) weights not tied; EOD token used for sequence packing with mLSTM memory-state reset at document borders^[23]

Training

Training tokens	2.3T^[24]
Data composition	Two-stage recipe using only publicly available datasets. Stage 1 (500K steps): exclusively the DCLM dataset. Stage 2 (50K steps, prioritizing math/coding/Q&A): DCLM 40%, FineWeb-Edu 15%, ProofPile-2 15%, TheStack 15%, Cosmopedia 10%, and SFT Q&A datasets 5% (NuminaMath CoT, MetaMathQA, Tulu v3.1, OpenHermes 2.5, GSM8K, Smoltalk subsets magpie-ultra/longalign/self-oss-instruct)^[25]
Curriculum / staging	two-stage: 500K steps on DCLM, then 50K steps on a math/coding/Q&A-heavy mixture including small SFT Q&A datasets; batch-size ramp-up (128 → 256 → 512); sequence packing with EOD-token memory-state reset; the experimental LCTX variant replaces the final cool-down with a 32K-context stage (batch 128, long-context corpora)^[26]
Optimizer	AdamW with peak learning rate 5×10⁻⁴, β1=0.99, β2=0.95, ε=1e-8, weight decay 0.1, gradient clipping norm 0.5^[27]
LR schedule	linear warm-up over 3,000 steps, exponential decay over 540,000 steps (decay factor chosen so 0.1× peak is reached after 500,000 steps), linear cool-down over 7,000 steps to 0^[27]
Batch schedule	batch size ramp-up: 128 for the first 2,000 steps, 256 for the next 2,000 steps, full batch size 512 afterward; LCTX variant reduces batch to 128 at 32K context to keep tokens-per-batch constant^[26]
Precision	not disclosed
Parallelism	Fully Sharded Data Parallel (FSDP) with activation checkpointing^[27]
Hardware	128 NVIDIA H100 GPUs^[28]
Disclosed compute	not disclosed

Post-training

SFT	yes^[29]
Preference optimisation	none disclosed^[30]
Reasoning training	none disclosed
Distillation	none^[31]

Modality

Type	text-only
Attachment	n/a^[32]

Inference efficiency

KV-cache design	recurrent mLSTM state: constant-memory fixed-size state per head (C_t, n_t, m_t) instead of a growing KV cache — 8 heads × d_hv 512 = 134.2 MB total memory state, equivalent to a 128-token KV cache in a similar-sized Transformer (Table 3)
Quantisation shipped	none disclosed^[30]
Speculative decoding	none disclosed^[30]
Serving optimisations	none disclosed^[33]

Evaluation

Benchmarks (report's own numbers only)

Benchmark	Value	Ref
BBH (Open LLM Leaderboard v2)	0.381	Table 1
MMLU-Pro (Open LLM Leaderboard v2)	0.242	Table 1
Math (Open LLM Leaderboard v2)	0.036	Table 1
MuSR (Open LLM Leaderboard v2)	0.379	Table 1
GPQA (Open LLM Leaderboard v2)	0.28	Table 1
IFEval (Open LLM Leaderboard v2)	0.244	Table 1
Average (Open LLM Leaderboard v2)	0.26	Table 1
ARC-C (Open LLM Leaderboard v1)	0.574	Table 6
MMLU (Open LLM Leaderboard v1)	0.578	Table 6
HellaSwag (Open LLM Leaderboard v1)	0.714	Table 6
Winogrande (Open LLM Leaderboard v1)	0.738	Table 6
TruthfulQA (Open LLM Leaderboard v1)	0.419	Table 6
OpenBookQA (Open LLM Leaderboard v1)	0.448	Table 6
PiQA (Open LLM Leaderboard v1)	0.819	Table 6
Average (Open LLM Leaderboard v1)	0.613	Table 6

Provenance references

stated · Title ('xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference')
stated · Title page byline; model/code links (huggingface.co/NX-AI/xLSTM-7b, github.com/NX-AI)
stated · pipeline
stated · Abstract ('Our model weights, model code and training code are open-source'); §1 ('We release the pre-trained model xLSTM 7B on Huggingface')
stated · Table 4 (Num Params 6,865,424,896 ≈ 6.87B); Abstract ('7-billion-parameter LLM')
stated · Table 4 (Num Blocks 32); Appendix A ('32 post-up projection blocks')
stated · Table 4 (Model Dim 4096); Table 3 ('embedding dimension 4096')
stated · Table 4 (Num Heads 8; multi-head mLSTM, §2.1)
n/a · §2 (mLSTM cells keep a fixed-size recurrent matrix state (C_t, n_t, m_t) per head; there are no KV heads and no KV cache)
stated · Table 4 (Vocab Size 50257); Appendix A ('vocabulary size 50257')
stated · Appendix A ('we do not tie the weights for input layers (embedding) and output layers (logits)')
stated · §2 (mLSTM recurrent cell with matrix memory: recurrent mode for generation, fully parallelizable mode for training; constant memory, linear compute in sequence length)
stated · Fig. 8 ('the upper part is a Gated MLP (SwiGLU) as a feature/channel-mix layer'); §3.1 (projection factor 2.66)
n/a · §2 (no MoE in the architecture)
n/a · no extension mechanism (see extension.method)
n/a · report-wide absence (no RoPE anywhere in the model)
n/a · report-wide absence (no RoPE)
stated · Fig. 8 ('The Norm layers are RMS norms (Zhang & Sennrich, 2019), the Headwise Norm is a Layernorm (Ba et al., 2016)'); §3.2 ('we replace the LayerNorm by RMSNorm in our xLSTM architecture')
stated · §3.2 ('pre-norm at the block entry', 'in the pre-norm setting'); §5.3 ('two normalization layers, a pre-norm at the block entry and a head-wise norm layer after the mLSTM cell')
stated · §3.2 Eq. 13 ('we apply soft-capping with a=30 to the final layer logits, similar to Gemma Team (2024b)')
stated · §1 ('trained on 2.3T tokens from the DCLM dataset with context length 8192'); App. B ('batch size 512 and context length 8192')
stated · Appendix A ('We use the GPT-NeoX-20B tokenizer (Black et al., 2022) with vocabulary size 50257')
stated · Appendix A, Appendix B
stated · §1, App. B ('a total of 2.3T (trillion) training tokens'; 550K steps × batch 512 × context 8192)
stated · App. B (Table 5)
stated · App. B
stated · App. B (Optimization)
stated · §1, App. B
stated · App. B ('the second training stage includes a collection of small supervised fine-tuning (SFT) Q&A datasets'); note: this is a training-recipe stage, not a chat-model post-training pipeline — only the pre-trained base model is released (§1)
stated · report-wide absence
stated · §1 (model trained from scratch on 2.3T tokens from DCLM; no distillation mentioned)
n/a · §1 (text-only architecture)
stated · report-wide absence (inference speed benchmarks optimize with torch.compile and CUDA graphs, §5.2, but no serving-system optimizations are disclosed)