Jamba interleaves Transformer and Mamba layers into repeated 'Jamba blocks' — the report's term for l=8 layers mixed at a 1:7 attention-to-Mamba ratio — and adds mixture-of-experts to some MLPs. The released configuration stacks 4 blocks: 32 layers total, of which only 4 are GQA attention layers (positions 4, 12, 20, 28) and the rest are Mamba state-space layers; every other layer's MLP is replaced by an MoE with 16 experts and top-2 routing, giving 52B total but only 12B active parameters. The design deliberately trades attention for Mamba to shrink the KV cache (reported 8x smaller than a vanilla Transformer; 4GB at 256K context) and raise long-context throughput (3x Mixtral at 128K), while keeping enough attention for in-context learning — the report shows pure-Mamba models fail to follow few-shot formats, and the hybrid restores Transformer-like behaviour with as little as one attention layer in eight. Notable choices: no explicit positional encoding (Mamba layers are said to provide implicit position), RMSNorm inside Mamba layers for training stability, SwiGLU activations, a 64K-token BPE tokenizer with per-digit tokens, and GQA. Trained on contexts up to 1M tokens on H100s with FSDP/tensor/sequence/expert parallelism, the released base model serves 256K-token contexts from a single 80GB GPU with int8 weights and reaches parity with Mixtral-8x7B and Llama-2 70B on standard benchmarks. The report covers a single configuration and discloses few per-dimension numbers; most ablation evidence comes from 1.3B and 7B runs.
What the report claims is novel
First production-grade hybrid Attention-SSM language model, interleaving Transformer and Mamba layers (prior hybrids were small-scale or lagged pure models). [§1]
MoE applied to every other MLP layer (16 experts, top-2, load balancing), increasing model capacity without proportionally increasing active parameters or compute, and shown beneficial for the hybrid at 7B scale. [§1 / §3.1 / §6.3]
Hybrid reaches Transformer-level quality — comparable to Mixtral-8x7B and the larger Llama-2 70B on standard benchmarks — while achieving up to 3x the throughput and an 8x smaller KV cache. [Abstract / §1 / §5.1]
Long-context capability of 256K tokens — described as the longest context supported by a production-grade publicly available model at the time — with strong needle-in-a-haystack performance using only 4 attention layers. [§1 / §5.2]
Ablation-derived insights: explicit positional information is unnecessary in the hybrid (Mamba layers before attention provide implicit position), and RMSNorm on Mamba internal activations stabilises large-scale training. [§6.4 / §6.5]
Open release of weights under Apache 2.0, plus a plan to release checkpoints from ablation runs to encourage community study of the architecture. [Abstract / §1]
Atlas assessment: Every ingredient is pre-existing — the report itself cites H3, Hyena, StripedHyena, Block-State Transformer, and MoE-Mamba as earlier Attention-SSM hybrids — so Jamba's novelty is scale and integration: the first production-grade hybrid at 52B total / 12B active parameters, a 1:7 attention-to-Mamba recipe with MoE on every other layer, and open weights. The empirical findings (no positional encoding needed; RMSNorm on Mamba internals to prevent loss spikes) are genuine, reusable contributions, and the 256K-token context with a 4GB KV cache stood out against contemporaneous open models. Relative to atlas peers, Jamba is the anchor for hybrid sequence mixing (taxonomy Axis 2).
Relation to the atlas
Lineage in
Mamba selective state-space layers (the SSM component of each block) from Mamba (Gu & Dao, 2023)
Sparsely-gated mixture-of-experts on MLP layers (n experts, top-K router, load balancing) from Switch Transformers / sparsely-gated MoE (Shazeer et al.)
Transformer decoder architecture with self-attention layers from Transformer (Vaswani et al., 2017)
Each digit is a separate token; the dummy space used in the Llama and Mistral tokenizers is removed for more consistent and reversible tokenization. Vocabulary size 64K (recorded in scale.vocab_size).[6]
Training
Training tokens
not disclosed
Data composition
In-house dataset containing text from the Web, books, and code, last updated March 2024; processing pipeline includes quality filters and deduplication; mixture proportions not disclosed.[16]
Curriculum / staging
none disclosed
Optimizer
not disclosed
LR schedule
not disclosed
Batch schedule
not disclosed
Precision
not disclosed
Parallelism
In-house proprietary training framework with FSDP, tensor parallelism, sequence parallelism, and expert parallelism[16]
GQA attention on only 4 of 32 layers: report states an 8x smaller KV cache than a vanilla Transformer and 4GB at 256K context (16-bit) vs 32GB for Mixtral and 128GB for Llama-2 (Table 1)[18]
Quantisation shipped
INT8 (inference): model fits in a single 80GB GPU with int8/8bit weights, including at contexts over 128K tokens[19]