ZAYA1-8B is Zyphra's reasoning-focused MoE: 8.4B total parameters, 0.76B active, 40 layers, 16 experts per layer with top-1 routing and no shared or residual experts. Its block makes three departures from standard MoE transformers. First, sequence mixing uses Compressed Convolutional Attention in GQA mode (CCGQA): attention is performed in a compressed latent space after low-rank down-projections and lightweight sequence-mixing convolutions (a short convolution plus grouped head-wise convolution) precondition queries and keys; half the value heads are time-delayed by one token; query/key mean skip connections and an RMSNorm that applies a learned head-wise temperature to the keys (parameterized linearly as T, not exp(T)) keep attention logits bounded under QK-norm. With 8 query heads sharing 2 KV heads on top of 2× query compression, the KV cache is 8× smaller than full MHA. Second, the ZAYA1 router replaces the linear router with a down-projection to 256 dims, an Exponential Depth Averaging blend with the previous layer's routing representation (a DenseFormer-style depth-weighted averaging variant), and a three-layer GeLU MLP, with PID-controller-style bias balancing (AdamW on p_e − 1/E) rather than DeepSeek-style sign updates. Third, learned residual scaling (per-channel α, β on both the residual stream and layer outputs) controls residual-norm growth through depth. The model was trained from scratch for reasoning — long-CoT data in every pretraining phase with answer-preserving trimming, a 32K midtraining and 131K SFT on AMD MI300X + Pollara hardware, then a four-stage verifiable-RL cascade — and is co-trained with Markovian RSA, a test-time-compute workflow that recursively aggregates candidate reasoning traces while carrying forward only bounded tails (e.g. 4K of a 40K budget), lifting AIME'25 to 91.9%.
What the report claims is novel
CCA (Compressed Convolutional Attention) validated at 8B scale: sequence mixing in a compressed latent space via low-rank projections and convolutional preconditioners, competitive with MLA/GQA while cutting prefill FLOPs and KV-cache memory (8× compression in CCGQA mode) [§I, §II-A1, Appendix C]
ZAYA1 router: MLP-based router with Exponential Depth Averaging and PID-controller-style bias balancing, substantially more expressive than linear routers; better routing decisions reduce balancing instability and improve quality [§I, §II-A2]
Learned residual scaling (per-channel α, β on the residual stream and layer input at each block) controlling residual-norm growth through depth at negligible parameter/FLOP cost [§I, §II-A3]
Answer-preserving (AP) trimming: training-data construction that truncates the tail of over-length reasoning traces while preserving the final answer, enabling long-CoT data in short-context pretraining [§I, §III-A]
Four-stage cascaded RL pipeline (reasoning warmup, 400-task RLVE-Gym adaptive curriculum, math+code+TTC with synthetic code environments, behavioral RL) with DPPO Binary-TV trust region, Dr-GRPO aggregation, and MaxRL advantages [§I, §IV-B]
Markovian RSA: a test-time compute method recursively aggregating parallel reasoning traces while carrying forward only bounded-length tails (τ of β), trained into the model via SFT aggregation examples and RL variants; 91.9% AIME'25 / 89.6% HMMT'25 at β=40K/τ=4K [§I, §VI]
Evidence that a full-stack AMD platform (MI300X + Pensando Pollara 400) can support sustained pretraining, long-context midtraining, and SFT for an 8B-total-parameter MoE reasoning model [§I, §III, Appendix A]
With under 1B active parameters, ZAYA1-8B matches or exceeds DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks and stays competitive with much larger open-weight reasoning models [Abstract, §I, Table XI]
Atlas assessment: The three architectural changes are refinements of known families rather than new mechanisms: CCA/CCGQA is softmax attention moved into a compressed latent space with convolutional preconditioning (a compute-efficiency variant of GQA — the atlas's first compressed-latent attention, but conceptually adjacent to MLA's latent KV); the ZAYA1 router upgrades DeepSeek-V3-style bias balancing (PID update via AdamW) and adds a DenseFormer-style depth-averaged MLP router; residual scaling is a cheap alternative to Qwen-style attention gating. Top-1 routing with no shared experts is an aggressive sparsity choice not seen in atlas peers (deepseek-v3/mixtral use top-8 with shared experts). The genuinely novel contribution is Markovian RSA with training-time integration — the first TTC workflow trained into the model in the atlas — though it is an inference/training scheme, not an architecture change. vs deepseek-r1: same reasoning-specialised, verifiable-RL recipe at far smaller scale. The report is exemplary on training disclosure (phases, mixtures, precision, optimizer, stability engineering) but omits the expert FFN activation, tokenizer algorithm, training compute, and any weights-release statement.
Relation to the atlas
Lineage in
Compressed Convolutional Attention (CCA) — compressed latent-space attention with convolutional preconditioners, evaluated here at 8B scale from CCA (Figliolia et al., 2025) (not in atlas)
Exponential Depth Averaging (EDA) — a variant of Depth-Weighted Averaging for the router representation from DenseFormer (Pagliardini et al., 2024) (not in atlas)
Auxiliary-loss-free bias-based load balancing — PID-style update of the DeepSeek bias scheme from DeepSeek-V3
Grouped-Query Attention — KV sharing retained in CCGQA's 8-query/2-KV structure from GQA (Ainslie et al., 2023)
Markovian RSA combines Recursive Self-Aggregation with the bounded-workspace principle of Markovian Thinker from RSA (Venkatraman et al., 2025) / Markovian Thinker (Aghajohari et al., 2025) (not in atlas)
Muon optimizer (with AdamW RMS matching; momentum-free variant for RL) from Muon (Jordan et al., 2024) (not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
No statement that weights are released and no license named (openness undisclosed)
Expert FFN activation never named (GeLU appears only for the router MLP); Appendix D implies a gated FFN ('linear_fc1') without naming the activation
Tokenizer algorithm not described (only 'Gemma3 tokenizer, 262,272 vocabulary size')
No training compute (FLOPs or GPU-hours) disclosed
No learning-rate schedule disclosed (only per-stage RL LR ranges)
Input/output embedding tying not disclosed
No per-stage ablations of the RL cascade ('We do not report per-stage ablations in this release')
Individual training dataset names omitted (category-level proportions only)
No dedicated multi-turn agentic RL stage; agentic benchmarks (BFCL-v4, τ²) acknowledged to lag
Controlled optimizer ablation not run (momentum-free Muon presented as an empirical recipe choice)
Markovian RSA vs full-chain RSA comparison left for future work (only the τ=β limit is characterized analytically)
ZAYA1 router: down-projection to 256 dims, Exponential Depth Averaging blend with the previous layer's routing representation, then softmax(MLP(RMSNorm(r))) scores plus learned bias b; top-1 selection via argmax_e(s_e + b_e)[19]
Load balancing
Auxiliary-loss-free PID-controller-style bias balancing: bias gradient ∇b_e = p_e − 1/E (empirical routing fraction minus uniform), updated with AdamW across the global batch; builds on DeepSeek-V3's bias scheme[20]
Expert granularity
Relatively fine-grained: 16 experts per layer with a 2× hidden expansion (expert FFN 4096 pre-activation / 2048 post-activation)[21]
Learned key temperature T scaled linearly (not exp(T)) to bound the maximum attention logit at T·√d_h under QK-norm (stability modification of standard CCA); Query/key mean skip connections enforcing representational similarity; Learned residual scaling (α, β per layer on residual stream and layer output) controlling residual-norm growth through depth, with α initialized to ones and β to zeros[31]
Base pretraining: broad web-crawl distribution with code, math, multilingual, and reasoning data mixed in progressively; phase 2 upweights code, math, reasoning, and instruction-formatted data. 32K midtraining mixture: 86.1% long-CoT reasoning traces, 5.7% web/synthetic web/multilingual, 0.8% natively long-context, 3.0% code corpus, 3.0% math/STEM, 1.4% short instruction/few-shot. 131K SFT mixture: 75.0% long-CoT, 9.8% web, 6.4% natively long-context, 5.0% code corpus/code SFT, 2.6% math/STEM, 1.2% short instruction. Individual dataset names are omitted.[34]
Curriculum / staging
Staged: 4K base pretraining (two phases) → 32K reasoning-focused midtraining (1.2T tokens) → 131K SFT (660B tokens); long-CoT reasoning data in every pretraining/midtraining phase (majority of the midtrain mix) with stage-aware answer-preserving (AP) re-trimming at each context length; RL cascade follows post-training[35]
Optimizer
Muon with AdamW RMS matching (pretraining, midtraining, SFT); RL: momentum-free Muon for matrix-valued actor weights with AdamW for embeddings and LM head[36]
LR schedule
not disclosed
Batch schedule
RL: minibatches of 128 prompts with rollout group size G=16, 131,072-token budget per GPU per microbatch with sequence packing and dynamic batching; pretraining run example (Appendix B): G=4096, s=4096, 4 bytes/token[37]
Precision
BF16 weights and activations with a matched FP32 op set on trainer and inference engine: fused cross-entropy + LM-head matmul, CCA cache state, QK-norm, QK-mean, RMSNorm, router softmax, residual-stream additions[38]
Parallelism
Data parallelism plus all-gather KV context parallelism (2 ranks at 32K, 8 ranks at 131K); FSDP2 shard size 4 with sequence parallelism disabled; post-training uses asynchronous PipelineRL with disjoint rollout/trainer GPU pools (2–5× more rollout workers, weight sync every 2 trainer iterations)[39]
Trained from scratch for reasoning: long-CoT data in all pretraining/midtraining phases with answer-preserving trimming; verifiable RL over math and puzzles (232-step warmup; 400-task RLVE-Gym adaptive difficulty curriculum with Thompson-sampling IRT calibration), code (CodeI/O prediction, CodeARC reconstruction, falsification synthetic environments built from competitive-programming references), and TTC aggregation prompts (Markovian RSA expert- and self-aggregation); per-rollout response budgets up to 81,920 tokens; no dedicated multi-turn agentic RL stage[43]
CCGQA: KV-cache compression 8× relative to full multi-head attention (2 KV heads shared across 8 query heads on top of 2× query compression); CCA's compressed latent KV representation kept activation and KV-cache memory low enough for 131K-context SFT and RL[46]
stated · Appendix C ('The scaled query-key inner product is then upper bounded (assuming QK norm) to T·d_h/√d_h = T√d_h'); §IV-D (QK-norm and QK-mean in the FP32 op set)
stated · Appendix C, §II-A3 Eq. 6
stated · Table II (SFT phase at 131K context, 660B tokens, RoPE base 5M)
stated · Table I
stated · §III, Tables II–III
stated · §III, §III-A
stated · §III, §IV-E
stated · §IV-B1, §IV-C, Appendix B
stated · §IV-D
stated · §III, §IV-C, §IV-B1
stated · Table I, Appendix A (Table XIV)
stated · §IV-A ('SFT phase at 131K context for 660B tokens'; chat, IF, code, math, reasoning, tool-calling and TTC-aggregation mixture; best-fit decreasing bin packing)
stated · §IV-B, §IV-B1, §IV-B7
stated · Abstract, §III-A, §IV-B3–B6, §VI-B
stated · Abstract ('ZAYA1-8B was trained from scratch for reasoning')