Trinity Large is Arcee AI's 400B-parameter MoE (13B active per token) and the flagship of a three-model family (Nano 6B/1B, Mini 26B/3B) trained as a scaling ladder. The block is a conservative synthesis of the 2025 open-model toolkit, tuned for training stability under the Muon optimizer and for inference efficiency. Attention interleaves three local sliding-window layers (window 4096, RoPE) with one global layer per four, the global layers running without positional embeddings (NoPE); the 3:1 schedule follows the RoPE-to-NoPE hybrid results of Yang et al. and gives the 45:15 layer split. On top of GQA and QK-norm (RMSNorm on queries and keys, chosen because Muon inflates attention logits), each head's output is elementwise gated by a sigmoid projection — gated attention — which prior work ties to fewer attention sinks and fewer loss spikes. Layers use a depth-scaled sandwich norm (pre- and post-RMSNorm with the post gain initialized to 1/sqrt(L)). The MoE follows DeepSeekMoE: 256 fine-grained routed experts plus one always-active shared expert, but Trinity Large deliberately activates only 4 experts per token with larger experts (size 3072) for throughput, reaching ~30x sparsity; routing is normalized sigmoid with bias-decoupled top-K, and load balancing combines the new SMEBU scheme (tanh soft-clamped, momentum-smoothed bias updates) with a small sequence-wise auxiliary loss. The first 6 layers are dense to stabilize early representations. Training ran 17T tokens on 2048 B300 GPUs with Muon + AdamW, HSDP/FSDP with in-node expert parallelism, and DatologyAI's >8T-token synthetic data mix; context was extended by training directly at 256K with only the NoPE global layers 'adjusted', yielding MK-NIAH 0.994 at 256K and 0.976 extrapolated at 512K.
What the report claims is novel
Trinity family culminating in Trinity Large: 400B total / 13B active open-weight MoE with extreme sparsity, trained with zero loss spikes across all three models [Abstract, §7]
SMEBU (Soft-clamped Momentum Expert Bias Updates): tanh soft-clamped magnitude-aware momentum bias updates for auxiliary-loss-free MoE load balancing, addressing sign-update oscillation near convergence as expert count grows [§2.3]
RSDB (Random Sequential Document Buffer) and the BatchHet metric: reduce intra-batch document correlation in sequence packing (BatchHet -4.23x and step-to-step variance -2.4x in Trinity Large phase 3) [§3.2]
One of the largest publicly documented synthetic-data efforts for pretraining: over 8 trillion synthetic tokens generated via DatologyAI's Ray/vLLM-on-Kubernetes stack [§3.1]
Muon optimizer at 2048-B300 scale and smooth context extension: trained at 256K for a 256K target with MK-NIAH @256K 0.994, extrapolating to 0.976 @512K and 0.42 @1M without training at those lengths [§3.3, §3.5]
Atlas assessment: The block is a well-documented synthesis of mechanisms already in the atlas rather than a new mixing family: the 3:1 local/global RoPE/NoPE schedule follows Yang et al. 2025 and the Gemma/Qwen3-Next local-global lineage; QK-norm, gated attention, sandwich norm, sigmoid routing and aux-loss-free balancing are each prior work (Henry 2020, Qiu 2025, Yin/Pangu Ultra 2025, Wang 2024a/DeepSeek-V3). Gated attention is the first appearance in the atlas. The genuinely new items are engineering-level: SMEBU (a tanh+momentum refinement of DeepSeek-V3's sign-based bias update) and RSDB/BatchHet for dataloader stability, plus the Muon-at-scale training recipe (2048 B300s, zero loss spikes). The report is candid that the six stability fixes were applied jointly without ablations. Note: the gallery card labels Trinity 'reasoning-specialised', but the report frames the family as general-purpose open-weight foundations for enterprise/agentic deployment with an emphasis on efficiency and stability; the light post-training (Preview, AIME25 24.36) does not support a reasoning-specialisation claim.
Relation to the atlas
Lineage in
3:1 local/global attention (RoPE local, NoPE global) hybrid schedule from Yang et al. 2025 'RoPE to NoPE' (not in atlas)
Gated attention (elementwise sigmoid gating of attention output) from Gated Attention (Qiu et al. 2025, not in atlas)
QK-normalization (RMSNorm on queries and keys) from QK-Norm (Henry et al. 2020)
Fine-grained routed experts with always-active shared expert (DeepSeekMoE design) from DeepSeekMoE / DeepSeek-V3
Sigmoid routing and auxiliary-loss-free bias-based load balancing from DeepSeek-V3
Depth-scaled sandwich norm (pre + post normalization, depth-scaled gain) from Pangu Ultra (Yin et al. 2025, not in atlas)
Muon optimizer for hidden layers with AdamW for embeddings from Muon (Jordan et al. 2024, not in atlas; also used by GLM-4.5)
Sequence-wise load-balance auxiliary loss from DeepSeek-V3
Influence out
No descendants recorded yet.
Notable omissions
No license named for the weights release
No compute disclosed (FLOPs/GPU-hours)
RL algorithm not named (prime-rl; no PPO/GRPO/DPO designation)
Post-training deliberately light: Trinity-Large-Preview is a 'preliminary release' with only 5 instruct benchmarks; no agentic or long-context evaluation tables for the chat model
No per-benchmark comparison table vs peers (Figure 3 is chart-only)
RoPE base frequencies not disclosed (theta discussed qualitatively)
The six stability fixes were applied jointly with no controlled ablations (admitted in §6)
Embedding/output-head tying not disclosed
MK-NIAH long-context results appear only in the context-extension section, not the main evaluation suite
Tokenizer trained before the Large corpus was finalized, so CJK compression trails peers (stated)
No training loss curves for Nano/Mini (Figure 1 covers Large only)
SFT data recipe described qualitatively; no data volumes for post-training
3:1 local:global repeating for full depth: 3 local SWA layers (RoPE, window 4096) then 1 global NoPE layer; totals 45 local + 15 global (60 layers), with 6 initial dense layers[14]
sigmoid routing with normalized router scores (no softmax); top-K selected by router score + expert bias, gating scores (bias excluded) renormalized and applied to each expert's output[19]
Load balancing
Trinity Large: SMEBU (Soft-clamped Momentum Expert Bias Updates) — tanh soft-clamped magnitude-aware bias updates with momentum buffer (lambda=5e-4, beta=0.5, kappa=2), plus sequence-wise auxiliary loss (alpha=1e-4); Mini/Nano: standard auxiliary-loss-free sign-based bias updates with re-centering[20]
Expert granularity
fine-grained routed experts (DeepSeekMoE design); Trinity Large opts for coarser-grained experts (expert size 3072, 4 active per token) for throughput, with greatly increased sparsity[21]
direct training at 256K from the final pretrained checkpoint (no progressive extension); only global (NoPE) layers adjusted while local SWA window/RoPE base stay fixed; 117B-token long-context dataset (length-biased pretraining sampling, olmOCR/FinePDF-edu OCR PDFs, regenerated ProLong at full length, FLAN/math/code); MK-NIAH @256K 0.994, @512K 0.976, @1M 0.42[22]
DeepSeek-V3-inspired multi-stage pretokenizer: digit isolation with place-aligned 3-digit chunking (510-char cap avoids regex catastrophic backtracking), script-aware isolation extended to Thai/Lao/Khmer/Myanmar/Hangul, DeepSeek V3 main text regex, byte-level fallback; SuperBPE evaluated but rejected (better compression, no downstream gain); trained on ~48GB (~10B tokens) before the Large corpus was finalized, so CJK coverage trails DeepSeek V3/Qwen 3[31]
DatologyAI-curated: 20T-token mix (13T/4T/3T across three phases) combining curated web-scale data with >8T synthetic tokens (approx 6.5T rephrased web via BeyondWeb-style generation, ~1T multilingual, ~800B code); targets English plus 14 languages (Arabic, Mandarin, Japanese, Spanish, German, French, Italian, Portuguese, Indonesian, Russian, Vietnamese, Hindi, Korean, Bengali); 10T mix reuses the AFM-4.5B dataset with more math and code[33]
Curriculum / staging
three phases with midtraining mix shifts toward code/math/science and higher-quality data; on-the-fly tokenization with sequence packing; RSDB (Random Sequential Document Buffer) + BatchHet metric introduced for Trinity Large phase 3 (BatchHet reduced 4.23x, step-to-step variance 2.4x)[34]
Optimizer
Muon for hidden layers + AdamW for embedding and output layers; no RMS rescaling of Muon updates; fanout/fanin learning-rate adjustment rule lr_adj = lr * sqrt(max(1, fanout/fanin))[35]
LR schedule
linear warmup of 2000 steps; Large peak LR 8.0e-4 (Muon) / 2.0e-4 (AdamW); cosine decay to 1/10 peak in decay phase; context extension continues cosine decay from 1/10 to 1/20 of peak (8.0e-5 -> 4.0e-5 Muon, 2.0e-5 -> 1.0e-5 AdamW)[35]
Batch schedule
Large: global batch 12288 at sequence length 8192, increased to 16384 after crossing 4.9T tokens; Nano/Mini: 4096 -> 8192 when scaling GPU clusters[35]
Precision
BF16 (MXFP8 kernels for linear layers and grouped GEMMs were disabled mid-run with fallback to BF16 to stabilize training)[36]
Parallelism
HSDP: multiple model replicas with FSDP within replica groups (FSDP group size 128); Trinity Large additionally uses Expert Parallelism within a GPU node (EP group size 8); context parallelism degree 4 for Large context extension[37]
short RL stage on verifiable rewards (strict answer-format validation) with learned reward model for prompts without ground truth; no reasoning/CoT-specific curriculum described; report explicitly frames Trinity-Large-Preview as a light, preliminary post-training[40]
GQA with 8 KV heads + 3:1 local/global pattern: 45/60 layers are SWA (window 4096) so their KV cache is bounded by the window; 15 global (NoPE) layers keep full cache; GQA reduces KV-cache size relative to MHA
Quantisation shipped
FP8 (all inference benchmarks run with models quantized to FP8 in vLLM)[42]
stated · §2.4 (depth-scaled sandwich norm: 'Both the input and output of the module are normalized' — pre-norm and post-norm per sublayer)
stated · §2.2 (QK-normalization: RMSNorm applied to queries and keys before scaled dot-product attention, Eq. 4-5)
stated · §3.5, §3.4.2 (Large trained at 256K sequence length)
stated · §3.4.2 ('We train to 256k context for inference at 512k')
stated · §2.1 ('We train a custom 200,000-token BPE vocabulary')
stated · §2.1.1, §2.1.2, §2.1.3
stated · Abstract, §1, §3.1 (Large: 17T of a 20T mix; Nano/Mini: 10T each)
stated · §3.1
stated · §3.1, §3.2
stated · §3.4.2
stated · §6; inference benchmarks run with FP8 quantization (§5.2)
stated · §3.3
stated · §4 (SFT at 64K sequence length; public + custom instruction data with synthetic teacher-model instructions; heavy agentic coding supervision via OpenCode trajectories; Cut Cross-Entropy)
stated · §4 (short RL stage using prime-rl with verifiable rewards and a learned reward-model fallback; no RLHF/DPO/GRPO named)
stated · §4, §7
n/a · modality.type (text-only)
stated · §5.2
stated · report is silent on speculative decoding
stated · vLLM used for inference benchmarks (§5.2) but no serving optimizations disclosed