Ling-2.6-1T is an 80-layer, ~1T-parameter sparse MoE (256 routed experts, 8 active, 1 shared per layer; 4 dense blocks up front) that is remarkable less for its block than for how the block was obtained: it is an architectural transplant of the trained Ling-2.0-1T GQA checkpoint rather than a from-scratch model. A four-step migration (Lightning Attention Conversion, Linear Warmup, MLA Conversion, MLA Warmup) replaced most GQA layers with Lightning Attention — a simple O(N) linear attention — and converted the remaining full-attention layers from GQA to MLA, fusing the old QK Norm into the projections by calibration and adapting the Partial RoPE (rotary dim 64, theta 6M) to TransMLA. The final block mixes 7 linear-attention layers with 1 MLA layer per group of 8 (70:10 over the stack), keeping KV-cache pressure and long-context FLOPs low at the 262,144-token context, which is reached by a 4K-to-256K training curriculum rather than any post-hoc extension. FFNs are fine-grained MoE with grouped routing (8 groups, top-4), sigmoid FP32 scoring, expert bias for auxiliary-loss-free balancing, and a 2.5x output scale. Post-training diverges into two lines from the shared base: Ling-2.6 is an instant model whose Evo-CoT/LPO RL, bidirectional preference alignment, and shortest-correct-response distillation push capability per output token (~4x token efficiency claimed), while Ring-2.6 adds KPop agentic RL (binary-KL masking) and adaptive thinking for long-horizon tool use. Training is FP8 throughout, served by the linghe fused kernels and an AllGather context-parallel scheme, with MTP layers doubling as speculative-decoding drafts.
What the report claims is novel
Hybrid linear attention retrofit: Lightning Attention + MLA at a 7:1 ratio chosen by equal-FLOPs scaling laws, transplanted into the trained Ling-2.0-1T checkpoint via a four-step lossless migration (hybrid initialization, QK Norm absorption, Partial-RoPE-aware TransMLA conversion, MLA warmup) [Abstract, §2.1]
Token-efficiency post-training suite — Evo-CoT, Linguistic Unit Policy Optimization (LPO), bidirectional preference alignment, and shortest-correct-response distillation — delivering approximately 4x higher token efficiency on reasoning workloads than the 2.0 generation (AA Intelligence Index 34 at ~16M output tokens) [Abstract, §3.1]
KPop, a novel RL algorithm that replaces IcePop's uniform fixed-ratio constraint with symmetric binary KL divergence masking, enabling stable trillion-parameter-scale agentic RL (SWE-bench Verified solve rate 70.8% -> 76.28% during training) [§3.2.3]
Infrastructure co-design: AllGather context parallelism for linear attention (free of head-divisibility constraints), linghe fused-kernel library, continued MTP training for speculative decoding, ARouter asynchronous RL with bounded staleness, and integrated FP8 training/inference [§4]
Atlas assessment: The closest atlas peer is moonshot-kimi-linear-48b-a3b: both are text-only, open-weights hybrids of a linear-attention family with MLA over a fine-grained MoE backbone aimed at agentic/long-context efficiency — and Kimi Linear's own hybrid-ratio ablation explicitly tested and rejected 7:1 in favor of 3:1, which Ling-2.6 adopts on its own scaling-law evidence. The genuinely distinctive axis is that Ling-2.6 is not trained from scratch: it is an architectural transplant of a trained trillion-parameter GQA checkpoint (the QK-Norm absorption via calibration statistics and the Partial-RoPE-decoupled TransMLA conversion are a novel retrofit procedure with no atlas precedent), and it uses Lightning Attention (TransNormerLLM lineage) rather than the delta-rule KDA used by both Moonshot records. The token-efficiency post-training direction (Evo-CoT, LPO, bidirectional preference alignment) and the grouped-routing fine-grained MoE inherited from Ling-2.0 are family-specific rather than novel-at-large; KPop is an incremental refinement of the team's own IcePop (Ring-1T), which is not itself an atlas record. The '1T' scale, ~4x token-efficiency gain, and 87.6 PinchBench lead are the report's own figures and not independently verifiable from the atlas.
Relation to the atlas
Lineage in
Lightning Attention — linear attention with O(N) recurrence used as the majority mixing layer from TransNormerLLM (not in atlas)
Multi-head Latent Attention (MLA) — low-rank latent KV compression for the full-attention layers from DeepSeek-V2
TransMLA weight-conversion method for GQA-to-MLA structural conversion, adapted to Partial RoPE by decoupling the RoPE-affected dimensions from TransMLA (not in atlas)
Hybrid linear-attention conversion recipe (GQA dims expanded to MHA, gating parameters Wgate / gamma_gate introduced) from Ring-flash-linear-2.0 (Inclusion AI, own prior work; not in atlas)
GQA backbone, fine-grained MoE with grouped routing, Evo-CoT framework, and WSM learning-rate scheduler from Ling-2.0 (Inclusion AI, own prior work; not in atlas)
IcePop double-sided-masking RL stabilization — KPop replaces its uniform constant-ratio constraint with binary KL divergence from Ring-1T (Inclusion AI, own prior work; not in atlas)
Group Sequence Policy Optimization (GSPO) for token-efficient agentic RL from GSPO (Zheng et al., 2025; not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
Exact parameter counts never printed — only 'scales from 104B to 1T parameters'; no active-parameters-per-token figure (the 8/256 routing shape is given but not the resulting active count)
Training hardware and compute (FLOPs/GPU-hours) not disclosed
Optimizer not named ('all other hyper-parameters remain consistent with Ling-2.0'); Muon appears only as a future direction
Peak learning rate and batch size values not printed (scaling laws re-derived but figures withheld)
Tokenizer algorithm not described (only the 157,184 vocab size appears)
Input/output embedding tying not disclosed
Normalization placement (pre/post) not described
No license named for the open-sourced checkpoints
Ring-2.6-1T's architecture is not in Table 1 (only flash and 1T columns); its identity as the shared 1T base is inferred from Figure 6
The position of the single MLA layer within each 8-layer group is not disclosed
KPop technical details deferred to an external blog (ringtech.notion.site/kpop)
Some Table 6 peer numbers are cited from other labs or marked 'results from our evaluation' (*), and several cells are missing (-); SWE benchmarks use Claude Code scaffolding rather than the model's own agent harness
No attention-sink or logit-softcapping mechanisms discussed (absence defaults apply)
Source extraction: ar5iv HTML conversion failed (fatal conversion banner, 229 chars), so text was extracted from the official arXiv PDF (fitz); embedded formulas and figure-internal labels are partially garbled (e.g., Eq. 5, Table 6 footnote markers), and a handful of control bytes were stripped during cleaning
layer groups of M = 8 throughout the 80-layer stack: 7 Lightning Attention (linear) layers + 1 Full Attention (MLA) layer per group, i.e. 70 linear + 10 MLA layers total; the in-group position of the MLA layer is not disclosed[14]
grouped routing: ngroup = 8 groups with top-4 group selection; router scores with sigmoid in FP32; routed output scaled by 2.5 with normalized top-k probabilities[19]
Load balancing
auxiliary-loss-free load balancing with expert bias (bias-update rate gamma = 0.001 during continue pre-training, reduced to 0.0001 for mid-training)[20]
Expert granularity
fine-grained MoE inherited from Ling-2.0: 1 shared + 256 routed experts per layer, per-expert intermediate width 2,048 (1,024 for flash)[21]
none — training ran at the full deployed length via a progressive context curriculum (4K -> 32K -> 256K during pre/mid-training; post-training SFT at 256K); no post-hoc extension method used[22]
Three-stage mixture. Migration/continue pre-training: ~46% reasoning-intensive domains (math, code), ~50% general corpora (web text), ~4% multilingual; agentic corpus spanning tool use (500+ real-world MCP environments, 3,000+ tools) and agentic coding with bash/web/repository tasks, teacher-generated with verification; long-context corpus (math, web parsing, summarization, RAG fusion, multi-hop reasoning) with rule+model defect detection; web corpus with STEM recall and atomic-fact construction from Wikipedia; multilingual coverage for 21 languages including 1.1T tokens from Fineweb2/Fineweb2-hq and ~70B synthetic web code. Mid-training strips general web data to ~43% general / 42% reasoning / 15% agentic at 256K.[30]
Curriculum / staging
Multi-stage: (1) Migration Pre-Training (~400B tokens, four steps: Lightning Attention Conversion -> Linear Warmup -> MLA Conversion [QK Norm absorption, partial-parameter training, Partial-RoPE-aware TransMLA conversion] -> MLA Warmup); (2) Continue Pre-Training (8T tokens, 4K context, aggressive data-switching strategy adopted over conservative); (3) Mid-Training (~1.2T tokens: 250B at mixed 32K, then 425B at 32K, then 525B at 256K, high-quality mixture). WSM scheduler throughout.[31]
Optimizer
not disclosed
LR schedule
WSM scheduler from Ling-2.0: linear warmup to peak learning rate, constant phase until training concludes, final annealing achieved through checkpoint merging[32]
Batch schedule
not disclosed
Precision
FP8 training from pretraining through supervised fine-tuning, with FP32 optimizer master weights retained for RL initialization; FP8 continuation training for RL; FP32 LM Head in both training and inference[33]
Parallelism
co-designed expert, pipeline, and context parallelism with selective activation recomputation; AllGather-based Context Parallel for Lightning Attention (local-recurrence-then-global-correction, no head-divisibility constraint); Triton-fused varlen state-correction kernel (~68% end-to-end speedup at 256K)[34]
Evo-CoT RL (from Ling-2.0) with composite rewards: accuracy, formatting penalty for reasoning markers, dynamic length penalty (difficulty-specific), and LLM-judge semantic redundancy penalty; Linguistic Unit Policy Optimization (LPO); Dynamic Pass Rating (DPR) adaptive curriculum; GSPO for token-efficient tool use (process reward on tool-call sequence + zlib-compression repetition penalty); Ring-2.6 adds KPop agentic RL (binary-KL divergence masking) and adaptive thinking with high/xhigh effort modes[37]
Distillation
specialist distillation: reasoning and agentic specialist models are distilled back into the unified Ling-2.6 / Ring-2.6 model (§3.1, §3.2); shortest-correct-response distillation, where proprietary expert models generate SFT responses and only the shortest accurate candidate is retained (§3.1.2)[38]
hybrid: MLA layers compress the KV cache into a low-rank latent space (KV LoRA Rank 512, Q LoRA Rank 1536, Table 1); Lightning Attention layers use an O(N) linear-attention recurrence with no growing KV cache. The 7:1 mix 'reduces long-context compute cost, KV-cache pressure, and decoding latency'.[41]
Quantisation shipped
FP8 training and inference; module-aware FP8 quantization at deployment (attention linears and shared-expert linears in BF16, routed-expert linears blockwise FP8); BF16 inference path also supported; BF16-input/FP32-output router and LM-Head GEMMs[42]
Speculative decoding
MTP: two additional MTP layers added during post-training and continued-trained; MTP layers serve as a draft model for speculative decoding (4 speculative steps); parameter sharing across MTP layers with gradient detachment from all but the first (MTP-3-share) raises accepted length to 3.31 vs 2.71 for MTP-1[43]
Serving optimisations
linghe fused-kernel library (open-sourced): QK Norm + RoPE fusion, group RMSNorm + sigmoid-gate fusion, MLA RoPE and Top-K optimization, split-K blockwise FP8 GEMM for small batches; prefix caching and multi-token generation; AllGather context parallelism; ARouter global rollout scheduling with tail-request migration, spillover training-inference overlap, and instance failover; up to 4x prefill/decode acceleration for Ling-2.6-flash vs similar-size peers[44]