Solar Open 2 is a 250B-parameter sparse MoE (15B active) that scales the Solar Open 1 backbone — 48 layers, hidden 4,096, head dim 128, 64 query / 8 KV heads, 196,608-token vocab — while replacing its all-softmax GQA stack with a 3:1 linear:softmax hybrid. Every block of four layers runs the softmax layer first (S-L-L-L, opposite to Kimi Linear and Qwen3.5), then three KDA linear-attention layers, each paired with an MoE FFN (320 routed experts, top-8, one shared expert, SiLU, per-expert width 1,280). Three extensions define the design: NoPE — no positional encoding anywhere, so the linear layers' sequential state carries all token order and the model extrapolates in principle without bound; a sigmoid output gate on the softmax layers that adds query-dependent sparsity and suppresses attention-sink collapse; and the negative-eigenvalue extension of the gated delta rule (β = 2σ(·) ∈ (0,2)), which widens the state-transition eigenvalues to [−1,1] so the recurrent state can erase and invert, not merely decay — restoring provable state-tracking (parity, modular counting) that [0,1]-eigenvalue cores lack. The 1M context is reached by staged training: 10T general pre-training, 1T intensive, then 0.9T length expansion, with four checkpoints merged. Training efficiency is attacked from two sides: a selective weight transfer that reuses only the 5.69B-parameter shape-compatible skeleton from Solar Open 1 (no distillation), and a value-per-token data recipe (20T pool → 10T, 4:6 real:synthetic). Post-training runs SFT → multi-domain RL → twelve domain specialists → Multi-teacher On-Policy Distillation, folding the specialists into one model with exact full-vocabulary KL on the student's own rollouts.
What the report claims is novel
Hybrid attention stack — three KDA linear-attention layers per one GQA softmax layer (softmax-first S-L-L-L ordering), with NoPE, a sigmoid output gate, and the negative-eigenvalue extension of the gated delta rule — reaching a 1M-token window at about one quarter of an all-softmax stack's memory and computation [Abstract, §2.2]
Selective weight transfer: the 5.69B-parameter shared skeleton (2.3% of the 250B total; 5.6% of Solar Open 1's 102B) is initialized from Solar Open 1 without distillation, absorbing the 128→320 expert expansion through large-scale full pre-training — claimed as the first large hybrid model pre-trained this way [Abstract, §3.1]
Value-per-token data curation: quality- and rarity-aware curation with mixture-ratio optimization refines a 20T pool into a 10T mixture that outperforms the Solar Open 1 recipe at equal token budget [Abstract, §3.2]
Twelve domain specialists consolidated into one deployable model by Multi-teacher On-Policy Distillation (MOPD), with the distillation KL computed exactly over the full 196,608-token vocabulary (no single-draw estimator) and teacher compute kept O(batch) by per-sample teacher routing [Abstract, §4.2.2]
A fully asynchronous RL system for 250B-scale training: stale-trajectory masking by fresh-token ratio, trust-region updates with token-level ratios, and packed teacher-pool management (one GPU-resident teacher plus twelve CPU snapshots) [§4.2.1, §4.2.2]
Korean-efficient tokenizer (inherited from Solar Open 1): 4.41 bytes per token on Ko-GDPval task prompts, first among 12 tokenizers compared and about 24% above the best global model (MiniMax-M3); global-model tokenizers spend 1.2–1.9× more tokens on the same Korean text [§1, §2.1]
Atlas assessment: The block is a well-executed synthesis of mechanisms already in the atlas: KDA linear attention (moonshot-kimi-k3), GQA (llama-3.1 lineage), NoPE hybrid stacking (kimi-k3), and sigmoid output gating (Qiu et al. 2025, per the report's own citation). The two genuinely differentiating choices are (1) softmax-first (S-L-L-L) ordering, explicitly opposite to Kimi Linear/Qwen3.5's L-L-L-S, and (2) the negative-eigenvalue extension (β = 2σ(·) ∈ (0,2)) that widens the gated delta rule's state-transition eigenvalues to [−1,1] — an incremental but real generalization of the KDA recurrence that Kimi's own lower-bounded-decay parameterization does not provide. At the system level, selective weight transfer across an architectural change without distillation (only 2.3% of parameters reused) and exact full-vocabulary MOPD at 250B scale are notable engineering contributions. The 250B-A15B footprint with 1M context and 3:1 linear:softmax ratio is essentially the kimi-k3 hybrid recipe at one-ninth the total size with GQA instead of MLA.
Relation to the atlas
Lineage in
KDA (Kimi Delta Attention) — gated delta-rule linear attention with fixed-size recurrent state for the 36 linear layers from Kimi (Kimi Linear / Kimi K3 family)
Hybrid attention stack interleaving linear and softmax layers (following Kimi Linear, Qwen3.5, and Nemotron 3) from Kimi Linear / Qwen3.5 / Nemotron 3
GQA (grouped-query attention) for the softmax layers from GQA (Ainslie et al.)
Elementwise sigmoid output gate on softmax attention outputs from Sigmoid-attention line of work (Qiu et al., not in atlas)
Influence out
No descendants recorded yet.
Notable omissions
No optimizer, learning-rate schedule, batch size, or training precision disclosed
No training hardware or FLOPs/GPU-hours disclosed (Lablup's GPU clusters credited but never specified)
Router function and load-balancing mechanism not described (only the top-8 count)
Normalization layer type not named (only 'pre-norm residuals')
Quantization formats not disclosed (Nota Inc. credited only in acknowledgments)
No license named for the weights release
No embedding-tying statement
SFT and Multi-domain RL data compositions explicitly 'outside the scope of this report' (§4)
Checkpoint-merging method (Stage 4) not detailed
Per-stage data-mixture proportions not disclosed (only the overall recipe)
MOPD hyperparameters (ρ threshold, trust-region bounds, G) not quantified
per block of four: one softmax (GQA) layer first, then three linear (KDA) layers — S-L-L-L; repeated 12 times; 12 softmax (25%) + 36 linear (75%) layers; softmax-first ordering explicitly contrasts with the L-L-L-S ordering of Kimi Linear and Qwen3.5[16]
none — no positional-encoding extension needed (NoPE); the 1M window is reached by staged training, with Stage 4 (Length Expansion, 0.9T tokens) extending the context window to 1M[29]
inherited unchanged from Solar Open 1; trained on a corpus oversampling Korean and target domains, with digit splitting and whitespace preservation for arithmetic and code fidelity; 4.41 bytes/token on Ko-GDPval task prompts (first among 12 tokenizers compared, ~24% above the best global model MiniMax-M3); identical vocabulary across generations is a precondition for transferring embedding/output-layer weights[31]
20T initially cleaned pool refined to 10T: every dataset receives an in-house quality score and rarity tracking by source/type; exact and semantic deduplication; mixture-ratio optimization via ablations — real-to-synthetic ratio 4:6, math and code each at least 15%, English share above 80%[33]
Curriculum / staging
four-stage pre-training: Stage 1 Selective Weight Transfer (2.3% of parameters from Solar Open 1), Stage 2 General Pre-training (10T tokens, broad-coverage mixture), Stage 3 Intensive Pre-training (1T tokens, higher quality threshold on the retained subset), Stage 4 Length Expansion (0.9T tokens, long-document data incl. repo-level code, extends context to 1M); four intermediate checkpoints merged into the final model (highest PLM score 0.745)[34]
Optimizer
not disclosed
LR schedule
not disclosed
Batch schedule
not disclosed
Precision
not disclosed
Parallelism
Context Parallelism (CP): student and teacher models are sharded by CP rank during RL/MOPD training, and teacher transport ships only the matching CP shard, skipping the CP all-gather; other parallelism dimensions (TP/PP/EP) not disclosed[35]
Multi-domain RL strengthens core reasoning with verifiable rewards over a broad set of STEM-centered tasks; twelve domain specialists (reasoning: math/STEM/code; agents & tools: coding, general agents, officework) trained via domain SFT and domain-specific RL with verifiable rewards; SFT covers the full range of reasoning and agent tasks[38]
Distillation
Multi-Teacher On-Policy Distillation (MOPD): the student is trained on its own rollouts against twelve domain-specialist teachers; per-position distillation KL computed in closed form over the full 196,608-token vocabulary (teacher ships pre-lm_head hidden states, student rebuilds logits in 1,024-token tiles); each sample is routed to exactly one teacher, keeping teacher compute O(batch); note the selective weight transfer from Solar Open 1 is initialization, explicitly 'without distillation'[39]
hybrid cache: 12 softmax GQA layers keep a KV cache with 8 KV heads, while 36 linear KDA layers keep a fixed-size recurrent state and no KV cache — total cache constant in sequence length, roughly one quarter of an all-softmax stack's memory
Quantisation shipped
none disclosed
Speculative decoding
none disclosed
Serving optimisations
none disclosed
Evaluation
Benchmarks (report's own numbers only)
Benchmark
Value
Ref
MMLU-Pro
86.2
Table 3
GPQA-Diamond
86.3
Table 3
HLE (w/o tools)
28.8
Table 3
LiveCodeBench (v6)
92.4
Table 3
ArtifactsBench
55.9
Table 3
HMMT2602
93.9
Table 3
AIME2026
95.7
Table 3
Multi-Challenge
61.0
Table 3
IFBench
80.0
Table 3
AA-LCR
62.3
Table 3
SWE-Bench Verified
70.4
Table 3
Terminal Bench Hard
28.3
Table 3
APEX-Agents
16.6
Table 3
MCP-Atlas
58.2
Table 3
τ3 (banking)
19.6
Table 3
GDPval-AA v2 (ELO)
1128
Table 3
KMMLU-Pro
78.4
Table 4
CLIcK
90.7
Table 4
Ko-AIME'25
97.7
Table 4
HRM8K
92.2
Table 4
KorMedMCQA
93.0
Table 4
Ko-GDPval
86.8
Table 4
Korean suite Avg.
85.4
Table 4
Provenance references
stated · Title ('Solar Open 2 Technical Report')
stated · Title page byline ('Upstage Solar Team'); website upstage.ai
stated · §1 ('an openly available, nationally grounded frontier model'); §4 (MOPD consolidates twelve teachers 'into the single released model'); Acknowledgments ('the development and release of Solar Open 2')
stated · Abstract, §2, Table 1 ('250B total parameters')
stated · Abstract, §2, Table 1 ('15B active parameters per token')
stated · Table 1 ('Attention Heads (Q / KV) 64 / 8'), note a: 'KV heads apply only to the softmax layers; the linear layers use 64 heads and keep no KV cache'
stated · §2.2, Table 1 ('Hybrid: [Softmax ×1, Linear ×3] ×12'; GQA softmax layers + KDA linear layers)
stated · §2.2 ('interleaves one softmax-attention layer with three linear-attention layers in every block of four... so that 12 of the 48 layers (25%) are softmax and 36 (75%) are linear'; 'the softmax layer comes first (S-L-L-L)')
stated · Table 1 ('Activation: SiLU')
stated · §2, Table 1 ('Num Experts (Routed) 320')
stated · Table 1 ('Experts per Token (Top-k) 8')
stated · §2, Table 1 ('Num Shared Experts 1'; 'a single shared expert with no dense layers')
stated · §2.1 ('a custom byte-level BPE tokenizer with a 196,608-token vocabulary')
stated · §2.1
stated · §3, Figure 5 caption ('Stages 2–4 then train on about 12T tokens in total: General Pre-training (10T), Intensive Pre-training (1T), and Length Expansion (0.9T)')
stated · §3.2
stated · §3, §3.3
stated · §4.2.2 ('Under context parallelism, the teacher's CP rank r already holds exactly the shard student CP rank r needs')
stated · §4 ('Post-training of Solar Open 2 proceeds in four stages: (1) Supervised Fine-Tuning (SFT)...'; SFT 'targets the full range of reasoning and agent tasks')
stated · §4.2.1 (fully asynchronous RL with verifiable rewards; 'Retaining token-level ratios as in GRPO (Shao et al., 2024)' with trust-region clipping, stale-trajectory masking via fresh-token ratio threshold ρ, and sample dropping — GRPO-style but modified, so tagged other)