Solar Open 2 Technical Report
Abstract Overview
Solar Open 2 is a 250B-parameter, 15B-active Mixture-of-Experts language model designed for long-horizon agentic tasks and Korean-language strength. The report attributes its 1M-token context window to a hybrid attention stack that mixes softmax and linear attention, removes positional encoding, and extends the linear attention state dynamics to allow negative eigenvalues. Training efficiency is addressed through selective transfer of 5.69B compatible parameters from Solar Open 1, value-oriented curation of a 20T-token pool into a 10T mixture, and a staged pre-training curriculum followed by agent-focused post-training. Post-training builds twelve specialists across conversational, coding, and officework scenarios and consolidates them into one model using multi-teacher on-policy distillation.
Novelty
The paper’s distinctive contribution is the combination of a 1M-context hybrid-attention MoE architecture with cross-generation selective weight transfer across a substantial architectural change. It also presents a specialist-to-generalist consolidation scheme via multi-teacher on-policy distillation and introduces purpose-built Korean officework-agent training and evaluation pipelines.
Results
The report shows that Solar Open 2 leads comparably sized open-weight models on some English benchmarks, including MMLU-Pro, LiveCodeBench, and APEX-Agents, while remaining competitive with strong open baselines on others. On Korean benchmarks, it achieves the highest average score among the compared models, including fast-tier closed APIs. On the in-house Ko-GDPval officework benchmark, it scores 86.8, essentially matching DeepSeek-V4-Pro at far larger scale and substantially improving over Solar Open 1.
Key Points
- Solar Open 2 uses a hybrid softmax/linear attention design with no positional encoding to support a 1M-token context window for long agent trajectories.
- The training recipe combines selective reuse of 5.69B parameters from Solar Open 1, curated high-value pre-training data, and twelve domain specialists merged through multi-teacher on-policy distillation.
- Empirical evaluation indicates strong English benchmark performance and especially strong Korean performance, with notable gains on Korean officework-agent tasks.