Motif 3: Technical Report

• URL: http://arxiv.org/abs/2608.09119v1
• Date: Mon, 10 Aug 2026 04:53:05 GMT
• Title: Motif 3: Technical Report
• Abstract summary: We introduce a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token.<n>MoE layer contains 384 routed experts, with eight selected per token.<n>We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora.
• Score: 26.937626894426916
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token. Each sparse MoE layer contains 384 routed experts, with eight selected per token. This fine-grained sparsity provides substantial expert capacity while limiting computation. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. Expert-balancing and numerical-stabilization techniques support stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.

Related papers

• Maestro: Reinforcement Learning to Orchestrate Hierarchical Model-Skill Ensembles [32.54156025863882]
  We present Maestro, a Reinforcement Learning-driven orchestration framework.<n>Maestro reframes heterogeneous multimodal tasks as a sequential decision-making process over a hierarchical model-skill registry.<n>We evaluate Maestro across ten representative multimodal benchmarks spanning mathematical reasoning, chart understanding, high-resolution perception, and domain-specific analysis.
  arXiv  Detail & Related papers  (2026-05-21T08:47:49Z)
• InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing [87.5130783098133]
  InternVL-U is a lightweight 4B- parameter UMM that democratizes multimodal understanding and reasoning capabilities.<n>It integrates a state-of-the-art Multimodal Large Language Model (MLLM) with a specialized MMDiT-based visual generation head.<n>It consistently outperforms unified baseline models with over 3x larger scales such as BAGEL (14B) on various generation and editing tasks.
  arXiv  Detail & Related papers  (2026-03-10T16:38:33Z)
• Scalable Training of Mixture-of-Experts Models with Megatron Core [26.9162079065285]
  Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models.<n>Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation.<n>We address these challenges for MoE training through integrated optimizations spanning memory (fine-dimensional recomputation, offloading), communication, and computation.
  arXiv  Detail & Related papers  (2026-03-08T15:42:43Z)
• DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing [67.77471070868852]
  DeepGen 1.0 is a lightweight 5B unified model for image generation and editing.<n>It is trained on only 50M samples, surpassing the 80B HunyuanImage by 28% on WISE and the 27B Qwen-Image-Edit by 37% on UniREditBench.<n>By open-sourcing our training code, weights, and datasets, we provide an efficient, high-performance alternative to democratize unified multimodal research.
  arXiv  Detail & Related papers  (2026-02-12T17:44:24Z)
• Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models [78.73992315826035]
  We introduce Youtu-LLM, a lightweight language model that harmonizes high computational efficiency with native agentic intelligence.<n>Youtu-LLM is pre-trained from scratch to systematically cultivate reasoning and planning capabilities.
  arXiv  Detail & Related papers  (2025-12-31T04:25:11Z)
• AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent [80.83250816918861]
  Large Reasoning Models (LRMs) like o3 and DeepSeek-R1 have achieved remarkable progress in natural language reasoning with long chain-of-thought.<n>However, they remain computationally inefficient and struggle with accuracy when solving problems requiring complex mathematical operations.<n>We present AgentMath, an agent framework that seamlessly integrates language models' reasoning capabilities with code interpreters' computational precision.
  arXiv  Detail & Related papers  (2025-12-23T19:57:49Z)
• Motif 2 12.7B technical report [8.084150960631142]
  We introduce Motif-2-12.7B, a new open-weight foundation model that pushes the efficiency frontier of large language models.<n>Model is pre-trained on 5.5 trillion tokens spanning diverse linguistic, mathematical, scientific, and programming domains.<n>Post-training employs a three-stage supervised fine-tuning pipeline that enhances general instruction adherence, compositional understanding, and linguistic precision.
  arXiv  Detail & Related papers  (2025-11-07T10:32:16Z)
• Multi-Head Mixture-of-Experts [100.60556163597946]
  We propose Multi-Head Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each token into multiple sub-tokens. MH-MoE is straightforward to implement and decouples from other SMoE optimization methods, making it easy to integrate with other SMoE models for enhanced performance.
  arXiv  Detail & Related papers  (2024-04-23T13:47:09Z)
• Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models [93.92762966380793]
  Large language models (LLMs) strive to achieve high performance across all three domains simultaneously. In this paper, we propose to fuse models that are already highly-specialized directly. The proposed fusing framework, UltraFuser, consists of three distinct specialists that are already sufficiently trained on language, coding, and mathematics.
  arXiv  Detail & Related papers  (2024-03-13T06:18:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.