MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention

• URL: http://arxiv.org/abs/2506.13585v1
• Date: Mon, 16 Jun 2025 15:08:02 GMT
• Title: MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
• Abstract summary: MiniMax-M1 is an open-weight, large-scale hybrid-attention reasoning model.<n>It supports a context length of 1 million tokens, 8x the context size of DeepSeek R1.<n>MiniMax-M1 is trained using large-scale reinforcement learning.
• Score: 90.68287470551311
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce MiniMax-M1, the world's first open-weight, large-scale hybrid-attention reasoning model. MiniMax-M1 is powered by a hybrid Mixture-of-Experts (MoE) architecture combined with a lightning attention mechanism. The model is developed based on our previous MiniMax-Text-01 model, which contains a total of 456 billion parameters with 45.9 billion parameters activated per token. The M1 model natively supports a context length of 1 million tokens, 8x the context size of DeepSeek R1. Furthermore, the lightning attention mechanism in MiniMax-M1 enables efficient scaling of test-time compute. These properties make M1 particularly suitable for complex tasks that require processing long inputs and thinking extensively. MiniMax-M1 is trained using large-scale reinforcement learning (RL) on diverse problems including sandbox-based, real-world software engineering environments. In addition to M1's inherent efficiency advantage for RL training, we propose CISPO, a novel RL algorithm to further enhance RL efficiency. CISPO clips importance sampling weights rather than token updates, outperforming other competitive RL variants. Combining hybrid-attention and CISPO enables MiniMax-M1's full RL training on 512 H800 GPUs to complete in only three weeks, with a rental cost of just $534,700. We release two versions of MiniMax-M1 models with 40K and 80K thinking budgets respectively, where the 40K model represents an intermediate phase of the 80K training. Experiments on standard benchmarks show that our models are comparable or superior to strong open-weight models such as the original DeepSeek-R1 and Qwen3-235B, with particular strengths in complex software engineering, tool utilization, and long-context tasks. We publicly release MiniMax-M1 at https://github.com/MiniMax-AI/MiniMax-M1.

Related papers

• Opt.Gear Technical Report [3.573729952898132]
  OptGear is a foundation model designed for efficient on-device deployment, real-tim inference, and strong task capability.<n>It includes a dense model (1M, 270M, and 1B) with a context length of 64K.<n>OptGear-1M is the first generative language model to achieve 20 TPS with W4A32 quantization on the ARM Cortex-M7 CPU.
  arXiv  Detail & Related papers  (2026-08-02T06:43:25Z)
• MiniMax Sparse Attention [16.750980846187343]
  MiniMax Sparse Attention (MSA) is a blockwise sparse attention built upon Grouped Query Attention (GQA)<n>A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group.<n>The Main Branch then performs exact block-sparse attention over only the selected blocks.
  arXiv  Detail & Related papers  (2026-06-11T14:23:41Z)
• The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence [161.9185072343856]
  MiniMax-M2 is a family of Mixture-of-Experts language models.<n>The flagship M2 contains 229.9B total parameters with only 9.8B activated per token.
  arXiv  Detail & Related papers  (2026-05-26T03:16:11Z)
• MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling [80.48332380100915]
  MiniCPM-SALA is a hybrid model that integrates the high-fidelity long-context modeling of sparse attention with the global efficiency of linear attention.<n>On a single NVIDIA A6000D GPU, the model achieves up to 3.5x the inference speed of the full-attention model at the sequence length of 256K tokens.
  arXiv  Detail & Related papers  (2026-02-12T09:37:05Z)
• MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization [74.04867639197445]
  MiroMind-M1 is a set of fully open-source RLMs built on the Qwen-2.5-based benchmarks.<n>Our models are trained in two stages: SFT on a carefully curated corpus of 719K math-reasoning problems with verified CoT trajectories, followed by RLVR on 62K challenging and verifiable problems.
  arXiv  Detail & Related papers  (2025-07-19T16:21:23Z)
• MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning [24.59797320737557]
  We introduce MT-R1-Zero, the first open-source adaptation of the R1-Zero RL framework for machine translation (MT)<n>On the WMT 24 English-Chinese benchmark, our MT-R1-Zero-3B-Mix achieves competitive performance, surpassing TowerInstruct-7B-v0.2 by an average of 1.26 points.<n>Our work exhibits strong generalization capabilities on out-of-distribution MT tasks, robustly supporting multilingual and low-resource settings.
  arXiv  Detail & Related papers  (2025-04-14T12:14:18Z)
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs [195.24565517943802]
  We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models.<n>Phi-4-Mini is a 3.8-billion- parameter language model trained on high-quality web and synthetic data.<n>Phi-4-Multimodal is a multimodal model that integrates text, vision, and speech/audio input modalities into a single model.
  arXiv  Detail & Related papers  (2025-03-03T17:05:52Z)
• MiniMax-01: Scaling Foundation Models with Lightning Attention [59.38940023647236]
  MiniMax-Text-01 and MiniMax-VL-01 offer superior capabilities in processing longer contexts.<n>MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost.<n>Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens.
  arXiv  Detail & Related papers  (2025-01-14T18:50:05Z)
• Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient [9.519619751861333]
  We propose a state space model (SSM)-based world model, Drama, that achieves $O(n)$ memory and computational complexity.<n>We also introduce a novel sampling method to mitigate the suboptimality caused by an incorrect world model in the early training stages.<n>Drama is accessible and trainable on off-the-shelf hardware, such as a standard laptop.
  arXiv  Detail & Related papers  (2024-10-11T15:10:40Z)
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies [85.57899012821211]
  Small Language Models (SLMs) are a resource-efficient alternative to Large Language Models (LLMs) We introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter variants. We also introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE and MiniCPM-128K.
  arXiv  Detail & Related papers  (2024-04-09T15:36:50Z)
• ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters [14.029865087214436]
  Self-attention mechanism distinguishes transformer-based large language models (LLMs) apart from convolutional and recurrent neural networks. achieving real-time LLM inference on silicon remains challenging due to the extensive use of Softmax in self-attention. We propose Constant Softmax (ConSmax), a software- hardware co-design that serves as an efficient alternative to Softmax.
  arXiv  Detail & Related papers  (2024-01-31T17:52:52Z)
• CPM-2: Large-scale Cost-effective Pre-trained Language Models [71.59893315671997]
  We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference. We introduce knowledge inheritance to accelerate the pre-training process by exploiting existing PLMs instead of training models from scratch. We implement a new inference toolkit, namely InfMoE, for using large-scale PLMs with limited computational resources.
  arXiv  Detail & Related papers  (2021-06-20T15:43:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.