MiniMax-01: Scaling Foundation Models with Lightning Attention

• URL: http://arxiv.org/abs/2501.08313v1
• Date: Tue, 14 Jan 2025 18:50:05 GMT
• Title: MiniMax-01: Scaling Foundation Models with Lightning Attention
• Abstract summary: MiniMax-Text-01 and MiniMax-VL-01 offer superior capabilities in processing longer contexts.<n>MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost.<n>Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens.
• Score: 59.38940023647236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI.

Related papers

• Opt.Gear Technical Report [3.573729952898132]
  OptGear is a foundation model designed for efficient on-device deployment, real-tim inference, and strong task capability.<n>It includes a dense model (1M, 270M, and 1B) with a context length of 64K.<n>OptGear-1M is the first generative language model to achieve 20 TPS with W4A32 quantization on the ARM Cortex-M7 CPU.
  arXiv  Detail & Related papers  (2026-08-02T06:43:25Z)
• MiniMax Sparse Attention [16.750980846187343]
  MiniMax Sparse Attention (MSA) is a blockwise sparse attention built upon Grouped Query Attention (GQA)<n>A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group.<n>The Main Branch then performs exact block-sparse attention over only the selected blocks.
  arXiv  Detail & Related papers  (2026-06-11T14:23:41Z)
• Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models [63.96252564303143]
  Embodied-R1.5 is a unified Embodied Foundation Model (EFM) that integrates comprehensive embodied reasoning capabilities.<n>We build a large-scale data system of over 15B tokens, and design a multi-task balanced RL recipe.<n>With only 8B parameters, Embodied-R1.5 SOTA on 16 out of 24 embodied VLM benchmarks, surpassing leading models like Gemini-Robotics-ER-1.5 and GPT-5.4.
  arXiv  Detail & Related papers  (2026-06-09T18:07:50Z)
• MolmoAct2: Action Reasoning Models for Real-world Deployment [67.6315757474802]
  MolmoAct2 is a fully open action reasoning model built for practical deployment.<n>We introduce MolmoER, a VLM backbone specialized for spatial and embodied reasoning.<n>We release three new datasets spanning low-to-medium cost platforms.
  arXiv  Detail & Related papers  (2026-05-04T17:51:21Z)
• RadLite: Multi-Task LoRA Fine-Tuning of Small Language Models for CPU-Deployable Radiology AI [1.5161700234179216]
  Large language models (LLMs) show promise in radiology but their deployment is limited by computational requirements that preclude use in resource-constrained clinical environments.<n>We investigate whether small language models (SLMs) of 3-4 billion parameters can achieve strong multi-task radiology performance through LoRA fine-tuning, enabling deployment on consumer-grade CPUs.<n>Our work demonstrates that small, efficiently fine-tuned models can serve as practical multi-task radiology AI assistants deployable entirely on consumer hardware without GPU requirements.
  arXiv  Detail & Related papers  (2026-05-01T05:39:08Z)
• AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model [40.488271586857884]
  AndesVL is a suite of mobile-side MLLMs with 0.6B to 4B parameters based on Qwen3's LLM and various visual encoders.<n>We introduce a 1+N LoRA architecture alongside a Quantization-Aware LoRA Fine-Tuning framework to facilitate efficient task adaptation and model compression.<n>We achieve a 6.7x peak decoding speedup ratio, up to 30.9% memory reduction, and 1.8 bits-per-weight when deploying AndesVL-4B on MediaTek Dimensity 9500 chips.
  arXiv  Detail & Related papers  (2025-10-13T15:04:38Z)
• LongCat-Flash Technical Report [165.64670448930875]
  LongCat-Flash is a 560-billion- parameter Mixture-of-Experts (MoE) language model.<n>It is designed for both computational efficiency and advanced agentic capabilities.<n>We complete model training on more than 20 trillion tokens within 30 days, while achieving over 100 tokens per second (TPS) for inference at a cost of $0.70 per million output tokens.
  arXiv  Detail & Related papers  (2025-09-01T10:05:45Z)
• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention [90.68287470551311]
  MiniMax-M1 is an open-weight, large-scale hybrid-attention reasoning model.<n>It supports a context length of 1 million tokens, 8x the context size of DeepSeek R1.<n>MiniMax-M1 is trained using large-scale reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-16T15:08:02Z)
• Test-Time Training Done Right [61.8429380523577]
  Test-Time Training (TTT) models context by adapting part of the model's weights (referred to as fast weights) during inference.<n>Existing TTT methods struggled to show effectiveness in handling long-context data.<n>We develop Large Chunk Test-Time Training (LaCT) which improves hardware utilization by orders of magnitude.
  arXiv  Detail & Related papers  (2025-05-29T17:50:34Z)
• OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models [36.0400717590138]
  We present OmniMamba, the first linear-architecture-based multimodal generation model. It generates both text and images through a unified next-token prediction paradigm. It achieves competitive performance with JanusFlow while surpassing Show-o across benchmarks.
  arXiv  Detail & Related papers  (2025-03-11T17:59:46Z)
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs [195.24565517943802]
  We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models. Phi-4-Mini is a 3.8-billion- parameter language model trained on high-quality web and synthetic data. Phi-4-Multimodal is a multimodal model that integrates text, vision, and speech/audio input modalities into a single model.
  arXiv  Detail & Related papers  (2025-03-03T17:05:52Z)
• OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference [9.78611123915888]
  We present OmniVLM, a vision-language model for efficient on-device inference.<n>It reduces visual token sequence length from 729 to 81 tokens, significantly reducing computational overhead.<n>It outperforms existing baselines like nanoLLAVA within a 968M- parameter footprint.
  arXiv  Detail & Related papers  (2024-12-16T06:38:00Z)
• Puzzle: Distillation-Based NAS for Inference-Optimized LLMs [17.72841008597783]
  Large language models (LLMs) have demonstrated remarkable capabilities, but their adoption is limited by high computational costs during inference.<n>We present Puzzle, a framework to accelerate LLM inference on specific hardware while preserving their capabilities.<n>Nemotron-51B stands as the most accurate language model capable of inference on a single GPU with large batch sizes.
  arXiv  Detail & Related papers  (2024-11-28T13:45:42Z)
• Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance [78.48606021719206]
  Mini-InternVL is a series of MLLMs with parameters ranging from 1B to 4B, which achieves 90% of the performance with only 5% of the parameters. We develop a unified adaptation framework for Mini-InternVL, which enables our models to transfer and outperform specialized models in downstream tasks.
  arXiv  Detail & Related papers  (2024-10-21T17:58:20Z)
• 1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data [0.0]
  This paper presents a compute-efficient approach to pre-training a Language Model-the "1.5-Pints"-in only 9 days. Based on MT-Bench (a benchmark that emulates human judgments), 1.5-Pints outperforms Apple's OpenELM and Microsoft's Phi. This is achieved by a carefully curated pre-training dataset of 57 billion tokens, using a mix of automated and manual human review.
  arXiv  Detail & Related papers  (2024-08-07T02:14:52Z)
• Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models [62.4691912312317]
  Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4$times$ compared to dense models without sacrificing performance. We propose a hybrid dense training and sparse inference framework for MoE models (DS-MoE) which achieves strong computation and parameter efficiency.
  arXiv  Detail & Related papers  (2024-04-08T14:39:49Z)
• eP-ALM: Efficient Perceptual Augmentation of Language Models [70.47962271121389]
  We propose to direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning.
  arXiv  Detail & Related papers  (2023-03-20T19:20:34Z)
• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference [17.97893143555333]
  Sparse Mixture-of-Experts (MoE) has been a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation. In this work, we investigate routing strategies at different granularity (token, sentence, task) in MoE models to bypass distillation. Experiments on WMT and a web-scale dataset suggest that task-level routing (task-MoE) enables us to extract smaller, ready-to-deploy sub-networks from large sparse models.
  arXiv  Detail & Related papers  (2021-09-24T20:42:16Z)
• CPM-2: Large-scale Cost-effective Pre-trained Language Models [71.59893315671997]
  We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference. We introduce knowledge inheritance to accelerate the pre-training process by exploiting existing PLMs instead of training models from scratch. We implement a new inference toolkit, namely InfMoE, for using large-scale PLMs with limited computational resources.
  arXiv  Detail & Related papers  (2021-06-20T15:43:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.