MiniMax Sparse Attention

• URL: http://arxiv.org/abs/2606.13392v2
• Date: Fri, 12 Jun 2026 09:42:25 GMT
• Title: MiniMax Sparse Attention
• Abstract summary: MiniMax Sparse Attention (MSA) is a blockwise sparse attention built upon Grouped Query Attention (GQA)<n>A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group.<n>The Main Branch then performs exact block-sparse attention over only the selected blocks.
• Score: 16.750980846187343
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: https://github.com/MiniMax-AI/MSA. A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3.

Related papers

• PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs [0.8460698440162889]
  We present PersistentKV, a block-table decode attention engine and page-aware scheduling study.<n>Per PersistentKV maps work by KV-head group, executes directly over native page tables, and adds a compact workqueue schedule.<n>We show that five held-out seeds improve mean wall decode-token throughput by 1.04x to 1.08x on bimodal, uniform, and Zipf-like workloads.
  arXiv  Detail & Related papers  (2026-06-25T06:56:43Z)
• dMoE: dLLMs with Learnable Block Experts [71.572316901001]
  We propose dMoE, a simple yet effective block-level MoE framework.<n>We show that dMoE substantially reduces the number of uniquely activated experts during inference.<n>It also reduces memory usage by 76.64% to 79.84% and achieves 1.14$times$ to 1.66$times$ end-to-end latency speedup.
  arXiv  Detail & Related papers  (2026-05-29T06:03:50Z)
• The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence [161.9185072343856]
  MiniMax-M2 is a family of Mixture-of-Experts language models.<n>The flagship M2 contains 229.9B total parameters with only 9.8B activated per token.
  arXiv  Detail & Related papers  (2026-05-26T03:16:11Z)
• Measuring Maximum Activations in Open Large Language Models [60.3514350516308]
  We measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.<n>We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size.
  arXiv  Detail & Related papers  (2026-05-15T03:31:51Z)
• GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding [5.930949606554186]
  Group-Query Latent Attention (GQLA) is a minimal modification of Multi-head Latent Attention (MLA)<n>Group-Query Latent Attention (GQLA) is a minimal modification of Multi-head Latent Attention (MLA)
  arXiv  Detail & Related papers  (2026-05-14T15:50:01Z)
• BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding [28.11907989174509]
  We introduce BLASST, a drop-in sparse attention method that dynamically prunes the attention matrix without any pre-computation or proxy scores.<n>Our method uses a fixed threshold and existing information from online softmax to identify negligible attention scores.<n>We develop an automated calibration procedure that reveals a simple inverse relationship between optimal threshold and context length.
  arXiv  Detail & Related papers  (2025-12-12T23:30:43Z)
• xLLM Technical Report [57.13120905321185]
  We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework.<n>xLLM builds a novel decoupled service-engine architecture.<n>xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources.
  arXiv  Detail & Related papers  (2025-10-16T13:53:47Z)
• Replacing Softmax Similarity with a Sharpened Angular Similarity: Theory and Practice of Scaling To Billion-Context Attention [21.00061869921723]
  We introduce RACE Attention, a kernel-inspired alternative to Softmax Attention.<n> RACE Attention replaces the exponential kernel with a sharpened angular (cosine) similarity.<n>In a controlled scale test, it processes up to 12 million tokens during a single forward-backward pass on an NVIDIA GH200 GPU.
  arXiv  Detail & Related papers  (2025-10-05T02:57:40Z)
• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention [90.68287470551311]
  MiniMax-M1 is an open-weight, large-scale hybrid-attention reasoning model.<n>It supports a context length of 1 million tokens, 8x the context size of DeepSeek R1.<n>MiniMax-M1 is trained using large-scale reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-16T15:08:02Z)
• MiniMax-01: Scaling Foundation Models with Lightning Attention [59.38940023647236]
  MiniMax-Text-01 and MiniMax-VL-01 offer superior capabilities in processing longer contexts.<n>MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost.<n>Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens.
  arXiv  Detail & Related papers  (2025-01-14T18:50:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.