Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

• URL: http://arxiv.org/abs/2606.20945v2
• Date: Tue, 23 Jun 2026 11:48:31 GMT
• Title: Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
• Abstract summary: Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths.<n>Standard dense attention applies the same set of attention heads to every token regardless of token difficulty or information content.<n>We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA)
• Score: 0.8307668828380428
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.

Related papers

• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration [67.17709877327232]
  Cate KV is a hybrid KV cache method that retains only critical token information for consistent heads.<n>Cate KV reduces memory usage by up to $2.72times$ and accelerates decoding by $2.18times$ in single-sample inputs.
  arXiv  Detail & Related papers  (2026-08-31T06:02:37Z)
• HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment [76.68875424389057]
  Existing methods score each visual token by averaging text-to-visual attention uniformly across all heads.<n> misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues.<n>We propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions.
  arXiv  Detail & Related papers  (2026-08-24T23:55:36Z)
• From Item-Only to Query-Item: Query-Conditioned Generative Search with QGS in Quark [8.408261237886899]
  Generative sequence models have shown strong results in recommendation.<n>Applying them to search ranking is more challenging.<n>We present Query-Conditioned Generative Search (QGS)
  arXiv  Detail & Related papers  (2026-05-25T07:18:51Z)
• Route Experts by Sequence, not by Token [58.92918003265283]
  Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token.<n>The standard TopK routing assigns the same fixed number of experts to all tokens, ignoring their varying complexity.<n>We propose Sequence-level TopK (SeqTopK), a minimal modification that shifts the expert budget from the token level to the sequence level.
  arXiv  Detail & Related papers  (2025-11-09T18:36:07Z)
• Causal Attention with Lookahead Keys [52.63961482746826]
  In standard causal attention, each token's query, key, and value (QKV) are static and encode only preceding context.<n>We introduce CAuSal aTtention with Lookahead kEys (CASTLE), an attention mechanism that continually updates each token's keys as the context unfolds.
  arXiv  Detail & Related papers  (2025-09-09T00:15:23Z)
• CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation [7.119276797399788]
  Increasing key-value (KV) cache size poses critical challenges to memory and execution efficiency.<n>Most KV cache compression methods rely on token eviction using all attention heads in Grouped Query Attention (GQA)-based LLMs.<n>We introduce a layer-adaptive KV cache allocation strategy, which consistently outperforms state-of-the-art approaches under various memory budgets.
  arXiv  Detail & Related papers  (2025-08-04T13:26:16Z)
• Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query [48.52389201779425]
  KV cache memory usage grows substantially with longer text sequences.<n>Existing KV cache eviction methods prune tokens using prefilling-stage attention scores.<n>Lookahead Q-Cache generates low-cost pseudo lookahead queries to better approximate the true decoding-stage queries.
  arXiv  Detail & Related papers  (2025-05-24T10:34:38Z)
• Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention [3.3457276841127315]
  Transformer architecture has revolutionized deep learning through its Self-Attention mechanism. Grouped Query Attention (GQA) addresses this issue by grouping queries and mean-pooling the corresponding key-value heads. We introduce enhancements to GQA, focusing on two novel approaches that deviate from the static nature of grouping.
  arXiv  Detail & Related papers  (2024-08-15T23:34:04Z)
• QCQA: Quality and Capacity-aware grouped Query Attention [5.121164018825873]
  Excessive memory requirements of key and value features (KV-cache) present significant challenges in the autoregressive inference of large language models (LLMs) We propose Quality and Capacity-Aware Grouped Query Attention (QCQA) which identifies optimal query head groupings using an evolutionary algorithm with a computationally efficient and inexpensive fitness function.
  arXiv  Detail & Related papers  (2024-06-08T07:49:55Z)
• Advancing Vision Transformers with Group-Mix Attention [59.585623293856735]
  Group-Mix Attention (GMA) is an advanced replacement for traditional self-attention. GMA simultaneously captures token-to-token, token-to-group, and group-to-group correlations with various group sizes. GroupMixFormer achieves state-of-the-art performance in image classification, object detection, and semantic segmentation.
  arXiv  Detail & Related papers  (2023-11-26T01:25:03Z)
• Self-Calibrated Cross Attention Network for Few-Shot Segmentation [65.20559109791756]
  We design a self-calibrated cross attention (SCCA) block for efficient patch-based attention. SCCA groups the patches from the same query image and the aligned patches from the support image as K&V. In this way, the query BG features are fused with matched BG features in support FG, and thus the aforementioned issues will be mitigated.
  arXiv  Detail & Related papers  (2023-08-18T04:41:50Z)
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints [25.154477500940626]
  We propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute. We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA.
  arXiv  Detail & Related papers  (2023-05-22T17:16:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.