Adaptive Head Budgeting for Efficient Multi-Head Attention

• URL: http://arxiv.org/abs/2604.22583v1
• Date: Fri, 24 Apr 2026 14:15:22 GMT
• Title: Adaptive Head Budgeting for Efficient Multi-Head Attention
• Abstract summary: We introduce BudgetFormer, a Transformer architecture equipped with an adaptive multi-head attention mechanism.<n>Our approach learns, for each input, both a head budget corresponding to the number of attention heads required, and a relevance distribution that selects the most informative heads.<n>Experiments show that our method reduces inference cost in terms of FLOPs and memory, while also achieving performance that can surpass standard full multi-head attention.
• Score: 1.9365949510386082
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers have become the dominant architecture across a wide range of domains, largely due to the effectiveness of multi-head attention in capturing diverse representation subspaces. However, standard multi-head attention activates all heads uniformly for every input, regardless of task requirements or input complexity. In many scenarios, particularly for coarse-grained tasks such as text classification, the relevant information is often global and does not require the full diversity of attention heads. As a consequence, using a fixed number of heads can introduce unnecessary computational cost or lead to suboptimal performance when the allocation does not match the input. To address this limitation, we introduce BudgetFormer, a Transformer architecture equipped with an adaptive multi-head attention mechanism that dynamically allocates computational resources. Our approach learns, for each input, both a head budget corresponding to the number of attention heads required, and a relevance distribution that selects the most informative heads. We also propose a training strategy based on an exploration and exploitation trade-off, allowing the model to discover effective head configurations before converging to efficient usage patterns. Experiments on text classification tasks of varying complexity show that our method reduces inference cost in terms of FLOPs and memory, while also achieving performance that can surpass standard full multi-head attention. These results highlight the potential of adaptive head allocation as a principled approach to improving both efficiency and effectiveness in Transformer models.

Related papers

• Explicit Multi-head Attention for Inter-head Interaction in Large Language Models [70.96854312026319]
  Multi-head Explicit Attention (MEA) is a simple yet effective attention variant that explicitly models cross-head interaction.<n>MEA shows strong robustness in pretraining, which allows the use of larger learning rates that lead to faster convergence.<n>This enables a practical key-value cache compression strategy that reduces KV-cache memory usage by 50% with negligible performance loss.
  arXiv  Detail & Related papers  (2026-01-27T13:45:03Z)
• Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-off [20.259111403684006]
  Existing sparse methods often trade information integrity for computational efficiency.<n>We propose SPAttention, whose core contribution is the introduction of a new paradigm we term Principled Structural Sparsity.<n> SPAttention reorganizes the total attention workload into balanced, non-overlapping distance bands, assigning each head a unique segment.
  arXiv  Detail & Related papers  (2025-11-12T14:48:23Z)
• SAS: Simulated Attention Score [75.1409882298863]
  We introduce Simulated Attention Score (SAS), which maintains a compact model size while simulating a larger number of attention heads and hidden feature dimension per head.<n>Comprehensive experiments on a variety of datasets and tasks demonstrate the effectiveness of the proposed SAS method.
  arXiv  Detail & Related papers  (2025-07-10T12:16:16Z)
• From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning [102.18178065928426]
  Efficient Visual Instruction Fine-Tuning (EVIT) seeks to adapt Multimodal Large Language Models (MLLMs) to downstream tasks with minimal computational overhead.<n>We propose the Dual Low-Rank Adaptation (Dual-LoRA), a holistic-to-local framework that enhances the adapter's capacity to address data conflict.
  arXiv  Detail & Related papers  (2024-11-19T11:03:09Z)
• Open Domain Generalization with a Single Network by Regularization Exploiting Pre-trained Features [37.518025833882334]
  Open Domain Generalization (ODG) is a challenging task as it deals with distribution shifts and category shifts. Previous work has used multiple source-specific networks, which involve a high cost. This paper proposes a method that can handle ODG using only a single network.
  arXiv  Detail & Related papers  (2023-12-08T16:22:10Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
  Multi-task learning aims to acquire a set of functions that perform well for diverse tasks. In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks. We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
  arXiv  Detail & Related papers  (2022-10-27T16:06:47Z)
• Adaptive Multi-Resolution Attention with Linear Complexity [18.64163036371161]
  We propose a novel structure named Adaptive Multi-Resolution Attention (AdaMRA) for short. We leverage a multi-resolution multi-head attention mechanism, enabling attention heads to capture long-range contextual information in a coarse-to-fine fashion. To facilitate AdaMRA utilization by the scientific community, the code implementation will be made publicly available.
  arXiv  Detail & Related papers  (2021-08-10T23:17:16Z)
• A Dynamic Head Importance Computation Mechanism for Neural Machine Translation [22.784419165117512]
  Multiple parallel attention mechanisms that use multiple attention heads facilitate greater performance of the Transformer model for various applications. In this work, we focus on designing a Dynamic Head Importance Computation Mechanism (DHICM) to dynamically calculate the importance of a head with respect to the input. We add an extra loss function to prevent the model from assigning same score to all heads, to identify more important heads and improvise performance.
  arXiv  Detail & Related papers  (2021-08-03T09:16:55Z)
• Diversity-Based Generalization for Unsupervised Text Classification under Domain Shift [16.522910268114504]
  We propose a novel method for domain adaptation of single-task text classification problems based on a simple but effective idea of diversity-based generalization. Our results demonstrate that machine learning architectures that ensure sufficient diversity can generalize better.
  arXiv  Detail & Related papers  (2020-02-25T15:11:02Z)
• Low-Rank Bottleneck in Multi-head Attention Models [74.83235382203604]
  We argue that the scaling between the number of heads and the size of each head in the current architecture gives rise to a low-rank bottleneck in attention heads. We propose to set the head size of an attention unit to input sequence length, and independent of the number of heads, resulting in multi-head attention layers with provably more expressive power.
  arXiv  Detail & Related papers  (2020-02-17T16:16:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.