SAS: Simulated Attention Score

• URL: http://arxiv.org/abs/2507.07694v1
• Date: Thu, 10 Jul 2025 12:16:16 GMT
• Title: SAS: Simulated Attention Score
• Authors: Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Peihao Wang, Jing Xiong, Liliang Ren, Hao Cheng, Janardhan Kulkarni, Yelong Shen, Atlas Wang, Mac Schwager, Anderson Schneider, Xiaodong Liu, Jianfeng Gao,
• Abstract summary: We introduce Simulated Attention Score (SAS), which maintains a compact model size while simulating a larger number of attention heads and hidden feature dimension per head.<n>Comprehensive experiments on a variety of datasets and tasks demonstrate the effectiveness of the proposed SAS method.
• Score: 75.1409882298863
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The attention mechanism is a core component of the Transformer architecture. Various methods have been developed to compute attention scores, including multi-head attention (MHA), multi-query attention, group-query attention and so on. We further analyze the MHA and observe that its performance improves as the number of attention heads increases, provided the hidden size per head remains sufficiently large. Therefore, increasing both the head count and hidden size per head with minimal parameter overhead can lead to significant performance gains at a low cost. Motivated by this insight, we introduce Simulated Attention Score (SAS), which maintains a compact model size while simulating a larger number of attention heads and hidden feature dimension per head. This is achieved by projecting a low-dimensional head representation into a higher-dimensional space, effectively increasing attention capacity without increasing parameter count. Beyond the head representations, we further extend the simulation approach to feature dimension of the key and query embeddings, enhancing expressiveness by mimicking the behavior of a larger model while preserving the original model size. To control the parameter cost, we also propose Parameter-Efficient Attention Aggregation (PEAA). Comprehensive experiments on a variety of datasets and tasks demonstrate the effectiveness of the proposed SAS method, achieving significant improvements over different attention variants.

Related papers

• Quantifying Memory Utilization with Effective State-Size [73.52115209375343]
  We develop a measure of textitmemory utilization'<n>This metric is tailored to the fundamental class of systems with textitinput-invariant and textitinput-varying linear operators
  arXiv  Detail & Related papers  (2025-04-28T08:12:30Z)
• Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads [4.797407445026818]
  KV cache is a widely used technique for large language models (LLMs) inference.<n>Previous studies have reduced the size of KV cache by either removing the same number of unimportant tokens for all attention heads or by allocating differentiated KV cache budgets for pre-identified attention heads.<n>We propose Task-KV, a method that leverages the semantic differentiation of attention heads to allocate differentiated KV cache budgets across various tasks.
  arXiv  Detail & Related papers  (2025-01-25T07:28:13Z)
• Mixture of Hidden-Dimensions Transformer [50.40325486463241]
  We study hidden dimension sparsity and observe that trained Transformers utilize only a small fraction of token dimensions.<n>We propose MoHD (Mixture of Hidden Dimensions), a sparse conditional activation architecture.<n>It achieves 1.7% higher performance with 50% fewer activation parameters and 3.7% higher performance with a 3x parameter expansion at constant activation cost.
  arXiv  Detail & Related papers  (2024-12-07T13:15:22Z)
• MoH: Multi-Head Attention as Mixture-of-Head Attention [63.67734699877724]
  We upgrade the multi-head attention mechanism, the core of the Transformer model, to improve efficiency while maintaining or surpassing the previous accuracy level.<n>We propose Mixture-of-Head attention (MoH), a new architecture that treats attention heads as experts in the Mixture-of-Experts mechanism.<n>MoH has two significant advantages: First, MoH enables each token to select the appropriate attention heads, enhancing inference efficiency without compromising accuracy or increasing the number of parameters.
  arXiv  Detail & Related papers  (2024-10-15T17:59:44Z)
• Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention [42.92397219764559]
  We propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE) We empirically demonstrate that our MHE attention is substantially more memory efficient compared to alternative attention mechanisms.
  arXiv  Detail & Related papers  (2023-10-11T21:38:40Z)
• Understanding Self-attention Mechanism via Dynamical System Perspective [58.024376086269015]
  Self-attention mechanism (SAM) is widely used in various fields of artificial intelligence. We show that intrinsic stiffness phenomenon (SP) in the high-precision solution of ordinary differential equations (ODEs) also widely exists in high-performance neural networks (NN) We show that the SAM is also a stiffness-aware step size adaptor that can enhance the model's representational ability to measure intrinsic SP.
  arXiv  Detail & Related papers  (2023-08-19T08:17:41Z)
• Finding the Pillars of Strength for Multi-Head Attention [35.556186723898485]
  Recent studies have revealed some issues of Multi-Head Attention (MHA) We propose Grouped Head Attention, trained with a self-supervised group constraint that group attention heads. We additionally propose a Voting-to-Stay procedure to remove redundant heads, thus achieving a transformer with lighter weights.
  arXiv  Detail & Related papers  (2023-05-22T03:44:44Z)
• A Dynamic Head Importance Computation Mechanism for Neural Machine Translation [22.784419165117512]
  Multiple parallel attention mechanisms that use multiple attention heads facilitate greater performance of the Transformer model for various applications. In this work, we focus on designing a Dynamic Head Importance Computation Mechanism (DHICM) to dynamically calculate the importance of a head with respect to the input. We add an extra loss function to prevent the model from assigning same score to all heads, to identify more important heads and improvise performance.
  arXiv  Detail & Related papers  (2021-08-03T09:16:55Z)
• Low-Rank Bottleneck in Multi-head Attention Models [74.83235382203604]
  We argue that the scaling between the number of heads and the size of each head in the current architecture gives rise to a low-rank bottleneck in attention heads. We propose to set the head size of an attention unit to input sequence length, and independent of the number of heads, resulting in multi-head attention layers with provably more expressive power.
  arXiv  Detail & Related papers  (2020-02-17T16:16:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.