More Expressive Attention with Negative Weights

• URL: http://arxiv.org/abs/2411.07176v3
• Date: Thu, 30 Jan 2025 18:17:13 GMT
• Title: More Expressive Attention with Negative Weights
• Authors: Ang Lv, Ruobing Xie, Shuaipeng Li, Jiayi Liao, Xingwu Sun, Zhanhui Kang, Di Wang, Rui Yan,
• Abstract summary: We propose a novel attention mechanism, named Cog Attention, that enables attention weights to be negative for enhanced expressiveness. Our approach suggests a promising research direction for rethinking and breaking the entrenched constraints of traditional softmax attention.
• Score: 36.40344438470477
• License:
• Abstract: We propose a novel attention mechanism, named Cog Attention, that enables attention weights to be negative for enhanced expressiveness, which stems from two key factors: (1) Cog Attention enhances parameter flexibility. For example, unlike traditional softmax attention heads that use a static output-value (OV) matrix to delete or copy inputs that the heads attend to, Cog Attention naturally learns to use the sign of dynamic query-key (QK) inner products to represent these operations. This enables Cog Attention to perform multiple operations simultaneously within a single head. Meanwhile, Cog Attention's OV matrix can focus more on refinement or modification. (2) Cog Attention enhances the model's robustness against representational collapse by preventing the ``over-squashing'' of earlier tokens into later positions. We develop Transformer-like models which use Cog Attention as attention modules, including decoder-only models at various scales for language modeling and U-ViT diffusion models for image generation. Experiments show that models using Cog Attention exhibit superior performance compared to those employing traditional softmax attention modules. Our approach suggests a promising research direction for rethinking and breaking the entrenched constraints of traditional softmax attention, such as the requirement for non-negative weights.

Related papers

• FuXi-$α$: Scaling Recommendation Model with Feature Interaction Enhanced Transformer [81.12174905444229]
  Recent advancements have shown that expanding sequential recommendation models to large-scale recommendation models can be an effective strategy. We propose a new model called FuXi-$alpha$ to address these issues. Our model outperforms existing models, with its performance continuously improving as the model size increases.
  arXiv  Detail & Related papers  (2025-02-05T09:46:54Z)
• Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models [7.80071686970278]
  Traditional Softmax attention suffers from numerical instability and reduced performance as the length of inference tokens increases. This paper addresses these issues by decomposing the Softmax operation into a non-linear transformation and the $l_1$-norm. We create a novel attention mechanism with performance better than conventional Softmax attention across various inference lengths.
  arXiv  Detail & Related papers  (2025-01-23T07:21:08Z)
• Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models [64.67721492968941]
  We propose a Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR) framework. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness. Our method yields a 9.58% enhancement in zero-shot robust accuracy over the current state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-10-29T07:15:09Z)
• A Primal-Dual Framework for Transformers and Neural Networks [52.814467832108875]
  Self-attention is key to the remarkable success of transformers in sequence modeling tasks. We show that the self-attention corresponds to the support vector expansion derived from a support vector regression problem. We propose two new attentions: Batch Normalized Attention (Attention-BN) and Attention with Scaled Head (Attention-SH)
  arXiv  Detail & Related papers  (2024-06-19T19:11:22Z)
• FAST: Factorizable Attention for Speeding up Transformers [1.3637227185793512]
  We present a linearly scaled attention mechanism that maintains the full representation of the attention matrix without compromising on sparsification. Results indicate that our attention mechanism has a robust performance and holds significant promise for diverse applications where self-attention is used.
  arXiv  Detail & Related papers  (2024-02-12T18:59:39Z)
• Revisiting Attention Weights as Explanations from an Information Theoretic Perspective [4.499369811647602]
  We show that attention mechanisms have the potential to function as a shortcut to model explanations when they are carefully combined with other model elements. Our findings indicate that attention mechanisms do have the potential to function as a shortcut to model explanations when they are carefully combined with other model elements.
  arXiv  Detail & Related papers  (2022-10-31T12:53:20Z)
• Causal Attention for Vision-Language Tasks [142.82608295995652]
  We present a novel attention mechanism: Causal Attention (CATT) CATT removes the ever-elusive confounding effect in existing attention-based vision-language models. In particular, we show that CATT has great potential in large-scale pre-training.
  arXiv  Detail & Related papers  (2021-03-05T06:38:25Z)
• SparseBERT: Rethinking the Importance Analysis in Self-attention [107.68072039537311]
  Transformer-based models are popular for natural language processing (NLP) tasks due to its powerful capacity. Attention map visualization of a pre-trained model is one direct method for understanding self-attention mechanism. We propose a Differentiable Attention Mask (DAM) algorithm, which can be also applied in guidance of SparseBERT design.
  arXiv  Detail & Related papers  (2021-02-25T14:13:44Z)
• Gaussian Constrained Attention Network for Scene Text Recognition [16.485898019983797]
  We argue that the existing attention mechanism faces the problem of attention diffusion, in which the model may not focus on a certain character area. We propose a 2D attention-based method integrated with a novel Gaussian Constrained Refinement Module. In this way, the attention weights will be more concentrated and the attention-based recognition network achieves better performance.
  arXiv  Detail & Related papers  (2020-10-19T01:55:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.