Exclusive Self Attention

• URL: http://arxiv.org/abs/2603.09078v1
• Date: Tue, 10 Mar 2026 01:39:31 GMT
• Title: Exclusive Self Attention
• Abstract summary: We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer's sequence modeling performance.<n>XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows.
• Score: 11.351690641271029
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer's sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token's own value vector (thus excluding information of self position), encouraging better context modeling. Evaluated on the standard language modeling task, XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows.

Related papers

• Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models [80.45129499188461]
  We introduce Qwen-Scope, an open-source suite of sparse autoencoders (SAEs) built on the Qwen model family.<n>We show that SAEs can go beyond post-hoc analysis to serve as practical interfaces for model development along four directions.
  arXiv  Detail & Related papers  (2026-05-12T10:01:06Z)
• Automated Attention Pattern Discovery at Scale in Large Language Models [13.589024367220198]
  We study repeated behaviors in large language models by mining completion scenarios in Java code datasets.<n>We introduce the Attention Pattern - Masked Autoencoder (AP-MAE), a vision transformer-based model that efficiently reconstructs masked attention patterns.
  arXiv  Detail & Related papers  (2026-04-04T15:32:25Z)
• RAE-AR: Taming Autoregressive Models with Representation Autoencoders [61.73674018219353]
  We introduce token simplification via distribution normalization to ease modeling difficulty and improve convergence.<n>We enhance prediction by incorporating Gaussian noise injection during training to mitigate exposure bias.<n>This work paves the way for a more unified architecture across visual understanding and generative modeling.
  arXiv  Detail & Related papers  (2026-04-02T02:39:28Z)
• SAS: Simulated Attention Score [75.1409882298863]
  We introduce Simulated Attention Score (SAS), which maintains a compact model size while simulating a larger number of attention heads and hidden feature dimension per head.<n>Comprehensive experiments on a variety of datasets and tasks demonstrate the effectiveness of the proposed SAS method.
  arXiv  Detail & Related papers  (2025-07-10T12:16:16Z)
• SeRpEnt: Selective Resampling for Expressive State Space Models [5.7918134313332414]
  State Space Models (SSMs) have recently enjoyed a rise to prominence in the field of deep learning for sequence modeling.<n>We show how selective time intervals in Mamba act as linear approximators of information.<n>We propose our SeRpEnt architecture, a SSM that further exploits selectivity to compress sequences in an information-aware fashion.
  arXiv  Detail & Related papers  (2025-01-20T20:27:50Z)
• Stanceformer: Target-Aware Transformer for Stance Detection [59.69858080492586]
  Stance Detection involves discerning the stance expressed in a text towards a specific subject or target. Prior works have relied on existing transformer models that lack the capability to prioritize targets effectively. We introduce Stanceformer, a target-aware transformer model that incorporates enhanced attention towards the targets during both training and inference.
  arXiv  Detail & Related papers  (2024-10-09T17:24:28Z)
• BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model [65.92173280096588]
  We address the challenge of image resolution variation for the Segment Anything Model (SAM) SAM, known for its zero-shot generalizability, exhibits a performance degradation when faced with datasets with varying image sizes. We present a bias-mode attention mask that allows each token to prioritize neighboring information.
  arXiv  Detail & Related papers  (2024-01-04T15:34:44Z)
• SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition [49.42625022146008]
  We present the advantages of applying SRU++ in ASR tasks by comparing with Conformer across multiple ASR benchmarks. Specifically, SRU++ can surpass Conformer on long-form speech input with a large margin, based on our analysis.
  arXiv  Detail & Related papers  (2021-10-11T19:23:50Z)
• Scaling Local Self-Attention For Parameter Efficient Visual Backbones [29.396052798583234]
  Self-attention has the promise of improving computer vision systems due to parameter-independent scaling of receptive fields and content-dependent interactions. We develop a new self-attention model family, emphHaloNets, which reach state-of-the-art accuracies on the parameter-limited setting of the ImageNet classification benchmark.
  arXiv  Detail & Related papers  (2021-03-23T17:56:06Z)
• SparseBERT: Rethinking the Importance Analysis in Self-attention [107.68072039537311]
  Transformer-based models are popular for natural language processing (NLP) tasks due to its powerful capacity. Attention map visualization of a pre-trained model is one direct method for understanding self-attention mechanism. We propose a Differentiable Attention Mask (DAM) algorithm, which can be also applied in guidance of SparseBERT design.
  arXiv  Detail & Related papers  (2021-02-25T14:13:44Z)
• Stochastic Attention Head Removal: A simple and effective method for improving Transformer Based ASR Models [40.991809705930955]
  We propose to randomly remove attention heads during training and keep all attention heads at test time, thus the final model is an ensemble of models with different architectures. Our method gives consistent performance gains over strong baselines on the Wall Street Journal, AISHELL, Switchboard and AMI datasets.
  arXiv  Detail & Related papers  (2020-11-08T15:41:03Z)
• SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection [51.376723069962]
  We present a method for accelerating and structuring self-attentions: Sparse Adaptive Connection. In SAC, we regard the input sequence as a graph and attention operations are performed between linked nodes. We show that SAC is competitive with state-of-the-art models while significantly reducing memory cost.
  arXiv  Detail & Related papers  (2020-03-22T07:58:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.