Characterizing the Expressivity of Local Attention in Transformers

• URL: http://arxiv.org/abs/2605.00768v1
• Date: Fri, 01 May 2026 16:30:52 GMT
• Title: Characterizing the Expressivity of Local Attention in Transformers
• Abstract summary: The transformer is the most popular neural architecture for language modeling.<n>We show that fixed-precision transformers with global attention correspond to a fragment of linear temporal logic containing a single past operator.<n>We additionally prove that adding local attention introduces a second temporal operator, strictly enlarging the class of recognizable regular languages.
• Score: 54.7971926111047
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The transformer is the most popular neural architecture for language modeling. The cornerstone of the transformer is its global attention mechanism, which lets the model aggregate information from all preceding tokens before generating the next token. One common variant of attention is called local attention, which restricts each token to aggregating information from a bounded window of predecessors, reducing the quadratic cost of global attention to linear. Although this restriction is usually motivated by efficiency, it has also been found to improve model quality, a phenomenon that has so far lacked a satisfactory explanation. We provide a formal account of this phenomenon in terms of recognizer expressivity. It has been shown that fixed-precision transformers with global attention correspond to a fragment of linear temporal logic containing a single past operator. We additionally prove that adding local attention introduces a second temporal operator, strictly enlarging the class of recognizable regular languages. Moreover, global and local attention are expressively complementary: neither subsumes the other, and combining them yields the richest fragment. Experiments on formal language recognition and natural language modeling corroborate the theory, showing that hybrid global--local transformers outperform their global-only counterparts.

Related papers

• LoGo: Token-Level Dynamic Local-Global Attention [27.98535137486496]
  We propose LoGo, a token-level dynamic local-global attention mechanism.<n>LoGo uses attention span as a direct proxy for attention budget allocation.<n>We show that LoGo improves over the full-attention Transformer and matched-budget static local-global hybrids.
  arXiv  Detail & Related papers  (2026-08-30T04:05:56Z)
• Efficiency Follows Global-Local Decoupling [62.05489838893081]
  We introduce ConvNeur, a two-branch architecture in which a lightweight neural memory branch aggregates global context on a compact set of tokens.<n>A learned gate lets global cues modulate local features without entangling their objectives.<n>On standard classification, detection, and segmentation benchmarks, ConvNeur matches or surpasses comparable alternatives at similar or lower compute.
  arXiv  Detail & Related papers  (2026-03-20T02:20:16Z)
• Krause Synchronization Transformers [63.8469912831803]
  Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer.<n>We introduce Krause Attention, a principled attention mechanism inspired by bounded-confidence consensus dynamics.
  arXiv  Detail & Related papers  (2026-02-12T03:47:53Z)
• Inceptive Transformers: Enhancing Contextual Representations through Multi-Scale Feature Learning Across Domains and Languages [3.294155819837931]
  transformer models compress information from all tokens in a sequence into a single [/] token to represent global context.<n>This approach risks diluting fine-grained or hierarchical features, leading to information loss in downstream tasks where local patterns are important.<n>We propose an inception-style 1-D convolution module that sits on top of the transformer layer and augments token representations with multi-scale local features.
  arXiv  Detail & Related papers  (2025-05-26T19:59:22Z)
• Unified Local and Global Attention Interaction Modeling for Vision Transformers [1.9571946424055506]
  We present a novel method that extends the self-attention mechanism of a vision transformer (ViT) for more accurate object detection across diverse datasets.<n>ViTs show strong capability for image understanding tasks such as object detection, segmentation, and classification.<n>We introduce two modifications to the traditional self-attention framework; a novel aggressive convolution pooling strategy for local feature mixing, and a new conceptual attention transformation.
  arXiv  Detail & Related papers  (2024-12-25T04:53:19Z)
• Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers [54.4919139401528]
  We show that it is possible to reduce interference by identifying and pruning language-specific parameters. We show that removing identified attention heads from a fixed model improves performance for a target language on both sentence classification and structural prediction.
  arXiv  Detail & Related papers  (2022-10-11T18:11:37Z)
• Pre-Training a Graph Recurrent Network for Language Representation [34.4554387894105]
  We consider a graph recurrent network for language model pre-training, which builds a graph structure for each sequence with local token-level communications. We find that our model can generate more diverse outputs with less contextualized feature redundancy than existing attention-based models.
  arXiv  Detail & Related papers  (2022-09-08T14:12:15Z)
• Local-to-Global Self-Attention in Vision Transformers [130.0369761612812]
  Transformers have demonstrated great potential in computer vision tasks. Some recent Transformer models adopt a hierarchical design, where self-attentions are only computed within local windows. This design significantly improves the efficiency but lacks global feature reasoning in early stages. In this work, we design a multi-path structure of the Transformer, which enables local-to-global reasoning at multiple granularities in each stage.
  arXiv  Detail & Related papers  (2021-07-10T02:34:55Z)
• Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers [78.26411729589526]
  We propose the first method to explain prediction by any Transformer-based architecture. Our method is superior to all existing methods which are adapted from single modality explainability.
  arXiv  Detail & Related papers  (2021-03-29T15:03:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.