LoGo: Token-Level Dynamic Local-Global Attention

• URL: http://arxiv.org/abs/2608.29539v1
• Date: Sun, 30 Aug 2026 04:05:56 GMT
• Title: LoGo: Token-Level Dynamic Local-Global Attention
• Abstract summary: We propose LoGo, a token-level dynamic local-global attention mechanism.<n>LoGo uses attention span as a direct proxy for attention budget allocation.<n>We show that LoGo improves over the full-attention Transformer and matched-budget static local-global hybrids.
• Score: 27.98535137486496
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As context lengths scale, attention increasingly becomes a primary computational bottleneck in large language models. Standard Transformers remain powerful but computationally inefficient, as they allocate the same attention budget to every token regardless of its contextual demand. Existing local-global hybrids provide a more efficient alternative by mixing restricted- and full-context attention, but they typically allocate span statically across layers or heads. To address these limitations, we propose LoGo, a token-level dynamic local-global attention mechanism that uses attention span as a direct proxy for attention budget allocation. Each LoGo layer contains coupled local and global branches: all tokens receive efficient local attention over a restricted context window, while a learned gate activates global attention with full-context access only for tokens requiring long-range information. A threshold-based budget controller maintains a target global ratio without auxiliary losses, and a progressive masking schedule stabilizes training before sparse routing takes effect. We further implement query-sparse Triton kernels that convert reduced global-attention computation into practical speedups. Extensive experiments validate LoGo's effectiveness, showing that it preserves the scaling behavior of full-attention Transformers across model sizes. In controlled comparisons, LoGo improves over the full-attention Transformer and matched-budget static local-global hybrids, with clear gains on long-range retrieval. Analysis further shows that LoGo learns interpretable span allocation patterns. These results suggest that learned token-level span allocation is an effective and scalable way to improve the long-context performance-compute trade-off.

Related papers

• Characterizing the Expressivity of Local Attention in Transformers [54.7971926111047]
  The transformer is the most popular neural architecture for language modeling.<n>We show that fixed-precision transformers with global attention correspond to a fragment of linear temporal logic containing a single past operator.<n>We additionally prove that adding local attention introduces a second temporal operator, strictly enlarging the class of recognizable regular languages.
  arXiv  Detail & Related papers  (2026-05-01T16:30:52Z)
• Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers [34.75985836977073]
  We propose a novel hybrid transformer that enables dynamic and fine-grained routing between full attention and sliding window attention.<n>SwiAttn routes dynamically the computation to either a full-attention branch for global information aggregation or a sliding-window branch for efficient local pattern matching.<n>Experiments are conducted on twenty-three benchmark datasets across both regular (4K) and long (32K) context lengths.
  arXiv  Detail & Related papers  (2026-03-27T13:04:29Z)
• Efficiency Follows Global-Local Decoupling [62.05489838893081]
  We introduce ConvNeur, a two-branch architecture in which a lightweight neural memory branch aggregates global context on a compact set of tokens.<n>A learned gate lets global cues modulate local features without entangling their objectives.<n>On standard classification, detection, and segmentation benchmarks, ConvNeur matches or surpasses comparable alternatives at similar or lower compute.
  arXiv  Detail & Related papers  (2026-03-20T02:20:16Z)
• Krause Synchronization Transformers [63.8469912831803]
  Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer.<n>We introduce Krause Attention, a principled attention mechanism inspired by bounded-confidence consensus dynamics.
  arXiv  Detail & Related papers  (2026-02-12T03:47:53Z)
• AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity [9.63873831179673]
  Large Language Models (LLMs) with extended context lengths face significant computational challenges during the pre-filling phase.<n>We propose textbfAnchorAttention, a difference-aware, dynamic sparse attention mechanism that efficiently identifies critical attention regions.<n>With its finer-grained sparsity strategy, textbfAnchorAttention achieves higher sparsity rates at the same recall level, significantly reducing computation time.
  arXiv  Detail & Related papers  (2025-05-29T14:59:06Z)
• Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation [1.0285749562751982]
  Backpropagation underpins modern deep learning, yet its reliance on global synchronization limits scalability and incurs high memory costs.<n>In contrast, fully local learning rules are more efficient but often struggle to maintain the cross-layer coordination needed for coherent global learning.<n>We introduce Layer-wise Learning (SLL), a layer-wise training algorithm that decomposes the global objective into coordinated layer-local updates.
  arXiv  Detail & Related papers  (2025-05-08T12:32:29Z)
• SG-Former: Self-guided Transformer with Evolving Token Reallocation [89.9363449724261]
  We propose a novel model, termed as Self-guided Transformer, towards effective global self-attention with adaptive fine granularity. We assign more tokens to the salient regions for achieving fine-grained attention, while allocating fewer tokens to the minor regions in exchange for efficiency and global receptive fields. The proposed SG-Former achieves superior performance superior to state of the art: our base size model achieves textbf84.7% Top-1 accuracy on ImageNet-1K, textbf51.2mAP BBAP on CoCo, textbf52.7mIoU
  arXiv  Detail & Related papers  (2023-08-23T15:52:45Z)
• Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation [87.03299519917019]
  We propose a dual-scale graph transformer (DUET) for joint long-term action planning and fine-grained cross-modal understanding. We build a topological map on-the-fly to enable efficient exploration in global action space. The proposed approach, DUET, significantly outperforms state-of-the-art methods on goal-oriented vision-and-language navigation benchmarks.
  arXiv  Detail & Related papers  (2022-02-23T19:06:53Z)
• Local-to-Global Self-Attention in Vision Transformers [130.0369761612812]
  Transformers have demonstrated great potential in computer vision tasks. Some recent Transformer models adopt a hierarchical design, where self-attentions are only computed within local windows. This design significantly improves the efficiency but lacks global feature reasoning in early stages. In this work, we design a multi-path structure of the Transformer, which enables local-to-global reasoning at multiple granularities in each stage.
  arXiv  Detail & Related papers  (2021-07-10T02:34:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.