Rethinking the Role of Efficient Attention in Hybrid Architectures

• URL: http://arxiv.org/abs/2606.15378v1
• Date: Sat, 13 Jun 2026 16:21:37 GMT
• Title: Rethinking the Role of Efficient Attention in Hybrid Architectures
• Abstract summary: We show how efficient-attention design affects how fast long-context capability emerges.<n>Larger SWA windows can delay the formation of retrieval heads in full-attention layers.<n>Applying NoPE to only the full-attention layers of a small-window SWA hybrid substantially improves long-context performance.
• Score: 40.06011907592891
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern language models increasingly adopt hybrid architectures that combine full attention with efficient attention modules, such as sliding-window attention (SWA) and recurrent sequence mixers. However, how these efficient modules shape model capabilities remains poorly understood. To address this gap, we conduct a systematic analysis across hybrid architectures from three perspectives: scaling behavior, mechanism analysis, and architecture design. First, from a scaling perspective, we find that efficient-attention design primarily affects how fast long-context capability emerges, while different hybrids eventually converge to comparable long-context performance under sufficient training. Second, mechanistically, we show that long-range retrieval is mainly carried by full attention, whereas efficient attention shapes its optimization trajectory. This explains a counter-intuitive phenomenon we call Large-Window Laziness: larger SWA windows can delay the formation of retrieval heads in full-attention layers. Third, guided by this mechanism, we show that applying NoPE to only the full-attention layers of a small-window SWA hybrid substantially improves long-context performance with negligible impact on short-context performance.

Related papers

• MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers [60.12595759596518]
  MATCH is a scalable and efficient framework that augments sparsified attention mechanisms with dynamically integrated in-context information.<n> Empirical results show that MATCH significantly improves the performance of sparse-attention models on both synthetic and real-world natural-language tasks.
  arXiv  Detail & Related papers  (2026-06-29T06:33:37Z)
• Attention Editing: A Versatile Framework for Cross-Architecture Attention Conversion [17.514757379631927]
  We present Attention Editing, a framework for converting already-trained large language models with new attention architectures without re-pretraining from scratch.<n>We instantiate the framework on two different target--MLA and GateSWA, a gated hybrid SWA design, and apply it to Qwen3-8B and Qwen3-30B-A3B.<n>Experiments are conducted on an Ascend 910B clusters, offering a practical training case study on domestic hardware.
  arXiv  Detail & Related papers  (2026-04-07T10:40:16Z)
• MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling [80.48332380100915]
  MiniCPM-SALA is a hybrid model that integrates the high-fidelity long-context modeling of sparse attention with the global efficiency of linear attention.<n>On a single NVIDIA A6000D GPU, the model achieves up to 3.5x the inference speed of the full-attention model at the sequence length of 256K tokens.
  arXiv  Detail & Related papers  (2026-02-12T09:37:05Z)
• Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling [59.84975924845338]
  We analyze hybrid architectures through the lens of memory utilization and overall performance.<n> sequential hybrids perform better on shorter contexts, whereas parallel hybrids are more effective for longer contexts.<n>We introduce a data-centric approach of continually training on datasets augmented with paraphrases, which further enhances recall while preserving other capabilities.
  arXiv  Detail & Related papers  (2025-10-30T18:19:52Z)
• ReGLA: Refining Gated Linear Attention [42.97193398172823]
  Linear attention has been designed to reduce the quadratic space-time complexity that is inherent in standard transformers.<n>We developed a feature mapping function to address some crucial issues that previous suggestions overlooked.<n>We also explored the saturation phenomenon of the gating mechanism and augmented it with a refining module.
  arXiv  Detail & Related papers  (2025-02-03T18:03:13Z)
• Hymba: A Hybrid-head Architecture for Small Language Models [65.94140459055244]
  Hymba is a family of small language models featuring a hybrid-head parallel architecture. We introduce learnable meta tokens that are prepended to prompts, storing critical information. This model is further optimized by incorporating cross-layer key-value sharing and partial sliding window attention.
  arXiv  Detail & Related papers  (2024-11-20T19:51:25Z)
• iiANET: Inception Inspired Attention Hybrid Network for efficient Long-Range Dependency [0.5497663232622965]
  We present iiANET, an efficient hybrid visual backbone designed to improve the modeling of long-range dependen-cies.<n>The core innovation of iiANET is the iiABlock, a unified building block that in-tegrates global r-MHSA (Multi-Head Self-Attention) and convolutional layers in paral-lel.
  arXiv  Detail & Related papers  (2024-07-10T12:39:02Z)
• Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers [58.5711048151424]
  We introduce SPARSEK Attention, a novel sparse attention mechanism designed to overcome computational and memory obstacles. Our approach integrates a scoring network and a differentiable top-k mask operator, SPARSEK, to select a constant number of KV pairs for each query. Experimental results reveal that SPARSEK Attention outperforms previous sparse attention methods.
  arXiv  Detail & Related papers  (2024-06-24T15:55:59Z)
• Faster Attention Is What You Need: A Fast Self-Attention Neural Network Backbone Architecture for the Edge via Double-Condensing Attention Condensers [71.40595908386477]
  We introduce a new faster attention condenser design called double-condensing attention condensers. The resulting backbone (which we name AttendNeXt) achieves significantly higher inference throughput on an embedded ARM processor. These promising results demonstrate that exploring different efficient architecture designs and self-attention mechanisms can lead to interesting new building blocks for TinyML applications.
  arXiv  Detail & Related papers  (2022-08-15T02:47:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.