Morphing into Hybrid Attention Models

• URL: http://arxiv.org/abs/2606.30562v1
• Date: Mon, 29 Jun 2026 17:02:34 GMT
• Title: Morphing into Hybrid Attention Models
• Abstract summary: Hybrid attention models improve long-context efficiency by retaining only a subset of full-attention layers and replacing the remaining layers with linear attention.<n>Existing hybrid layer selection methods typically rely on fixed placement patterns or layerwise scoring, implicitly treating layer importance as isolated and overlooking interdependent layer effect under a global hybrid configuration.<n>We propose FlashMorph, an effective, efficient and scalable layer selection method for Transformer-to-hybrid conversion.
• Score: 52.02140767770562
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hybrid attention models improve long-context efficiency by retaining only a subset of full-attention layers and replacing the remaining layers with linear attention. However, the effectiveness of Transformer-to-hybrid conversion critically depends on which layers preserve full attention. Existing hybrid layer selection methods typically rely on heuristic strategies such as fixed placement patterns or layerwise scoring, implicitly treating layer importance as isolated and overlooking the interdependent layer effect under a global hybrid configuration. In this work, we formulate hybrid layer selection as a budget-constrained subset optimization problem. We further propose FlashMorph (Fast LAyer Selection for Hybrid MORPHing), an effective, efficient and scalable layer selection method for Transformer-to-hybrid conversion. FlashMorph first constructs a morphable model by equipping each full-attention layer with a converted linear-attention branch. It then freezes all model weights and jointly optimizes layerwise gates on synthetic long-context retrieval data, with a linearization regularization that encourages the model to rely on linear attention for efficiency. The learned gates are discretized under a preset full-attention budget to instantiate the hybrid architecture, followed by standard logits distillation and long-context finetuning. Extensive experiments show that FlashMorph discovers more effective hybrid configurations, preserves strong long-context recall and general benchmark performance while substantially reducing layer selection cost compared with existing layer selection methods, demonstrating its effectiveness, efficiency, and scalability.

Related papers

• HyCO: A Hybrid Neural Solver for Combinatorial Optimization [19.59560767239393]
  We propose a hybrid inference algorithm that constructs a solution prefix with an RL solver and adaptively to a conditional DM switches to complete the remaining decisions.<n> Experimental results on diverse benchmarks demonstrate that HyCO achieves consistent improvements over both backbones.
  arXiv  Detail & Related papers  (2026-09-07T21:11:09Z)
• GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference [6.410595937323586]
  We propose GLIDE, a Guided Layerwise Hybrid Attention that integrates sliding-window softmax attention with linear recurrent aggregation.<n>Unlike uniform hybrid approaches, GLIDE non-uniformly compresses the softmax footprint across the model, reducing aggregate KV cache I/O while preserving expressive power where most vital.<n> Empirical evaluations demonstrate the GLIDE achieves superior performance-efficiency tradeoffs, reducing end-to-end latency for long-context generation without compromising quality.
  arXiv  Detail & Related papers  (2026-06-26T17:48:13Z)
• LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs [3.80555579179805]
  LayerBoost reduces inference latency and improves throughput by up to 68% at high.<n>It matches base model performance on several benchmarks, exhibits only minor degradations on others, and significantly outperforms state-of-the-art attention linearization methods.
  arXiv  Detail & Related papers  (2026-04-23T20:12:19Z)
• Olmo Hybrid: From Theory to Practice and Back [112.70006263561073]
  We show that hybrid models do not merely inherit the expressivity of transformers and linear RNNs, but can express tasks beyond both.<n>We show that Olmo Hybrid outperforms Olmo 3 across standard pretraining and mid-training evaluations.<n>Our results suggest that hybrid models mixing attention and recurrent layers are a powerful extension to the language modeling paradigm.
  arXiv  Detail & Related papers  (2026-04-03T20:36:34Z)
• MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling [80.48332380100915]
  MiniCPM-SALA is a hybrid model that integrates the high-fidelity long-context modeling of sparse attention with the global efficiency of linear attention.<n>On a single NVIDIA A6000D GPU, the model achieves up to 3.5x the inference speed of the full-attention model at the sequence length of 256K tokens.
  arXiv  Detail & Related papers  (2026-02-12T09:37:05Z)
• Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction [3.9660062354591754]
  Transformer architectures deliver state-of-the-art accuracy via dense full-attention, but their quadratic time and memory complexity limits practical deployment.<n> Linear attention mechanisms offer linear or near-linear scaling yet often incur performance degradation.<n>We introduce a greedy layer replacement strategy that iteratively substitutes full attention blocks with linear ones while monitoring validation performance on the target task.<n>This yields a task-specific hybrid model in a single efficient pass, without costly re-training or neural architecture search, and can be applied to any pretrained full-attention backbone for diverse downstream tasks.
  arXiv  Detail & Related papers  (2026-01-16T02:01:40Z)
• Distilling to Hybrid Attention Models via KL-Guided Layer Selection [66.06591032073744]
  This paper describes a simple and efficient recipe for layer selection that uses layer importance scores derived from a small amount of training on generic text data.<n>We find that this approach is more effective than existing approaches for layer selection, including approaches that uniformly interleave linear attentions based on a fixed ratio.
  arXiv  Detail & Related papers  (2025-12-23T18:12:22Z)
• Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling [59.84975924845338]
  We analyze hybrid architectures through the lens of memory utilization and overall performance.<n> sequential hybrids perform better on shorter contexts, whereas parallel hybrids are more effective for longer contexts.<n>We introduce a data-centric approach of continually training on datasets augmented with paraphrases, which further enhances recall while preserving other capabilities.
  arXiv  Detail & Related papers  (2025-10-30T18:19:52Z)
• Native Hybrid Attention for Efficient Sequence Modeling [12.306252523159197]
  Native Hybrid Attention (NHA) is a novel hybrid architecture of linear and full attention.<n>A single textttsoftmax attention operation is applied over all keys and values, enabling per-token and per-head context-dependent weighting.<n> Experimental results show that NHA surpasses Transformers on recall-intensive and commonsense reasoning tasks.
  arXiv  Detail & Related papers  (2025-10-08T13:44:57Z)
• Layer-wise Linear Mode Connectivity [52.6945036534469]
  Averaging neural network parameters is an intuitive method for the knowledge of two independent models. It is most prominently used in federated learning. We analyse the performance of the models that result from averaging single, or groups.
  arXiv  Detail & Related papers  (2023-07-13T09:39:10Z)
• Layer-adaptive sparsity for the Magnitude-based Pruning [88.37510230946478]
  We propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score. LAMP consistently outperforms popular existing schemes for layerwise sparsity selection.
  arXiv  Detail & Related papers  (2020-10-15T09:14:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.