LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs

• URL: http://arxiv.org/abs/2604.22050v1
• Date: Thu, 23 Apr 2026 20:12:19 GMT
• Title: LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs
• Abstract summary: LayerBoost reduces inference latency and improves throughput by up to 68% at high.<n>It matches base model performance on several benchmarks, exhibits only minor degradations on others, and significantly outperforms state-of-the-art attention linearization methods.
• Score: 3.80555579179805
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Transformers are mostly relying on softmax attention, which introduces quadratic complexity with respect to sequence length and remains a major bottleneck for efficient inference. Prior work on linear or hybrid attention typically replaces softmax attention uniformly across all layers, often leading to significant performance degradation or requiring extensive retraining to recover model quality. This work proposes LayerBoost, a layer-aware attention reduction method that selectively modifies the attention mechanism based on the sensitivity of individual transformer layers. It first performs a systematic sensitivity analysis on a pretrained model to identify layers that are critical for maintaining performance. Guided by this analysis, three distinct strategies can be applied: retaining standard softmax attention in highly sensitive layers, replacing it with linear sliding window attention in moderately sensitive layers, and removing attention entirely in layers that exhibit low sensitivity. To recover performance after these architectural modifications, we introduce a lightweight distillation-based healing phase requiring only 10M additional training tokens. LayerBoost reduces inference latency and improves throughput by up to 68% at high concurrency, while maintaining competitive model quality. It matches base model performance on several benchmarks, exhibits only minor degradations on others, and significantly outperforms state-of-the-art attention linearization methods. These efficiency gains make our method particularly well-suited for high-concurrency serving and hardware-constrained deployment scenarios, where inference cost and memory footprint are critical bottlenecks.

Related papers

• Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models [86.02826269126308]
  We propose Trend-aware Pruning, a novel framework that elevates pruning from a local snapshot decision to a temporal trajectory modeling problem.<n>This enables a dynamic rectification mechanism that selectively reactivates "late-blooming" tokens, those initially undervalued but exhibiting rising semantic importance, thereby preventing the loss of critical visual cues.
  arXiv  Detail & Related papers  (2026-07-30T15:07:00Z)
• Morphing into Hybrid Attention Models [52.02140767770562]
  Hybrid attention models improve long-context efficiency by retaining only a subset of full-attention layers and replacing the remaining layers with linear attention.<n>Existing hybrid layer selection methods typically rely on fixed placement patterns or layerwise scoring, implicitly treating layer importance as isolated and overlooking interdependent layer effect under a global hybrid configuration.<n>We propose FlashMorph, an effective, efficient and scalable layer selection method for Transformer-to-hybrid conversion.
  arXiv  Detail & Related papers  (2026-06-29T17:02:34Z)
• GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference [6.410595937323586]
  We propose GLIDE, a Guided Layerwise Hybrid Attention that integrates sliding-window softmax attention with linear recurrent aggregation.<n>Unlike uniform hybrid approaches, GLIDE non-uniformly compresses the softmax footprint across the model, reducing aggregate KV cache I/O while preserving expressive power where most vital.<n> Empirical evaluations demonstrate the GLIDE achieves superior performance-efficiency tradeoffs, reducing end-to-end latency for long-context generation without compromising quality.
  arXiv  Detail & Related papers  (2026-06-26T17:48:13Z)
• RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT [0.014257559724536567]
  Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-attention complexity.<n>We propose a depth-aware token reduction framework that adapts reduction strategies to the layer-wise characteristics of token representations.<n>Our framework provides a training-free template for optimizing vision models by aligning reduction strategies with hierarchical feature evolution.
  arXiv  Detail & Related papers  (2026-06-06T13:13:02Z)
• Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs [32.78500862748985]
  Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes.<n>In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer.<n>Experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning.
  arXiv  Detail & Related papers  (2026-06-03T23:42:09Z)
• Distilling to Hybrid Attention Models via KL-Guided Layer Selection [66.06591032073744]
  This paper describes a simple and efficient recipe for layer selection that uses layer importance scores derived from a small amount of training on generic text data.<n>We find that this approach is more effective than existing approaches for layer selection, including approaches that uniformly interleave linear attentions based on a fixed ratio.
  arXiv  Detail & Related papers  (2025-12-23T18:12:22Z)
• Breaking Complexity Barriers: High-Resolution Image Restoration with Rank Enhanced Linear Attention [54.42902794496325]
  Linear attention, a variant of softmax attention, demonstrates promise in global context modeling.<n>We propose Rank Enhanced Linear Attention (RELA), a simple yet effective method that enriches feature representations by integrating a lightweight depthwise convolution.<n>Building upon RELA, we propose an efficient and effective image restoration Transformer, named LAformer.
  arXiv  Detail & Related papers  (2025-05-22T02:57:23Z)
• Enhancing Layer Attention Efficiency through Pruning Redundant Retrievals [0.0]
  We propose a novel approach to quantify redundancy by leveraging the Kullback-Leibler (KL) divergence between adjacent layers.<n>We also introduce an Enhanced Beta Quantile Mapping (EBQM) method that accurately identifies and skips redundant layers.<n>Our proposed Efficient Layer Attention (ELA) architecture, improves both training efficiency and overall performance, achieving a 30% reduction in training time.
  arXiv  Detail & Related papers  (2025-03-09T06:20:11Z)
• Stabilizing Transformer Training by Preventing Attention Entropy Collapse [56.45313891694746]
  We investigate the training dynamics of Transformers by examining the evolution of the attention layers. We show that $sigma$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. We conduct experiments with $sigma$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks.
  arXiv  Detail & Related papers  (2023-03-11T03:30:47Z)
• Skip-Attention: Improving Vision Transformers by Paying Less Attention [55.47058516775423]
  Vision computation transformers (ViTs) use expensive self-attention operations in every layer. We propose SkipAt, a method to reuse self-attention from preceding layers to approximate attention at one or more subsequent layers. We show the effectiveness of our method in image classification and self-supervised learning on ImageNet-1K, semantic segmentation on ADE20K, image denoising on SIDD, and video denoising on DAVIS.
  arXiv  Detail & Related papers  (2023-01-05T18:59:52Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.