Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding

• URL: http://arxiv.org/abs/2603.23914v1
• Date: Wed, 25 Mar 2026 04:01:10 GMT
• Title: Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding
• Abstract summary: This paper presents AttentionPack, an adaptive and attention-aware optimization framework tailored for large vision-language models.<n>We show that AttentionPack improves memory efficiency by up to 8x, enabling higher batch sizes and faster batch inference.<n>We also report the effectiveness of AttentionPack combined with eviction, quantization and kernel fusion.
• Score: 32.02306732686021
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (VLMs) have achieved remarkable success in multi-modal reasoning, but their inference time efficiency remains a significant challenge due to the memory overhead during decoding, especially when the query and answer of VLMs consist of long sequences of visual and text tokens. This paper presents AttentionPack, an adaptive and attention-aware optimization framework tailored for large vision-language models with improving memory-efficiency during decoding, focusing on addressing the challenges due to the increased high number of visual inputs and interactions, particularly in long-context tasks with multiple high-resolution images or videos. AttentionPack is novel in two aspects: (i) We introduce a multi-head attention compaction method for economically storing key and value matrices by exploiting the implicit low-rank structure, and (ii) we develop a token-specific attention-aware decompression mechanism to reduce latency overhead. Experimental results on multiple benchmarks demonstrate that AttentionPack improves memory efficiency by up to 8x, enabling higher batch sizes and faster batch inference while preserving the model output quality or longer context lengths for superior retrieval performance. We also report the effectiveness of AttentionPack combined with eviction, quantization and kernel fusion, showing further efficiency gains for resource-limited environments.

Related papers

• Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects [26.144665202231014]
  Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance.<n>We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding, prefilling, and decoding.
  arXiv  Detail & Related papers  (2026-04-07T07:44:11Z)
• VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions [51.41587958253802]
  We introduce VISion On Request (VISOR), a method that reduces inference cost without discarding visual information.<n>VISOR improves efficiency by sparsifying the interaction between image and text tokens.<n>Experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results.
  arXiv  Detail & Related papers  (2026-03-24T17:58:17Z)
• MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval [111.99106496142474]
  Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure.<n>Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents.<n>We propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation.
  arXiv  Detail & Related papers  (2026-03-07T15:17:22Z)
• An Image Is Worth Ten Thousand Words: Verbose-Text Induction Attacks on VLMs [48.05423013052023]
  This paper proposes a novel verbose-text induction attack (VTIA) to inject imperceptible adversarial perturbations into benign images.<n>We first perform adversarial prompt search, employing reinforcement learning strategies to automatically identify adversarial prompts.<n>We then conduct vision-aligned perturbation optimization to craft adversarial examples on input images, maximizing the similarity between the perturbed image's visual embeddings and those of the adversarial prompt.
  arXiv  Detail & Related papers  (2025-11-20T09:03:43Z)
• FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding [55.700832127331324]
  FLoC is an efficient visual token compression framework based on the facility location function.<n>Our method achieves remarkable efficiency gains by swiftly selecting a compact subset of tokens.<n>Our approach is training-free, model-agnostic, and query-agnostic, providing a versatile solution.
  arXiv  Detail & Related papers  (2025-10-31T17:29:39Z)
• AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference [11.73134417321505]
  We propose AirCache, a novel KV cache compression method aimed at accelerating LVLMs inference.<n>We show that our method achieves comparable performance to the full cache while retaining only 10% of visual KV cache.
  arXiv  Detail & Related papers  (2025-03-31T11:13:18Z)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model [56.43860351559185]
  We introduce textbfTopV, a compatible textbfTOken textbfPruning with inference Time Optimization for fast and low-memory textbfVLM.<n>Our framework incorporates a visual-aware cost function to measure the importance of each source visual token, enabling effective pruning of low-importance tokens.
  arXiv  Detail & Related papers  (2025-03-24T01:47:26Z)
• FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance [9.782362715017596]
  We introduce FOLDER, a simple yet effective plug-and-play module designed to reduce the length of the visual token sequence.<n>We analyze the information loss introduced by different reduction strategies and develop FOLDER to preserve key information while removing visual redundancy.<n>FOLDER achieves comparable or even better performance than the original models, while dramatically reducing complexity by removing up to 70% of visual tokens.
  arXiv  Detail & Related papers  (2025-01-05T03:28:45Z)
• FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression [45.37530855889661]
  High-resolution images lead to a quadratic increase in the number of visual tokens input into Multi-modal Large Language Models. Current work develop visual token compression methods to achieve efficiency improvements, often at the expense of performance. We build a coarse-to-fine visual token compression method, with a vision-guided sampler for compressing redundant regions with low information density, and a text-guided sampler for selecting visual tokens that are strongly correlated with the user instructions.
  arXiv  Detail & Related papers  (2024-11-21T15:37:52Z)
• Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers [58.5711048151424]
  We introduce SPARSEK Attention, a novel sparse attention mechanism designed to overcome computational and memory obstacles. Our approach integrates a scoring network and a differentiable top-k mask operator, SPARSEK, to select a constant number of KV pairs for each query. Experimental results reveal that SPARSEK Attention outperforms previous sparse attention methods.
  arXiv  Detail & Related papers  (2024-06-24T15:55:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.