Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration

• URL: http://arxiv.org/abs/2501.05179v2
• Date: Wed, 15 Jan 2025 17:34:26 GMT
• Title: Compression with Global Guidance: Towards Training-free High-Resolution MLLMs Acceleration
• Abstract summary: Multimodal large language models (MLLMs) have attracted considerable attention due to their exceptional performance in visual content understanding and reasoning.<n> Token compression techniques, which reduce the number of visual tokens, have demonstrated their effectiveness in reducing computational costs.<n>We propose a novel token compression method, GlobalCom$2$, tailored for high-resolution MLLMs.
• Score: 28.311125014789905
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models (MLLMs) have attracted considerable attention due to their exceptional performance in visual content understanding and reasoning. However, their inference efficiency has been a notable concern, as the increasing length of multimodal contexts leads to quadratic complexity. Token compression techniques, which reduce the number of visual tokens, have demonstrated their effectiveness in reducing computational costs. Yet, these approaches have struggled to keep pace with the rapid advancements in MLLMs, especially the AnyRes strategy in the context of high-resolution image understanding. In this paper, we propose a novel token compression method, GlobalCom$^2$, tailored for high-resolution MLLMs that receive both the thumbnail and multiple crops. GlobalCom$^2$ treats the tokens derived from the thumbnail as the "commander" of the entire token compression process, directing the allocation of retention ratios and the specific compression for each crop. In this way, redundant tokens are eliminated while important local details are adaptively preserved to the highest extent feasible. Empirical results across 10 benchmarks reveal that GlobalCom$^2$ achieves an optimal balance between performance and efficiency, and consistently outperforms state-of-the-art token compression methods with LLaVA-NeXT-7B/13B models. Our code is released at https://github.com/xuyang-liu16/GlobalCom2.

Related papers

• VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling [81.92510561363197]
  We introduce VLZip, a framework that unifies visual and textual compression for high-fidelity reasoning within a pure Transformer.<n>We also introduce LongVLBench, a new benchmark derived from video narratives that demands holistic, narrative-level reasoning.
  arXiv  Detail & Related papers  (2026-08-09T10:31:03Z)
• VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression [52.23010472235226]
  Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference and overhead latency.<n>We propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor.
  arXiv  Detail & Related papers  (2026-07-14T13:21:36Z)
• One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding [51.08792182064565]
  Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames.<n>With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information.<n>We propose an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling.
  arXiv  Detail & Related papers  (2026-04-15T17:59:52Z)
• Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention [23.015486635502437]
  Video language models (Video-LLMs) face high computational costs due to large volumes of visual tokens.<n>We propose a unified selection mechanism that weights semantic similarity to globally select tokens.<n>Unselected tokens are merged via clustering and refilled, preserving information integrity.<n>Our unifiedtemporal token compression strategy establishes the state-of-theart in video understanding under ultra-low token retention.
  arXiv  Detail & Related papers  (2026-03-23T13:15:22Z)
• Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs [44.02979403592988]
  We propose a sample-adaptive visual token selection framework composed of semantic prominence-aware budget allocation and a two-stage selection pipeline.<n>On LLaVA-NeXT-7B, our approach reduces FLOPs by 88% and prefill latency by 22% while preserving 97.5% of the original accuracy.
  arXiv  Detail & Related papers  (2026-03-16T06:44:45Z)
• PIO-FVLM: Rethinking Training-Free Visual Token Reduction for VLM Acceleration from an Inference-Objective Perspective [59.24570811503256]
  We propose PIO-FVLM to reduce redundant visual tokens in vision-models (VLMs) to accelerate inference.<n>The proposed PIO-FVLM is training-free and compatible with FlashAttention, friendly to practical application and deployment.<n>On LLaVA-Next-7B, PIO-FVLM retains just 11.1% of visual tokens but maintains 97.2% of the original performance.
  arXiv  Detail & Related papers  (2026-02-04T15:33:10Z)
• VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs [82.72388893596555]
  Multimodal Large Language Models (MLLMs) encounter significant computational and memory bottlenecks.<n>Previous token compression techniques are often constrained by rules that risk discarding critical information.<n>We reformulate token compression as a lightweight plug-and-play framework that reformulates token compression into an end-to-end learnable decision process.
  arXiv  Detail & Related papers  (2025-10-18T17:54:18Z)
• LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit [29.877232989285833]
  Large Vision-Language Models (VLMs) exhibit impressive multi-modal capabilities but suffer from prohibitive computational and memory demands.<n>Current approaches do not decompose techniques into comparable modules, hindering fair evaluation across spatial and temporal redundancy.<n>We introduce LLMC+, a comprehensive VLM compression benchmark with a versatile, plug-and-play toolkit.
  arXiv  Detail & Related papers  (2025-08-13T17:54:49Z)
• A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models [94.49953824684853]
  We introduce a dynamic pruning framework, GlimpsePrune, inspired by human cognition.<n>It takes a data-driven ''glimpse'' and prunes irrelevant visual tokens in a single forward pass before answer generation.<n>An enhanced GlimpsePrune+ achieves 110% of the baseline performance while maintaining a similarly high pruning rate.
  arXiv  Detail & Related papers  (2025-08-03T02:15:43Z)
• LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models [62.240460476785934]
  We propose LaCo (Layer-wise Visual Token Compression), a novel framework that enables effective token compression within the intermediate layers of the vision encoder.<n>LaCo introduces two core components: 1) a layer-wise pixel-shuffle mechanism that systematically merges adjacent tokens through space-to-channel transformations, and 2) a residual learning architecture with non-parametric shortcuts.
  arXiv  Detail & Related papers  (2025-07-03T03:42:54Z)
• LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs [23.801172170798132]
  LLaVA-Scissor is a training-free token compression strategy designed for multimodal large language models.<n>We propose to leverage the Semantic Connected Components ( SCC) approach to ensure comprehensive semantic coverage.<n>We conduct extensive evaluations of the token compression capabilities of LLaVA-Scissor across diverse video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-06-27T02:29:58Z)
• Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention [30.580674811560613]
  Large Language Models (LLMs) encounter significant challenges in long-sequence inference due to computational inefficiency and redundant processing.<n>Existing methods often rely on token importance to perform hard local compression or encode context into latent representations for soft global compression.<n>We propose HyCo$$, which integrates both global and local perspectives to guide context compression.
  arXiv  Detail & Related papers  (2025-05-21T17:26:11Z)
• DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs [124.52164183968145]
  We present DyMU, an efficient, training-free framework that reduces the computational burden of vision-language models (VLMs) Our approach comprises two key components. First, Dynamic Token Merging (DToMe) reduces the number of visual token embeddings by merging similar tokens based on image complexity. Second, Virtual Token Unmerging (VTU) simulates the expected token sequence for large language models (LLMs) by efficiently reconstructing the attention dynamics of a full sequence.
  arXiv  Detail & Related papers  (2025-04-23T18:38:18Z)
• InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compression [1.8893427856534721]
  We propose InternVL-X, which outperforms the InternVL model in both performance and efficiency. By utilizing 20% or fewer visual tokens, InternVL-X achieves state-of-the-art performance on 7 public MLLM benchmarks, and improves the average metric by 2.34% across 12 tasks.
  arXiv  Detail & Related papers  (2025-03-27T09:31:35Z)
• Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models [36.16630765077807]
  We propose a Hybrid-level Instruction Injection Strategy for Conditional Token Compression in MLLMs (HICom) We use the instruction as a condition to guide the compression from both local and global levels. Experiments show that our HICom can obtain distinguished video understanding ability with fewer tokens.
  arXiv  Detail & Related papers  (2025-03-20T11:09:18Z)
• LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression [7.67622140575795]
  We present LVLM-Compress-Bench, a framework to study the broad impact of compression on the generative performance of LVLMs with multi-modal input driven tasks. We use four LVLM variants of the popular LLaVA framework to present our analysis via integrating various state-of-the-art KV and weight compression methods. Our framework demonstrates the compression impact on both general and critical metrics leveraging a combination of real world and synthetic datasets.
  arXiv  Detail & Related papers  (2025-03-06T21:21:18Z)
• Vision-centric Token Compression in Large Language Model [51.92055188780033]
  Vision Centric Token Compression (Vist) is a slow-fast compression framework that mirrors human reading.<n>On eleven in-context learning benchmarks, Vist achieves the same accuracy with 2.3 times fewer tokens, cutting FLOPs by 16% and memory by 50%.
  arXiv  Detail & Related papers  (2025-02-02T13:10:06Z)
• PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models [64.9366388601049]
  Visual token compression is leveraged to reduce the considerable token length of visual inputs. We introduce a unified token compression strategy called Progressive Visual Token Compression. Our model achieves state-of-the-art performance across various video understanding benchmarks.
  arXiv  Detail & Related papers  (2024-12-12T18:59:40Z)
• Large Language Models for Lossless Image Compression: Next-Pixel Prediction in Language Space is All You Need [53.584140947828004]
  Language large model (LLM) with unprecedented intelligence is a general-purpose lossless compressor for various data modalities. We propose P$2$-LLM, a next-pixel prediction-based LLM, which integrates various elaborated insights and methodologies. Experiments on benchmark datasets demonstrate that P$2$-LLM can beat SOTA classical and learned codecs.
  arXiv  Detail & Related papers  (2024-11-19T12:15:40Z)
• Inference Optimal VLMs Need Only One Visual Token but Larger Models [54.01228554126122]
  Vision Language Models (VLMs) have demonstrated strong capabilities across various visual understanding and reasoning tasks. VLMs are often constrained by high latency during inference due to substantial compute required to process the large number of input tokens. We take some initial steps towards building approaches tailored for high token compression settings.
  arXiv  Detail & Related papers  (2024-11-05T18:54:21Z)
• VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation [66.00245701441547]
  We introduce a novel approach to reduce vision compute by leveraging redundant vision tokens "skipping layers" rather than decreasing the number of vision tokens. Our method, VideoLLM-MoD, is inspired by mixture-of-depths LLMs and addresses the challenge of numerous vision tokens in long-term or streaming video.
  arXiv  Detail & Related papers  (2024-08-29T17:21:58Z)
• Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid [87.09900996643516]
  We introduce a Complementary Image Pyramid (CIP) to mitigate semantic discontinuity during high-resolution image processing. We also introduce a Scale Compression Mechanism (SCM) to reduce the additional computational overhead by compressing the redundant visual tokens. Our experiments demonstrate that CIP can consistently enhance the performance across diverse architectures.
  arXiv  Detail & Related papers  (2024-08-04T13:55:58Z)
• Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding [54.532578213126065]
  Most document understanding methods preserve all tokens within sub-images and treat them equally. This neglects their different informativeness and leads to a significant increase in the number of image tokens. We propose Token-level Correlation-guided Compression, a parameter-free and plug-and-play methodology to optimize token processing.
  arXiv  Detail & Related papers  (2024-07-19T16:11:15Z)
• VoCo-LLaMA: Towards Vision Compression with Large Language Models [56.20788367278211]
  Vision-Language Models (VLMs) have achieved remarkable success in various multi-modal tasks, but they are often bottlenecked by the limited context window. We propose VoCo-LLaMA, the first approach to compress vision tokens using LLMs. Our method achieves minimal performance loss with a compression ratio of 576$times$, resulting in up to 94.8$%$ fewer FLOPs and 69.6$%$ acceleration in inference time.
  arXiv  Detail & Related papers  (2024-06-18T05:05:12Z)
• Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models [44.437693135170576]
  We propose a new framework, LMM with Sophisticated Tasks, Local image compression, and Mixture of global Experts (SliME) We extract contextual information from the global view using a mixture of adapters, based on the observation that different adapters excel at different tasks. The proposed method achieves leading performance across various benchmarks with only 2 million training data.
  arXiv  Detail & Related papers  (2024-06-12T17:59:49Z)
• DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval [73.82017200889906]
  Text-video retrieval is a critical multi-modal task to find the most relevant video for a text query. We propose DGL, a cross-modal Dynamic prompt tuning method with Global-Local video attention. In contrast to previous prompt tuning methods, we employ the shared latent space to generate local-level text and frame prompts.
  arXiv  Detail & Related papers  (2024-01-19T09:58:06Z)
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models [22.06402870816756]
  Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. This paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity. We show that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss.
  arXiv  Detail & Related papers  (2023-10-09T14:10:21Z)
• Low-Resolution Self-Attention for Semantic Segmentation [93.30597515880079]
  We introduce the Low-Resolution Self-Attention (LRSA) mechanism to capture global context at a significantly reduced computational cost. Our approach involves computing self-attention in a fixed low-resolution space regardless of the input image's resolution. We demonstrate the effectiveness of our LRSA approach by building the LRFormer, a vision transformer with an encoder-decoder structure.
  arXiv  Detail & Related papers  (2023-10-08T06:10:09Z)
• ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models [70.45441031021291]
  Large Vision-Language Models (LVLMs) can understand the world comprehensively by integrating rich information from different modalities. LVLMs are often problematic due to their massive computational/energy costs and carbon consumption. We propose Efficient Coarse-to-Fine LayerWise Pruning (ECoFLaP), a two-stage coarse-to-fine weight pruning approach for LVLMs.
  arXiv  Detail & Related papers  (2023-10-04T17:34:00Z)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple [90.05366363633568]
  Knowledge-Intensive Compressed LLM BenchmarK aims to redefine the evaluation protocol for compressed Large Language Models. LLM-KICK unveils many favorable merits and unfortunate plights of current SoTA compression methods. LLM-KICK is designed to holistically access compressed LLMs' ability for language understanding, reasoning, generation, in-context retrieval, in-context summarization, etc.
  arXiv  Detail & Related papers  (2023-10-02T17:42:37Z)
• MLIC++: Linear Complexity Multi-Reference Entropy Modeling for Learned Image Compression [30.71965784982577]
  We introduce MEM++, which captures diverse range of correlations inherent in the latent representation. MEM++ achieves state-of-the-art performance, reducing BD-rate by 13.39% on the Kodak dataset compared to VTM-17.0 in PSNR. MLIC++ exhibits linear GPU memory consumption with resolution, making it highly suitable for high-resolution image coding.
  arXiv  Detail & Related papers  (2023-07-28T09:11:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.