QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression

• URL: http://arxiv.org/abs/2603.21232v1
• Date: Sun, 22 Mar 2026 13:42:24 GMT
• Title: QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression
• Abstract summary: We propose a novel framework that adaptively compresses visual tokens via three collaborative branches.<n>QMoP delivers significant savings in memory, computation, and inference time.
• Score: 35.802647459700125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models suffer from severe computational and memory bottlenecks, as the number of visual tokens far exceeds that of textual tokens. While recent methods employ projector modules to align and compress visual tokens into text-aligned features, they typically depend on fixed heuristics that limit adaptability across diverse scenarios. In this paper, we first propose Query Guided Mixture-of-Projector (QMoP), a novel and flexible framework that adaptively compresses visual tokens via three collaborative branches: (1) a pooling-based branch for coarse-grained global semantics, (2) a resampler branch for extracting high-level semantic representations, and (3) a pruning-based branch for fine-grained token selection to preserve critical visual detail. To adaptively coordinate these branches, we introduce the Query Guided Router (QGR), which dynamically selects and weights the outputs from different branches based on both visual input and textual queries. A Mixture-of-Experts-style fusion mechanism is designed to aggregate the outputs, harnessing the strengths of each strategy while suppressing noise. To systematically evaluate the effects of Visual Token Compression, we also develop VTCBench, a dedicated benchmark for evaluating the information loss induced by visual token compression. Extensive experiments demonstrate that despite relying on fundamental compression modules, QMoP outperforms strong baselines and delivers significant savings in memory, computation, and inference time.

Related papers

• MoECodec: Image Compression for joint human and machine perception via Mixture-of-Experts [37.49087932678935]
  MoECodec is a token-aware image compression framework that supports multiple downstream tasks within a single model.<n>MoECodec replaces the FFN layers in transformer-based compression model token-wise Mixture-of-Experts (MoE)
  arXiv  Detail & Related papers  (2026-06-19T01:56:25Z)
• EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling [7.872065731262025]
  EvoComp is a visual token compression framework that significantly reduces token count while preserving task accuracy.<n>We show that it retains 99.3% of the original accuracy under 3x token compression and delivers up to 1.6x speedup on mobile devices.
  arXiv  Detail & Related papers  (2026-04-18T17:52:02Z)
• Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation [66.53544128707817]
  Cheers is a unified multimodal model that decouples patch-level details from semantic representations.<n> Cheers matches or surpasses advanced UMMs in both visual understanding and generation.
  arXiv  Detail & Related papers  (2026-03-13T08:55:27Z)
• UniCompress: Token Compression for Unified Vision-Language Understanding and Generation [62.943173382496276]
  Unified models aim to support both understanding and generation by encoding images into discrete tokens and processing them alongside text.<n>We propose a unified token compression algorithm that significantly reduces visual token count while preserving performance on both image understanding and generation tasks.
  arXiv  Detail & Related papers  (2026-03-11T21:27:15Z)
• EfficientPosterGen: Semantic-aware Efficient Poster Generation via Token Compression and Accurate Violation Detection [31.15442597866692]
  EfficientPosterGen is an end-to-end framework for automated academic poster generation.<n>It introduces semantic-aware retrieval and token-efficient multimodal generation.<n>It achieves substantial improvements in token efficiency and layout reliability.
  arXiv  Detail & Related papers  (2026-02-25T17:03:39Z)
• Multi-Vector Index Compression in Any Modality [73.7330345057813]
  Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos.<n>We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC)<n>AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation.
  arXiv  Detail & Related papers  (2026-02-24T18:57:33Z)
• Fast SAM2 with Text-Driven Token Pruning [52.8350457627401]
  Segment Anything Model 2 (SAM2), a vision computation model has significantly advanced in prompt-driven video object segmentation.<n>SAM2 pipelines propagate all visual tokens produced by the image encoder through downstream temporal reasoning modules, regardless of their relevance to the target object.<n>We introduce a text-guided token pruning framework that improves inference efficiency by selectively reducing token density prior to temporal propagation.
  arXiv  Detail & Related papers  (2025-12-24T18:59:05Z)
• AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs [29.68162972167947]
  We propose an object-level token merging strategy for Adaptive Token compression.<n>Our approach averagely, utilizes only 10% tokens while achieving almost 96% of the vanilla model's performance.
  arXiv  Detail & Related papers  (2025-11-18T06:12:15Z)
• IUT-Plug: A Plug-in tool for Interleaved Image-Text Generation [23.61167100602915]
  IUT-Plug is a module grounded in an Image Understanding Tree (IUT)<n>A dynamic IUT-Plug extraction module parses visual scenes into hierarchical symbolic structures.<n>A coordinated narrative-flow and image synthesis mechanism ensures cross-modal consistency.
  arXiv  Detail & Related papers  (2025-10-13T03:19:45Z)
• QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehension [86.0749609778104]
  We propose QuoTA, an ante-hoc training-free modular that extends existing large video-language models.<n>QuoTA strategically allocates frame-level importance scores based on query relevance.<n>We decouple the query through Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame importance scoring.
  arXiv  Detail & Related papers  (2025-03-11T17:59:57Z)
• LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models [35.88374542519597]
  Large Multimodal Models (LMMs) have shown significant visual reasoning capabilities by connecting a visual encoder and a large language model. Recent LMMs incorporate more complex visual inputs, such as high-resolution images and videos, which further increases the number of visual tokens significantly. We propose PruMerge, a novel adaptive visual token reduction strategy that significantly reduces the number of visual tokens without compromising the performance of LMMs.
  arXiv  Detail & Related papers  (2024-03-22T17:59:52Z)
• Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction [88.6585431949086]
  We propose a novel Hierarchical Visual Prefix fusion NeTwork (HVPNeT) for visual-enhanced entity and relation extraction. We regard visual representation as pluggable visual prefix to guide the textual representation for error insensitive forecasting decision. Experiments on three benchmark datasets demonstrate the effectiveness of our method, and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-05-07T02:10:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.