OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models

• URL: http://arxiv.org/abs/2605.18041v1
• Date: Mon, 18 May 2026 08:33:52 GMT
• Title: OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models
• Abstract summary: We propose a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs.<n>Our method achieves efficient multimodal token reduction while maintaining strong performance, without requiring any additional training.
• Score: 49.34557936004057
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Omnimodal large language models (OmniLLMs) have recently gained increasing attention for unified audio-video understanding. However, processing long multimodal token sequences introduces substantial computational overhead, making efficient token compression crucial. Existing methods typically rely on fixed, modality-specific guidance, which fails to account for the varying importance of modalities across different queries. To address this limitation, we propose $\textbf{OmniSelect}$, a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs. Specifically, we leverage a lightweight AudioCLIP model to estimate cross-modal relevance and categorize each input into three pruning regimes: Audio-Centric, Video-Centric, and Uniform pruning. Based on these relevance scores, OmniSelect further performs fine-grained token pruning within each temporal group, adaptively allocating pruning ratios to preserve informative tokens across modalities. By explicitly modeling modality preference and enabling dynamic strategy selection, OmniSelect effectively avoids the pitfalls of one-size-fits-all compression. Extensive experiments demonstrate that our method achieves efficient multimodal token reduction while maintaining strong performance, without requiring any additional training.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• Stepwise Token Selection for Efficient Multimodal Large Language Models [0.7233065479782755]
  We introduce a pointer-style selection mechanism that iteratively chooses informative tokens, conditioning each decision on previously selected ones, and dynamically determines when to stop.<n>Under aggressive pruning that removes 88.9% of visual tokens, our method preserves 94.6% of the original accuracy while achieving a 1.88x speed-up in prefill latency.
  arXiv  Detail & Related papers  (2026-06-14T23:50:08Z)
• OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering [53.30273345137238]
  We propose OmniJigsaw, a generic self-supervised framework built upon a temporal reordering proxy task.<n>Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking are used to compel cross-modal integration.<n>Our analysis reveals a bi-modal shortcut phenomenon'' in joint modality integration and demonstrates that fine-grained clip-level modality masking mitigates this issue.
  arXiv  Detail & Related papers  (2026-04-09T13:09:40Z)
• DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression [5.201210378318278]
  We propose Dynamic Audio-driven Semantic cHunking (DASH) to align token compression with semantic structure.<n>DASH treats audio embeddings as a semantic anchor and detects boundary candidates via cosine-similarity discontinuities.<n> experiments on AVUT, VideoMME, and WorldSense demonstrate that DASH maintains superior accuracy while achieving higher compression ratios compared to prior methods.
  arXiv  Detail & Related papers  (2026-03-15T15:22:06Z)
• OmniRet: Efficient and High-Fidelity Omni Modality Retrieval [51.80205678389465]
  We present OmniRet, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio.<n>Our model demonstrates significant improvements on composed query, audio and video retrieval tasks, while achieving on-par performance with state-of-the-art models on others.
  arXiv  Detail & Related papers  (2026-03-02T17:19:55Z)
• Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder [18.13981375699951]
  We propose Omni-C, a single dense Transformer-based encoder that learns competitive shared representations across heterogeneous modalities.<n>By maximizing parameter sharing in the backbone, Omni-C effectively mitigates inter-modality conflicts without requiring MoE, paired supervision, or routing.<n>Experiments show Omni-C achieves performance comparable to expert models in unimodal and cross-model tasks.
  arXiv  Detail & Related papers  (2026-02-27T06:41:25Z)
• OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models [42.615226139135174]
  We propose OmniSIFT, a token compression framework tailored for Omni-LLMs.<n>The entire framework is optimized end-to-end via a differentiable straight-through estimator.<n>With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
  arXiv  Detail & Related papers  (2026-02-04T17:51:05Z)
• VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs [82.72388893596555]
  Multimodal Large Language Models (MLLMs) encounter significant computational and memory bottlenecks.<n>Previous token compression techniques are often constrained by rules that risk discarding critical information.<n>We reformulate token compression as a lightweight plug-and-play framework that reformulates token compression into an end-to-end learnable decision process.
  arXiv  Detail & Related papers  (2025-10-18T17:54:18Z)
• Efficient Multi-modal Long Context Learning for Training-free Adaptation [96.21248144937627]
  This paper introduces Efficient Multi-Modal Long Context Learning (EMLoC)<n>It embeds demonstration examples directly into the model input.<n>It condenses long-context multimodal inputs into compact, task-specific memory representations.
  arXiv  Detail & Related papers  (2025-05-26T10:49:44Z)
• Robust Multimodal Learning via Cross-Modal Proxy Tokens [15.489426398410346]
  Multimodal models often experience a significant performance drop when one or more modalities are missing during inference.<n>We propose a simple yet effective approach that enhances robustness to missing modalities while maintaining strong performance when all modalities are available.
  arXiv  Detail & Related papers  (2025-01-29T18:15:49Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.