Omni-Prune: Query-Aware Unified Token Pruning for Efficient Omnimodal Large Language Models

• URL: http://arxiv.org/abs/2607.23445v1
• Date: Sun, 26 Jul 2026 03:51:39 GMT
• Title: Omni-Prune: Query-Aware Unified Token Pruning for Efficient Omnimodal Large Language Models
• Abstract summary: Omni-Prune is a training-free, query-aware audio-visual token pruning framework.<n>It removes redundancy from both modalities while keeping task-relevant cross-modal evidence.<n>It delivers up to 3.25x prefill speedup and 1.3x memory reduction while retaining over 99% of full-model performance.
• Score: 18.902147033940434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Omnimodal large language models (OmniLLMs) are rapidly extending multimodal reasoning to cover synchronized audio and video. However, the resulting audio-video token sequences are long, leading to high prefill latency and GPU memory usage at inference time. Existing token pruning methods, designed mainly for vision-only inputs, miss both the cross-modal links between audio and video and the user query that decides which content matters. To bridge this gap, we present Omni-Prune, a training-free, query-aware audio-visual token pruning framework that jointly removes redundancy from both modalities while keeping task-relevant cross-modal evidence. Specifically, Omni-Prune first splits the token sequence into adaptive time windows placed at audio saliency peaks, then scores audio and video tokens on a single scale that combines encoder attention with text-query relevance, and pairs related audio-video tokens so that they are kept together. Within each window, a final K-medoids step then selects a few representative tokens, adding diverse cues that score-based selection alone would miss. Extensive experiments demonstrate that Omni-Prune outperforms established baseline methods, delivering up to 3.25x prefill speedup and 1.3x memory reduction while retaining over 99% of full-model performance.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models [4.202254591215664]
  We propose a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video.<n>Across four audio-video benchmarks and two Qwen2.5- Omni model scales, OmniScope achieves the best average accuracy across all compression settings.<n>These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates.
  arXiv  Detail & Related papers  (2026-07-25T13:16:04Z)
• OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models [72.34361439042634]
  We propose a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio.<n>Experiments on the Qwen2.5- Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios.
  arXiv  Detail & Related papers  (2026-07-03T07:41:00Z)
• OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models [49.34557936004057]
  We propose a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs.<n>Our method achieves efficient multimodal token reduction while maintaining strong performance, without requiring any additional training.
  arXiv  Detail & Related papers  (2026-05-18T08:33:52Z)
• OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance [11.057999826097829]
  The token explosion caused by high-resolution audio and video inputs remains a critical bottleneck for real-time applications and long-form reasoning.<n>Existing omni-modal token compression methods typically prune tokens at the input embedding level, relying on audio-video similarity or temporal co-occurrence as proxies for semantic relevance.<n>We propose OmniDrop, a training-free, layer-wise token pruning framework that progressively prunes audiovisual tokens within the LLM decoder layers rather than at the input-level.
  arXiv  Detail & Related papers  (2026-05-14T06:54:37Z)
• Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs [38.874953063372416]
  We propose ContextGuard as an inference-time token pruning framework.<n> ContextGuard predicts coarse visual semantics from audio and prunes video tokens.<n>It achieves full-token-level performance on five of six benchmarks while pruning 55% of input tokens.
  arXiv  Detail & Related papers  (2026-05-12T06:35:29Z)
• EchoingPixels: Cross-Modal Adaptive Token Reduction for Efficient Audio-Visual LLMs [28.295585578439212]
  EchoingPixels is a framework inspired by the coexistence and interaction of visuals and sound in real-world scenes.<n>It reduces tokens from an entire combined pool of audio-visual tokens rather than using fixed budgets per modality.<n>It achieves performance comparable to strong baselines using only 5-20% of the original tokens, with a 2-3x speedup and memory reduction.
  arXiv  Detail & Related papers  (2025-12-11T06:18:58Z)
• OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models [21.542205813984243]
  We present a training-free, audio-visual token-compression framework that optimize multimodal token representation and inference.<n>Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information.<n>For each time window, OmniZip compresses the video tokens using an interleaved-temporal scheme.
  arXiv  Detail & Related papers  (2025-11-18T15:22:32Z)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs [72.425061028374]
  We introduce OmniVideoBench, a benchmark dedicated to assessing synergistic audio-visual understanding.<n> OmniVideoBench comprises 1000 high-quality question-answer(QA) pairs, each annotated with step-by-step reasoning traces.<n>We will release OmniVideoBench to foster the development of MLLMs with stronger and more generalizable reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-12T16:34:00Z)
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs [53.248502396225724]
  AudioMarathon is a benchmark designed to evaluate both understanding and inference efficiency on long-form audio.<n>We evaluate state-of-the-art LALMs and observe clear performance drops as audio length grows.<n>The results show large gaps across current LALMs and highlight the need for better temporal reasoning.
  arXiv  Detail & Related papers  (2025-10-08T17:50:16Z)
• Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities [67.89368528234394]
  One of the main challenges of multimodal learning is the need to combine heterogeneous modalities. Video and audio are obtained at much higher rates than text and are roughly aligned in time. Our approach achieves the state-of-the-art on well established multimodal benchmarks, outperforming much larger models.
  arXiv  Detail & Related papers  (2023-11-09T19:15:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.