Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs

• URL: http://arxiv.org/abs/2605.11605v1
• Date: Tue, 12 May 2026 06:35:29 GMT
• Title: Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs
• Abstract summary: We propose ContextGuard as an inference-time token pruning framework.<n> ContextGuard predicts coarse visual semantics from audio and prunes video tokens.<n>It achieves full-token-level performance on five of six benchmarks while pruning 55% of input tokens.
• Score: 38.874953063372416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Omnimodal Large Language Models (Omni-LLMs) incur substantial computational overhead due to the large number of multimodal input tokens they process, making token reduction essential for real-world deployment. Existing Omni-LLM pruning methods typically reduce this cost by selecting tokens that are important for the current query or strongly aligned with cross-modal cues. However, such strategies can discard evidence that falls outside these criteria, even when needed for different questions or for understanding context beyond aligned audio-visual cues. To address this limitation, we reframe Omni-LLM token reduction as preserving broad audio-visual context while removing cross-modal redundancy. We propose ContextGuard, an inference-time token pruning framework built on this principle. ContextGuard predicts coarse visual semantics from audio and prunes video tokens whose coarse semantics are likely recoverable from audio, while retaining additional video tokens to preserve localized visual details that audio alone cannot specify. For further compression, our method merges temporally similar video tokens. The framework requires no downstream LLM fine-tuning and uses only an independently trained lightweight predictor. On Qwen2.5-Omni and Video-SALMONN2+ at 3B and 7B scales across six audio-visual benchmarks, ContextGuard outperforms prior inference-time pruning methods while pruning more tokens. Notably, on Qwen2.5-Omni 7B, ContextGuard achieves full-token-level performance on five of six benchmarks while pruning 55% of input tokens.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• Omni-Prune: Query-Aware Unified Token Pruning for Efficient Omnimodal Large Language Models [18.902147033940434]
  Omni-Prune is a training-free, query-aware audio-visual token pruning framework.<n>It removes redundancy from both modalities while keeping task-relevant cross-modal evidence.<n>It delivers up to 3.25x prefill speedup and 1.3x memory reduction while retaining over 99% of full-model performance.
  arXiv  Detail & Related papers  (2026-07-26T03:51:39Z)
• OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models [23.801162485653478]
  We propose a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video.<n>Across four audio-video benchmarks and two Qwen2.5- Omni model scales, OmniScope achieves the best average accuracy across all compression settings.<n>These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates.
  arXiv  Detail & Related papers  (2026-07-25T13:16:04Z)
• Out of Sight, Still in Mind: Token Compression for Omni-LLMs [34.925891815982865]
  We propose ReMo, a framework that compresses visual tokens by redistributing their information across modalities.<n>On Qwen2.5- Omni at two model scales, ReMo removes 54% of the input tokens with no loss in accuracy.
  arXiv  Detail & Related papers  (2026-07-23T11:09:24Z)
• OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models [72.34361439042634]
  We propose a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio.<n>Experiments on the Qwen2.5- Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios.
  arXiv  Detail & Related papers  (2026-07-03T07:41:00Z)
• Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioning [30.40330199901482]
  We propose a AVEX-Prune, an RL-based audio-visual dynamic token pruning method.<n> AVEX-Prune preserves full-token quality at a 40% retention ratio on both VILA 1.5-8B and VideoLLaMA 2.
  arXiv  Detail & Related papers  (2026-06-09T08:04:06Z)
• OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance [11.057999826097829]
  The token explosion caused by high-resolution audio and video inputs remains a critical bottleneck for real-time applications and long-form reasoning.<n>Existing omni-modal token compression methods typically prune tokens at the input embedding level, relying on audio-video similarity or temporal co-occurrence as proxies for semantic relevance.<n>We propose OmniDrop, a training-free, layer-wise token pruning framework that progressively prunes audiovisual tokens within the LLM decoder layers rather than at the input-level.
  arXiv  Detail & Related papers  (2026-05-14T06:54:37Z)
• AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding [81.07348307304547]
  AdaptToken is a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection.<n>It consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames)<n>It reduces inference time by about half with comparable performance.
  arXiv  Detail & Related papers  (2026-03-30T17:14:15Z)
• ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs [16.523460406504604]
  ReDiPrune is a training-free token pruning method applied before the vision-conditioned projector.<n>It selects informative tokens directly from vision encoder outputs, preserving fine-grained spatial and semantic cues.<n>It consistently improves the accuracy-efficiency trade-off across four video and five image benchmarks.
  arXiv  Detail & Related papers  (2026-03-25T18:01:19Z)
• DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression [5.201210378318278]
  We propose Dynamic Audio-driven Semantic cHunking (DASH) to align token compression with semantic structure.<n>DASH treats audio embeddings as a semantic anchor and detects boundary candidates via cosine-similarity discontinuities.<n> experiments on AVUT, VideoMME, and WorldSense demonstrate that DASH maintains superior accuracy while achieving higher compression ratios compared to prior methods.
  arXiv  Detail & Related papers  (2026-03-15T15:22:06Z)
• Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models [61.11154533305096]
  Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens.<n>We propose a new perspective that elaborates token textbfAnchors within intra-frame and inter-frame contexts.<n>Our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs.
  arXiv  Detail & Related papers  (2026-03-02T03:06:40Z)
• OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models [21.542205813984243]
  We present a training-free, audio-visual token-compression framework that optimize multimodal token representation and inference.<n>Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information.<n>For each time window, OmniZip compresses the video tokens using an interleaved-temporal scheme.
  arXiv  Detail & Related papers  (2025-11-18T15:22:32Z)
• Qwen2.5-Omni Technical Report [31.033323728153984]
  We present an end-to-end multimodal model designed to perceive diverse modalities, including text, images, audio, and video, while simultaneously generating text and natural speech responses in a streaming manner.<n>Qwen2.5- Omni achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench.
  arXiv  Detail & Related papers  (2025-03-26T04:17:55Z)
• VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models [35.38573641029626]
  We introduce the novel task of Extreme Short Token Reduction, which aims to represent entire videos using a minimal set of discrete tokens.<n>On the Extreme Short Token Reduction task, our VQToken compresses sequences to just 0.07 percent of their original length while incurring only a 0.66 percent drop in accuracy on the NextQA-MC benchmark.
  arXiv  Detail & Related papers  (2025-03-21T09:46:31Z)
• Whisper-GPT: A Hybrid Representation Audio Large Language Model [1.2328446298523066]
  A generative large language model (LLM) for speech and music that allows us to work with continuous audio representations and discrete tokens simultaneously as part of a single architecture.<n>We show how our architecture improves the perplexity and negative log-likelihood scores for the next token prediction compared to a token-based LLM for speech and music.
  arXiv  Detail & Related papers  (2024-12-16T05:03:48Z)
• ElasticTok: Adaptive Tokenization for Image and Video [109.75935878130582]
  We introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens.<n>During inference, ElasticTok can dynamically allocate tokens when needed.<n>Our evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage.
  arXiv  Detail & Related papers  (2024-10-10T20:54:15Z)
• LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models [66.40252169137447]
  We present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. This dual-token strategy significantly reduces the overload of long videos while preserving critical information.
  arXiv  Detail & Related papers  (2023-11-28T18:53:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.