OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models

• URL: http://arxiv.org/abs/2607.03050v1
• Date: Fri, 03 Jul 2026 07:41:00 GMT
• Title: OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
• Abstract summary: We propose a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio.<n>Experiments on the Qwen2.5- Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios.
• Score: 72.34361439042634
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Omni modal large language models (OmniLLMs) have attracted wide attention for their ability to jointly process audio and video, but they generate large token sequences under audio-visual inputs, leading to substantial inference cost. Existing audio-visual token compression methods often rely on unimodal guidance, overlooking the temporal locality of query-relevant evidence in audio-visual inputs and implicitly assuming that the two modalities share a temporally aligned information density distribution. We propose \textbf{OmniFocus}, a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio, enabling a modality-symmetric compression design that preserves modality-specific salient evidence while maintaining audio-visual alignment, thereby mitigating the modality bias issue that can arise from unimodal-guided compression. Experiments on the Qwen2.5-Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios and outperforms existing baselines on several major benchmark scores at 25\% token retention. On DailyOmni with Qwen2.5-Omni-7B at 25\% token retention, OmniFocus maintains 59.40 accuracy while delivering up to 1.38$\times$ prefill speedup relative to the full-token baseline, highlighting a favorable practical accuracy-efficiency trade-off.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs [27.392988683974256]
  We propose a training-free, skill-driven framework that couples intent-aware inter-modal allocation with content-aware intra-modal allocation.<n>At 25% token retention on Qwen2.5- Omni-7B, OmniDelta reduces GPU memory by 22.0% and achieves a 1.64x end-to-end speedup over full-token inference.
  arXiv  Detail & Related papers  (2026-07-28T12:50:31Z)
• OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models [23.801162485653478]
  We propose a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video.<n>Across four audio-video benchmarks and two Qwen2.5- Omni model scales, OmniScope achieves the best average accuracy across all compression settings.<n>These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates.
  arXiv  Detail & Related papers  (2026-07-25T13:16:04Z)
• O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding [16.003413095973784]
  We introduce OMAC, a training free plug in compression method that preserves salient visual memory and temporally grounded audio anchors.<n>We also introduce O-MARC, a compression distillation framework for learning with memory compressed multimodal contexts.<n>On Qwen2.5- Omni-3B, O-MARC improves the average score across four benchmarks to 45.8, outperforming full token inference at 44.1 and OmniZip at 41.0.
  arXiv  Detail & Related papers  (2026-05-26T06:07:11Z)
• OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models [49.34557936004057]
  We propose a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs.<n>Our method achieves efficient multimodal token reduction while maintaining strong performance, without requiring any additional training.
  arXiv  Detail & Related papers  (2026-05-18T08:33:52Z)
• OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models [14.355322423471018]
  We propose a training-free two-stage framework for efficient audio-visual token compression in Omni-LLMs.<n>First, Correspondence-Preserving Chunk Refinement refines native chunk boundaries into cross-modally aligned compression units.<n>Second, Modality-Aware Cooperative Compression jointly compresses video and audio tokens within each refined unit to reduce redundancy while preserving critical evidence.
  arXiv  Detail & Related papers  (2026-05-12T12:42:44Z)
• LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs [68.35684758116453]
  This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics.<n>We rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception.<n>Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%.
  arXiv  Detail & Related papers  (2026-03-19T17:58:13Z)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity [54.95089105944234]
  Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens.<n>We propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline.<n>We demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT.
  arXiv  Detail & Related papers  (2026-03-10T10:31:58Z)
• OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models [42.615226139135174]
  We propose OmniSIFT, a token compression framework tailored for Omni-LLMs.<n>The entire framework is optimized end-to-end via a differentiable straight-through estimator.<n>With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
  arXiv  Detail & Related papers  (2026-02-04T17:51:05Z)
• InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression [114.03378443007074]
  Current tokenizers rigidly compress all content at a fixed rate, leading to redundancy or information loss.<n>This paper introduces InfoTok, a principled framework for adaptive video tokenization.<n>We develop a transformer-based adaptive compressor that enables adaptive tokenization.
  arXiv  Detail & Related papers  (2025-12-18T17:13:59Z)
• EchoingPixels: Cross-Modal Adaptive Token Reduction for Efficient Audio-Visual LLMs [28.295585578439212]
  EchoingPixels is a framework inspired by the coexistence and interaction of visuals and sound in real-world scenes.<n>It reduces tokens from an entire combined pool of audio-visual tokens rather than using fixed budgets per modality.<n>It achieves performance comparable to strong baselines using only 5-20% of the original tokens, with a 2-3x speedup and memory reduction.
  arXiv  Detail & Related papers  (2025-12-11T06:18:58Z)
• OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models [21.542205813984243]
  We present a training-free, audio-visual token-compression framework that optimize multimodal token representation and inference.<n>Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information.<n>For each time window, OmniZip compresses the video tokens using an interleaved-temporal scheme.
  arXiv  Detail & Related papers  (2025-11-18T15:22:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.