Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs

• URL: http://arxiv.org/abs/2608.08794v1
• Date: Sun, 09 Aug 2026 16:18:41 GMT
• Title: Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs
• Abstract summary: A-Pack is a framework that defers audio pruning until query-conditioned multimodal interactions emerge.<n>A-Pack achieves the strongest average performance among the evaluated prior methods.<n>It reduces prefill FLOPs by up to 78% and improving decoding throughput by up to 2.21x.
• Score: 13.179456385239481
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Omni-modal LLMs jointly process audio, video, and text, but long multimodal sequences incur substantial prefill and KV-cache costs. Existing omni-modal compression methods primarily focus on pre-LLM token reduction, leaving modality-specific compression across the LLM boundary underexplored. We propose A-PACK, a two-stage framework that defers audio pruning until query-conditioned multimodal interactions emerge. Our analysis shows that audio exhibits higher task-relevant information density and representational diversity per token than video. We further find that local audio-visual dynamics provide a more effective cue for visual selection than token-wise matching. We therefore preserve audio and compress video with local dynamics before the LLM, then progressively prune low-relevance audio and visual tokens and their KV-cache entries inside the LLM. Across four benchmarks on Qwen2.5-Omni-7B/3B, A-PACK achieves the strongest average performance among the evaluated prior methods while reducing prefill FLOPs by up to 78% and improving decoding throughput by up to 2.21x.

Related papers

• OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models [72.34361439042634]
  We propose a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio.<n>Experiments on the Qwen2.5- Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios.
  arXiv  Detail & Related papers  (2026-07-03T07:41:00Z)
• AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression [22.818321106501788]
  AVOC is a framework for long-form audio-video understanding in Omni-modal Large Language Models.<n>AVOC achieves state-of-the-art performance on long-form audio-video benchmarks.
  arXiv  Detail & Related papers  (2026-06-23T08:06:58Z)
• AudioMosaic: Contrastive Masked Audio Representation Learning [53.52371029884106]
  We introduce textbfAudioMosaic, a contrastive learning-based audio encoder for general audio understanding.<n>AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches.<n>Experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T00:56:51Z)
• OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models [14.355322423471018]
  We propose a training-free two-stage framework for efficient audio-visual token compression in Omni-LLMs.<n>First, Correspondence-Preserving Chunk Refinement refines native chunk boundaries into cross-modally aligned compression units.<n>Second, Modality-Aware Cooperative Compression jointly compresses video and audio tokens within each refined unit to reduce redundancy while preserving critical evidence.
  arXiv  Detail & Related papers  (2026-05-12T12:42:44Z)
• 2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA [47.992210130090065]
  We present ASR-SaSa2VA, a resource-efficient framework for audio-guided video segmentation.<n>Our approach achieves a final score of 80.7 in the 5th PVUW Challenge (MeViS-v2-Audio track), earning the second-place ranking.
  arXiv  Detail & Related papers  (2026-04-27T01:19:56Z)
• One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding [51.08792182064565]
  Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames.<n>With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information.<n>We propose an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling.
  arXiv  Detail & Related papers  (2026-04-15T17:59:52Z)
• EchoingPixels: Cross-Modal Adaptive Token Reduction for Efficient Audio-Visual LLMs [28.295585578439212]
  EchoingPixels is a framework inspired by the coexistence and interaction of visuals and sound in real-world scenes.<n>It reduces tokens from an entire combined pool of audio-visual tokens rather than using fixed budgets per modality.<n>It achieves performance comparable to strong baselines using only 5-20% of the original tokens, with a 2-3x speedup and memory reduction.
  arXiv  Detail & Related papers  (2025-12-11T06:18:58Z)
• OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models [21.542205813984243]
  We present a training-free, audio-visual token-compression framework that optimize multimodal token representation and inference.<n>Specifically, OmniZip first identifies salient audio tokens, then computes an audio retention score for each time group to capture information.<n>For each time window, OmniZip compresses the video tokens using an interleaved-temporal scheme.
  arXiv  Detail & Related papers  (2025-11-18T15:22:32Z)
• AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs [53.248502396225724]
  AudioMarathon is a benchmark designed to evaluate both understanding and inference efficiency on long-form audio.<n>We evaluate state-of-the-art LALMs and observe clear performance drops as audio length grows.<n>The results show large gaps across current LALMs and highlight the need for better temporal reasoning.
  arXiv  Detail & Related papers  (2025-10-08T17:50:16Z)
• Make Some Noise: Towards LLM audio reasoning and generation using sound tokens [19.48089933713418]
  We introduce a novel approach that combines Variational Quantization with Flow Matching to convert audio into ultra-low discrete tokens of 0.23kpbs.<n>Our tokenizer outperforms a traditional VQ-VAE across various datasets with diverse acoustic events.
  arXiv  Detail & Related papers  (2025-03-28T09:43:47Z)
• Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models [56.776580717999806]
  Real-world applications often involve processing multiple audio streams simultaneously. We propose the first multi-audio evaluation benchmark that consists of 20 datasets from 11 multi-audio tasks. We propose a novel multi-audio-LLM (MALLM) to capture audio context among multiple similar audios.
  arXiv  Detail & Related papers  (2024-09-27T12:06:53Z)
• STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignment [61.83340833859382]
  Continuously learning a variety of audio-video semantics over time is crucial for audio-related reasoning tasks. This is a nontemporal problem and poses two critical challenges: sparse-temporal correlation between audio-video pairs and multimodal correlation overwriting that forgets audio-video relations. We propose a continual audio-video pre-training method with two novel ideas.
  arXiv  Detail & Related papers  (2023-10-12T10:50:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.