OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs

• URL: http://arxiv.org/abs/2607.25669v1
• Date: Tue, 28 Jul 2026 12:50:31 GMT
• Title: OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs
• Abstract summary: We propose a training-free, skill-driven framework that couples intent-aware inter-modal allocation with content-aware intra-modal allocation.<n>At 25% token retention on Qwen2.5- Omni-7B, OmniDelta reduces GPU memory by 22.0% and achieves a 1.64x end-to-end speedup over full-token inference.
• Score: 27.392988683974256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Emerging Omni-modal Large Language Models (OmniLLMs) enable unified understanding of text, audio, and video, but their long audio-video token sequences introduce substantial memory and inference costs. Existing compression methods mainly focus on selecting important tokens under fixed budgets, leaving the preceding budget-allocation problem underexplored. We show that direct query-to-audio/video similarity is unreliable for inter-modal budget allocation, and that uniform intra-modal budgets can miss key evidence while retaining redundant content. To address these limitations, we propose OmniDelta, a training-free, skill-driven framework that couples intent-aware inter-modal allocation with content-aware intra-modal allocation. OmniDelta first constructs audio and video skill pools to shift the fixed retained-token budget according to query demand, then reallocates modality budgets over audio segments and video frames using local complexity and temporal redundancy. The resulting local budgets can be combined with existing pruning strategies, preserving the total retained-token ratio while changing where the budget is spent. Experiments on four audio-video benchmarks with two Qwen2.5-Omni models show that OmniDelta establishes a new accuracy-efficiency Pareto frontier across pruning ratios. At 25% token retention on Qwen2.5-Omni-7B, OmniDelta reduces GPU memory by 22.0% and achieves a 1.64x end-to-end speedup over full-token inference.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models [4.202254591215664]
  We propose a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video.<n>Across four audio-video benchmarks and two Qwen2.5- Omni model scales, OmniScope achieves the best average accuracy across all compression settings.<n>These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates.
  arXiv  Detail & Related papers  (2026-07-25T13:16:04Z)
• Surprise Forcing: What to Remember, When to Skip in Long Video Generation [65.0022589149937]
  Streaming autoregressive diffusion makes minute-scale video synthesis practical, but its bounded context and fixed denoising schedule allocate resources uniformly.<n>We introduce Surprise Forcing, a training-free framework that treats both limitations as online resource-allocation problems.<n> Experiments on VBench, VBench-Long, and VBench-2.0 show that the proposed allocation strategy improves long-horizon consistency and visual quality while retaining real-time streaming throughput.
  arXiv  Detail & Related papers  (2026-07-20T18:37:58Z)
• OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models [72.34361439042634]
  We propose a training-free query-guided token compression method for OmniLLMs that performs independent importance estimation for video and audio.<n>Experiments on the Qwen2.5- Omni model family across four audio-visual benchmarks show that OmniFocus maintains strong compressed performance at low token retention ratios.
  arXiv  Detail & Related papers  (2026-07-03T07:41:00Z)
• Native Active Perception as Reasoning for Omni-Modal Understanding [96.19360756363501]
  We propose OmniAgent to formulate video understanding as a POMDP-based iterative Observation-Thought-Action cycle.<n> OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory.<n>We show that OmniAgent achieves state-of-the-art performance among open-source models.
  arXiv  Detail & Related papers  (2026-06-17T17:59:56Z)
• OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs [30.52254659395564]
  We present OmniMem, a memory-efficient streaming framework for audio-visual large language models (LLMs)<n>Unlike existing compression methods that treat all tokens uniformly, OmniMem introduces a modality-aware memory allocation strategy.<n>We show that OmniMem consistently improves over strong training-free compression baselines by 2-4% absolute accuracy under the same memory budgets.
  arXiv  Detail & Related papers  (2026-05-26T19:12:13Z)
• OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models [49.34557936004057]
  We propose a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs.<n>Our method achieves efficient multimodal token reduction while maintaining strong performance, without requiring any additional training.
  arXiv  Detail & Related papers  (2026-05-18T08:33:52Z)
• OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models [14.355322423471018]
  We propose a training-free two-stage framework for efficient audio-visual token compression in Omni-LLMs.<n>First, Correspondence-Preserving Chunk Refinement refines native chunk boundaries into cross-modally aligned compression units.<n>Second, Modality-Aware Cooperative Compression jointly compresses video and audio tokens within each refined unit to reduce redundancy while preserving critical evidence.
  arXiv  Detail & Related papers  (2026-05-12T12:42:44Z)
• Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation [53.176242285107485]
  We introduce motion-free inference that assigns varying step budgets to different redundant tokens based on their velocity dynamics.<n>To resolve the resulting sequence-length mismatch, HSA introduces a KV-cache mechanism that allows active tokens to attend to the full sequence.<n>We evaluate HSA on the Wan-2 and LTX-2 models for both text-to-video (T2V) and image-to-video (2V) generation.
  arXiv  Detail & Related papers  (2026-05-07T19:49:47Z)
• Small Vision-Language Models are Smart Compressors for Long Video Understanding [73.65465038390771]
  Long-form video understanding relies on intent-driven efficiency rather than greedily padded context.<n>We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding.<n>Tests show Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency.
  arXiv  Detail & Related papers  (2026-04-09T11:40:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.