OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs

• URL: http://arxiv.org/abs/2606.07577v1
• Date: Tue, 26 May 2026 19:12:13 GMT
• Title: OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs
• Abstract summary: We present OmniMem, a memory-efficient streaming framework for audio-visual large language models (LLMs)<n>Unlike existing compression methods that treat all tokens uniformly, OmniMem introduces a modality-aware memory allocation strategy.<n>We show that OmniMem consistently improves over strong training-free compression baselines by 2-4% absolute accuracy under the same memory budgets.
• Score: 30.52254659395564
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio-visual large language models (LLMs) hold strong promise for long-form video understanding, yet their long-video inference is fundamentally limited by the linear growth of video tokens and key-value (KV) caches. We present OmniMem, a memory-efficient streaming framework designed specifically for audio-visual LLMs. Unlike existing compression methods that treat all tokens uniformly, OmniMem introduces a modality-aware memory allocation strategy that separately manages visual and audio contexts, addressing the severe token imbalance between the two modalities. OmniMem further preserves informative and non-redundant KV states through perturbation-aware memory selection, enabling compact memory without sacrificing long-range understanding. To strengthen compression under realistic deployment constraints, we also explore budget-aware fine-tuning, which encourages the model to consolidate useful information into retained memory. Experiments on VideoMME Long, LVBench, and LVOmniBench with video-SALMONN 2+ and Qwen-2.5-Omni show that OmniMem consistently improves over strong training-free compression baselines by 2-4% absolute accuracy under the same memory budgets, with an additional 1-2% gain after fine-tuning.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs [13.179456385239481]
  A-Pack is a framework that defers audio pruning until query-conditioned multimodal interactions emerge.<n>A-Pack achieves the strongest average performance among the evaluated prior methods.<n>It reduces prefill FLOPs by up to 78% and improving decoding throughput by up to 2.21x.
  arXiv  Detail & Related papers  (2026-08-09T16:18:41Z)
• OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs [27.392988683974256]
  We propose a training-free, skill-driven framework that couples intent-aware inter-modal allocation with content-aware intra-modal allocation.<n>At 25% token retention on Qwen2.5- Omni-7B, OmniDelta reduces GPU memory by 22.0% and achieves a 1.64x end-to-end speedup over full-token inference.
  arXiv  Detail & Related papers  (2026-07-28T12:50:31Z)
• Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding [56.17043150838547]
  CausalMem is dedicated to constructing a dynamic visual memory update mechanism.<n>It estimates the redundancy of visual tokens and updates the memory bank via an online semantic basis.<n>It achieves more than textbf20$times$ visual token compression ratio and only occupies about textbf82 MB storage.
  arXiv  Detail & Related papers  (2026-06-24T10:11:08Z)
• MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering [54.54526361917178]
  MemoryCard organizes long videos into self-contained Memory Cards.<n>Results demonstrate that MemoryCard consistently improves long-video QA performance under comparable visual-token budgets.
  arXiv  Detail & Related papers  (2026-06-04T09:23:31Z)
• O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding [16.003413095973784]
  We introduce OMAC, a training free plug in compression method that preserves salient visual memory and temporally grounded audio anchors.<n>We also introduce O-MARC, a compression distillation framework for learning with memory compressed multimodal contexts.<n>On Qwen2.5- Omni-3B, O-MARC improves the average score across four benchmarks to 45.8, outperforming full token inference at 44.1 and OmniZip at 41.0.
  arXiv  Detail & Related papers  (2026-05-26T06:07:11Z)
• OmniRefine: Alignment-Aware Cooperative Compression for Efficient Omnimodal Large Language Models [14.355322423471018]
  We propose a training-free two-stage framework for efficient audio-visual token compression in Omni-LLMs.<n>First, Correspondence-Preserving Chunk Refinement refines native chunk boundaries into cross-modally aligned compression units.<n>Second, Modality-Aware Cooperative Compression jointly compresses video and audio tokens within each refined unit to reduce redundancy while preserving critical evidence.
  arXiv  Detail & Related papers  (2026-05-12T12:42:44Z)
• Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding [55.7992006853979]
  SAVEMem is a framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.<n>We present SAVEMem, a training-free dual-stage framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.
  arXiv  Detail & Related papers  (2026-05-08T15:40:40Z)
• FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding [16.693006630166316]
  We propose FreshMem, a Frequency-Space Hybrid Memory network inspired by the brain's logarithmic perception and memory consolidation.<n>FreshMem reconciles short-term fidelity with long-term coherence through two synergistic modules.<n>Experiments show that FreshMem significantly boosts the Qwen2-VL baseline, yielding gains of 5.20%, 4.52%, and 2.34% on StreamingBench, OV-Bench, and OVO-Bench, respectively.
  arXiv  Detail & Related papers  (2026-02-02T05:52:11Z)
• HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding [92.59317281526239]
  HERMES is a training-free architecture for real-time and accurate understanding of video streams.<n>Hermes reuses a compact KV cache, enabling efficient streaming understanding under resource constraints.<n>Hermes achieves superior or comparable accuracy across all benchmarks, with up to 11.4% gains on streaming datasets.
  arXiv  Detail & Related papers  (2026-01-21T07:26:15Z)
• video-SALMONN S: Streaming Audio-Visual LLMs Beyond Length Limits via Memory [51.03819128505358]
  Video-SALMONN S is first to process 3-hour videos at 1 FPS and 360p resolution under a fixed memory budget.<n>A test-time-training memory module continually updates token representations to capture long-range dependencies.<n>A prompt-dependent memory reader retrieves context-relevant content from fixed-size memory.
  arXiv  Detail & Related papers  (2025-10-13T08:20:15Z)
• METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding [55.38256656122857]
  We propose METok, a training-free, Multi-stage Event-based Token compression framework.<n>We show METok achieves an optimal trade-off between efficiency and accuracy by dynamically selecting informative visual tokens.<n>For instance, equipping LongVA-7B with METok realizes an 80.6% FLOPs reduction and 93.5% KV Cache memory savings.
  arXiv  Detail & Related papers  (2025-06-03T13:19:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.