MambaMia: A State-Space-Model-Based Compression for Efficient Video Understanding in Large Multimodal Models

• URL: http://arxiv.org/abs/2506.13564v1
• Date: Mon, 16 Jun 2025 14:49:49 GMT
• Title: MambaMia: A State-Space-Model-Based Compression for Efficient Video Understanding in Large Multimodal Models
• Authors: Geewook Kim, Minjoon Seo,
• Abstract summary: We propose an efficient framework to compress multiple video-frame features before feeding them into large multimodal models.<n>Our framework emphasizes resource-conscious efficiency, making it practical for real-world deployments.
• Score: 33.89483627891117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose an efficient framework to compress multiple video-frame features before feeding them into large multimodal models, thereby mitigating the severe token explosion arising from long or dense videos. Our design leverages a bidirectional state-space-based block equipped with a gated skip connection and a learnable weighted-average pooling mechanism applied to periodically inserted learned queries. This structure enables hierarchical downsampling across both spatial and temporal dimensions, preserving performance in a cost-effective manner. Across challenging long and dense video understanding tasks, our approach demonstrates competitive results against state-of-the-art models, while significantly reducing overall token budget. Notably, replacing our proposed state-space block with a conventional Transformer results in substantial performance degradation, highlighting the advantages of state-space modeling for effectively compressing multi-frame video data. Our framework emphasizes resource-conscious efficiency, making it practical for real-world deployments. We validate its scalability and generality across multiple benchmarks, achieving the dual objectives of efficient resource usage and comprehensive video understanding.

Related papers

• Conditional Video Generation for High-Efficiency Video Compression [47.011087624381524]
  We propose a video compression framework that leverages conditional diffusion models for perceptually optimized reconstruction.<n>Specifically, we reframe video compression as a conditional generation task, where a generative model synthesizes video from sparse, yet informative signals.
  arXiv  Detail & Related papers  (2025-07-21T06:16:27Z)
• LoViC: Efficient Long Video Generation with Context Compression [68.22069741704158]
  We introduce LoViC, a DiT-based framework trained on million-scale open-domain videos.<n>At the core of our approach is FlexFormer, an expressive autoencoder that jointly compresses video and text into unified latent representations.
  arXiv  Detail & Related papers  (2025-07-17T09:46:43Z)
• FCA2: Frame Compression-Aware Autoencoder for Modular and Fast Compressed Video Super-Resolution [68.77813885751308]
  State-of-the-art (SOTA) compressed video super-resolution (CVSR) models face persistent challenges, including prolonged inference time, complex training pipelines, and reliance on auxiliary information.<n>We propose an efficient and scalable solution inspired by the structural and statistical similarities between hyperspectral images (HSI) and video data.<n>Our approach introduces a compression-driven dimensionality reduction strategy that reduces computational complexity, accelerates inference, and enhances the extraction of temporal information across frames.
  arXiv  Detail & Related papers  (2025-06-13T07:59:52Z)
• Super-Resolution Generative Adversarial Networks based Video Enhancement [0.0]
  This work introduces an enhanced approach to video super-resolution by extending ordinary Single-Image-SIS (SRGAN) structure to handle-versarial data.<n>A modified framework that incorporates 3D Non-Local Blocks is developed, which is enabling the model to capture relationships across both spatial and temporal dimensions.<n>Results show improved temporal coherence, sharper textures, and fewer visual artifacts compared to traditional single-image methods.
  arXiv  Detail & Related papers  (2025-05-14T20:16:51Z)
• Towards Scalable Modeling of Compressed Videos for Efficient Action Recognition [6.168286187549952]
  We propose a hybrid end-to-end framework that factorizes learning across three key concepts to reduce inference cost by $330times$ versus prior art.<n> Experiments show that our method results in a lightweight architecture achieving state-of-the-art video recognition performance.
  arXiv  Detail & Related papers  (2025-03-17T21:13:48Z)
• RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks [93.18404922542702]
  We present a novel video generative model designed to address long-term spatial and temporal dependencies. Our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks. Our model synthesizes high-fidelity video clips at a resolution of $256times256$ pixels, with durations extending to more than $5$ seconds at a frame rate of 30 fps.
  arXiv  Detail & Related papers  (2024-01-11T16:48:44Z)
• A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames [57.758863967770594]
  We build on the common paradigm of transferring large-scale, image--text models to video via shallow temporal fusion.<n>We expose two limitations to the approach: (1) decreased spatial capabilities, likely due to poor video--language alignment in standard video datasets, and (2) higher memory consumption, bottlenecking the number of frames that can be processed.
  arXiv  Detail & Related papers  (2023-12-12T16:10:19Z)
• Video Frame Interpolation Transformer [86.20646863821908]
  We propose a Transformer-based video framework that allows content-aware aggregation weights and considers long-range dependencies with the self-attention operations. To avoid the high computational cost of global self-attention, we introduce the concept of local attention into video. In addition, we develop a multi-scale frame scheme to fully realize the potential of Transformers.
  arXiv  Detail & Related papers  (2021-11-27T05:35:10Z)
• An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement [132.60976158877608]
  We propose an efficient adversarial video enhancement framework that learns directly from unpaired video examples. In particular, our framework introduces new recurrent cells that consist of interleaved local and global modules for implicit integration of spatial and temporal information. The proposed design allows our recurrent cells to efficiently propagate-temporal-information across frames and reduces the need for high complexity networks.
  arXiv  Detail & Related papers  (2020-12-24T00:03:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.