MamFusion: Multi-Mamba with Temporal Fusion for Partially Relevant Video Retrieval

• URL: http://arxiv.org/abs/2506.03473v1
• Date: Wed, 04 Jun 2025 01:08:03 GMT
• Title: MamFusion: Multi-Mamba with Temporal Fusion for Partially Relevant Video Retrieval
• Authors: Xinru Ying, Jiaqi Mo, Jingyang Lin, Canghong Jin, Fangfang Wang, Lina Wei,
• Abstract summary: Partially Relevant Video Retrieval (PRVR) is a challenging task in the domain of multimedia retrieval.<n>In this work, we investigate long-sequence video content understanding to address information redundancy issues.<n>We introduce a multi-Mamba module with temporal fusion framework (MamFusion) tailored for PRVR task.
• Score: 4.747939043785552
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Partially Relevant Video Retrieval (PRVR) is a challenging task in the domain of multimedia retrieval. It is designed to identify and retrieve untrimmed videos that are partially relevant to the provided query. In this work, we investigate long-sequence video content understanding to address information redundancy issues. Leveraging the outstanding long-term state space modeling capability and linear scalability of the Mamba module, we introduce a multi-Mamba module with temporal fusion framework (MamFusion) tailored for PRVR task. This framework effectively captures the state-relatedness in long-term video content and seamlessly integrates it into text-video relevance understanding, thereby enhancing the retrieval process. Specifically, we introduce Temporal T-to-V Fusion and Temporal V-to-T Fusion to explicitly model temporal relationships between text queries and video moments, improving contextual awareness and retrieval accuracy. Extensive experiments conducted on large-scale datasets demonstrate that MamFusion achieves state-of-the-art performance in retrieval effectiveness. Code is available at the link: https://github.com/Vision-Multimodal-Lab-HZCU/MamFusion.

Related papers

• AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding [73.60257070465377]
  AdaVideoRAG is a novel framework that adapts retrieval based on query complexity using a lightweight intent classifier.<n>Our framework employs an Omni-Knowledge Indexing module to build hierarchical databases from text (captions, ASR, OCR), visual features, and semantic graphs.<n> Experiments demonstrate improved efficiency and accuracy for long-video understanding, with seamless integration into existing MLLMs.
  arXiv  Detail & Related papers  (2025-06-16T15:18:15Z)
• A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking [47.312955861553995]
  We propose Unified Video Fusion (UniVF), a novel framework for temporally coherent video fusion.<n>To support its development, we also introduce Video Fusion Benchmark (VF-Bench), the first comprehensive benchmark covering four video fusion tasks.
  arXiv  Detail & Related papers  (2025-05-26T11:45:10Z)
• VideoFusion: A Spatio-Temporal Collaborative Network for Mutli-modal Video Fusion and Restoration [26.59510171451438]
  Existing multi-sensor fusion research predominantly integrates complementary from multiple images rather than videos.<n>VideoFusion exploits cross-modal complementarity and temporal dynamics to generate context-temporally coherent videos.<n>Extensive experiments reveal that VideoFusion outperforms existing image-oriented fusion paradigms in sequential scenarios.
  arXiv  Detail & Related papers  (2025-03-30T08:27:18Z)
• Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment [0.0]
  Long Video Question Answering (LVQA) is challenging due to the need for temporal reasoning and large-scale multimodal data processing.<n>We introduce UMaT, a retrieval-augmented generation framework that efficiently processes extremely long videos.<n>We show that UMaT outperforms existing methods in multimodal integration, long-form video understanding, and sparse information retrieval.
  arXiv  Detail & Related papers  (2025-03-12T05:28:24Z)
• Token-Efficient Long Video Understanding for Multimodal LLMs [101.70681093383365]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis [52.050036778325094]
  We introduce SALOVA: Segment-Augmented Video Assistant, a novel video-LLM framework designed to enhance the comprehension of lengthy video content.<n>We present a high-quality collection of 87.8K long videos, each densely captioned at the segment level to enable models to capture scene continuity and maintain rich context.<n>Our framework mitigates the limitations of current video-LMMs by allowing for precise identification and retrieval of relevant video segments in response to queries.
  arXiv  Detail & Related papers  (2024-11-25T08:04:47Z)
• MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline [24.93092798651332]
  We introduce MERLIN, a training-free pipeline that leverages Large Language Models (LLMs) for iterative feedback learning. MERLIN refines query embeddings from a user perspective, enhancing alignment between queries and video content. Experimental results on datasets like MSR-VTT, MSVD, and ActivityNet demonstrate that MERLIN substantially improves Recall@1, outperforming existing systems.
  arXiv  Detail & Related papers  (2024-07-17T11:45:02Z)
• Improving Video Corpus Moment Retrieval with Partial Relevance Enhancement [72.7576395034068]
  Video Corpus Moment Retrieval (VCMR) is a new video retrieval task aimed at retrieving a relevant moment from a large corpus of untrimmed videos using a text query. We argue that effectively capturing the partial relevance between the query and video is essential for the VCMR task. For video retrieval, we introduce a multi-modal collaborative video retriever, generating different query representations for the two modalities. For moment localization, we propose the focus-then-fuse moment localizer, utilizing modality-specific gates to capture essential content.
  arXiv  Detail & Related papers  (2024-02-21T07:16:06Z)
• CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval [24.649068267308913]
  Video retrieval applications should enable users to retrieve a precise moment from a large video corpus. We propose a novel model for effective moment localization and ranking. We conduct studies on two datasets, TVR for closed-world TV episodes and DiDeMo for open-world user-generated videos.
  arXiv  Detail & Related papers  (2021-09-21T08:07:27Z)
• Temporal Context Aggregation for Video Retrieval with Contrastive Learning [81.12514007044456]
  We propose TCA, a video representation learning framework that incorporates long-range temporal information between frame-level features. The proposed method shows a significant performance advantage (17% mAP on FIVR-200K) over state-of-the-art methods with video-level features.
  arXiv  Detail & Related papers  (2020-08-04T05:24:20Z)
• Multi-modal Transformer for Video Retrieval [67.86763073161012]
  We present a multi-modal transformer to jointly encode the different modalities in video. On the natural language side, we investigate the best practices to jointly optimize the language embedding together with the multi-modal transformer. This novel framework allows us to establish state-of-the-art results for video retrieval on three datasets.
  arXiv  Detail & Related papers  (2020-07-21T07:38:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.