Interactive Video Corpus Moment Retrieval using Reinforcement Learning

• URL: http://arxiv.org/abs/2302.09522v1
• Date: Sun, 19 Feb 2023 09:48:23 GMT
• Title: Interactive Video Corpus Moment Retrieval using Reinforcement Learning
• Abstract summary: This paper tackles the problem by reinforcement learning, aiming to reach a search target within a few rounds of interaction by long-term learning from user feedbacks. We conduct experiments for the challenging task of video corpus moment retrieval (VCMR) to localize moments from a large video corpus.
• Score: 35.38916770127218
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Known-item video search is effective with human-in-the-loop to interactively investigate the search result and refine the initial query. Nevertheless, when the first few pages of results are swamped with visually similar items, or the search target is hidden deep in the ranked list, finding the know-item target usually requires a long duration of browsing and result inspection. This paper tackles the problem by reinforcement learning, aiming to reach a search target within a few rounds of interaction by long-term learning from user feedbacks. Specifically, the system interactively plans for navigation path based on feedback and recommends a potential target that maximizes the long-term reward for user comment. We conduct experiments for the challenging task of video corpus moment retrieval (VCMR) to localize moments from a large video corpus. The experimental results on TVR and DiDeMo datasets verify that our proposed work is effective in retrieving the moments that are hidden deep inside the ranked lists of CONQUER and HERO, which are the state-of-the-art auto-search engines for VCMR.

Related papers

• VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning [55.2367119060441]
  Video understanding is moving beyond closed-context perception toward open-world evidence exploration.<n>Existing multimodal search agents primarily target static images, and the current VDR benchmark relies on text-centric retrieval.<n>We propose VideoSearcher, a closed-loop agentic framework that empowers Vision-Language Models with multi-tool reasoning for VDR.
  arXiv  Detail & Related papers  (2026-07-03T03:50:09Z)
• CACR:Reinforcing Temporal Answer Grounding in Instructional Video via Candidate-Aware Causal Reasoning [8.230031783815749]
  TAGV aims to locate precise video segments that respond to natural language queries.<n>This task remains challenging due to the need to comprehend semantically complex questions.<n>We propose a Candidate-Aware Causal Reasoning framework to tackle these limitations.
  arXiv  Detail & Related papers  (2026-06-07T03:23:18Z)
• Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language [60.91064560080974]
  We propose an efficient approach, SpotVMR, to trim the query-relevant clip.<n>Our proposed SpotVMR can serve as plug-and-play module, which achieves efficiency for state-of-the-art VMR methods.
  arXiv  Detail & Related papers  (2026-05-28T11:42:28Z)
• IVCR-200K: A Large-Scale Multi-turn Dialogue Benchmark for Interactive Video Corpus Retrieval [36.33423199468626]
  Interactive Video Corpus Retrieval (IVCR) task enables multi-turn, conversational, and realistic interactions between the user and the retrieval system.<n> IVCR-200K is a high-quality, bilingual, multi-turn, conversational, and abstract semantic dataset that supports video retrieval and even moment retrieval.<n>We propose a comprehensive framework based on multi-modal large language models (MLLMs) to help users interact in several modes with more explainable solutions.
  arXiv  Detail & Related papers  (2025-12-01T06:12:59Z)
• TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learning [30.895558843927862]
  Temporal search aims to identify a minimal set of relevant frames from tens of thousands based on a given query.<n>TimeSearch-R reformulates temporal search as interleaved text-video thinking.<n>It seamlessly integrates searching video clips into the reasoning process through reinforcement learning.
  arXiv  Detail & Related papers  (2025-11-07T18:58:25Z)
• VideoExplorer: Think With Videos For Agentic Long-Video Understanding [117.68219930263153]
  Long-video understanding is a challenging problem in computer vision.<n>We propose VideoExplorer, a framework grounded in the principle of thinking with video''<n>Rather than reasoning over a static context, VideoExplorer iteratively formulates sub-questions, locates relevant moments, and performs task-oriented, temporally scalable video understanding.
  arXiv  Detail & Related papers  (2025-06-12T15:39:10Z)
• MomentSeeker: A Comprehensive Benchmark and A Strong Baseline For Moment Retrieval Within Long Videos [62.01402470874109]
  We present MomentSeeker, a benchmark to evaluate retrieval models' performance in handling general long-video moment retrieval tasks. It incorporates long videos of over 500 seconds on average, making it the first benchmark specialized for long-video moment retrieval. It covers a wide range of task categories (including Moment Search, Caption Alignment, Image-conditioned Moment Search, and Video-conditioned Moment Search) and diverse application scenarios. We further fine-tune an MLLM-based LVMR retriever on synthetic data, which demonstrates strong performance on our benchmark.
  arXiv  Detail & Related papers  (2025-02-18T05:50:23Z)
• A Flexible and Scalable Framework for Video Moment Search [51.47907684209207]
  This paper introduces a flexible framework for retrieving a ranked list of moments from collection of videos in any length to match a text query. Our framework, called Segment-Proposal-Ranking (SPR), simplifies the search process into three independent stages: segment retrieval, proposal generation, and moment refinement with re-ranking. Evaluations on the TVR-Ranking dataset demonstrate that our framework achieves state-of-the-art performance with significant reductions in computational cost and processing time.
  arXiv  Detail & Related papers  (2025-01-09T08:54:19Z)
• Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning [56.873534081386]
  A new topic HIREST is presented, including video retrieval, moment retrieval, moment segmentation, and step-captioning. We propose a query-centric audio-visual cognition network to construct a reliable multi-modal representation for three tasks. This can cognize user-preferred content and thus attain a query-centric audio-visual representation for three tasks.
  arXiv  Detail & Related papers  (2024-12-18T06:43:06Z)
• VCA: Video Curious Agent for Long Video Understanding [44.19323180593379]
  We introduce a curiosity-driven video agent with self-exploration capability, dubbed as VCA. Built upon VLMs, VCA autonomously navigates video segments and efficiently builds a comprehensive understanding of complex video sequences.
  arXiv  Detail & Related papers  (2024-12-12T23:39:54Z)
• Improving Video Corpus Moment Retrieval with Partial Relevance Enhancement [72.7576395034068]
  Video Corpus Moment Retrieval (VCMR) is a new video retrieval task aimed at retrieving a relevant moment from a large corpus of untrimmed videos using a text query. We argue that effectively capturing the partial relevance between the query and video is essential for the VCMR task. For video retrieval, we introduce a multi-modal collaborative video retriever, generating different query representations for the two modalities. For moment localization, we propose the focus-then-fuse moment localizer, utilizing modality-specific gates to capture essential content.
  arXiv  Detail & Related papers  (2024-02-21T07:16:06Z)
• Simple Baselines for Interactive Video Retrieval with Questions and Answers [33.17722358007974]
  We propose several simple yet effective baselines for interactive video retrieval via question-answering. We employ a VideoQA model to simulate user interactions and show that this enables the productive study of the interactive retrieval task. Experiments on MSR-VTT, MSVD, and AVSD show that our framework using question-based interaction significantly improves the performance of text-based video retrieval systems.
  arXiv  Detail & Related papers  (2023-08-21T00:32:19Z)
• Video-based Person Re-identification with Long Short-Term Representation Learning [101.62570747820541]
  Video-based person Re-Identification (V-ReID) aims to retrieve specific persons from raw videos captured by non-overlapped cameras. We propose a novel deep learning framework named Long Short-Term Representation Learning (LSTRL) for effective V-ReID.
  arXiv  Detail & Related papers  (2023-08-07T16:22:47Z)
• Deep Learning for Video-Text Retrieval: a Review [13.341694455581363]
  Video-Text Retrieval (VTR) aims to search for the most relevant video related to the semantics in a given sentence. In this survey, we review and summarize over 100 research papers related to VTR.
  arXiv  Detail & Related papers  (2023-02-24T10:14:35Z)
• Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval [20.493241098064665]
  Video corpus moment retrieval (VCMR) is the task to retrieve the most relevant video moment from a large video corpus using a natural language query. We propose a self-supervised learning framework: Modal-specific Pseudo Query Generation Network (MPGN) MPGN generates pseudo queries exploiting both visual and textual information from selected temporal moments. We show that MPGN successfully learns to localize the video corpus moment without any explicit annotation.
  arXiv  Detail & Related papers  (2022-10-23T05:05:18Z)
• E^2TAD: An Energy-Efficient Tracking-based Action Detector [78.90585878925545]
  This paper presents a tracking-based solution to accurately and efficiently localize predefined key actions. It won first place in the UAV-Video Track of 2021 Low-Power Computer Vision Challenge (LPCVC)
  arXiv  Detail & Related papers  (2022-04-09T07:52:11Z)
• CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval [24.649068267308913]
  Video retrieval applications should enable users to retrieve a precise moment from a large video corpus. We propose a novel model for effective moment localization and ranking. We conduct studies on two datasets, TVR for closed-world TV episodes and DiDeMo for open-world user-generated videos.
  arXiv  Detail & Related papers  (2021-09-21T08:07:27Z)
• Uncovering Hidden Challenges in Query-Based Video Moment Retrieval [29.90001703587512]
  We present a series of experiments assessing how well the benchmark results reflect the true progress in solving the moment retrieval task. Our results indicate substantial biases in the popular datasets and unexpected behaviour of the state-of-the-art models. We suggest possible directions to improve the temporal sentence grounding in the future.
  arXiv  Detail & Related papers  (2020-09-01T10:07:23Z)
• Temporal Context Aggregation for Video Retrieval with Contrastive Learning [81.12514007044456]
  We propose TCA, a video representation learning framework that incorporates long-range temporal information between frame-level features. The proposed method shows a significant performance advantage (17% mAP on FIVR-200K) over state-of-the-art methods with video-level features.
  arXiv  Detail & Related papers  (2020-08-04T05:24:20Z)
• ZSTAD: Zero-Shot Temporal Activity Detection [107.63759089583382]
  We propose a novel task setting called zero-shot temporal activity detection (ZSTAD), where activities that have never been seen in training can still be detected. We design an end-to-end deep network based on R-C3D as the architecture for this solution. Experiments on both the THUMOS14 and the Charades datasets show promising performance in terms of detecting unseen activities.
  arXiv  Detail & Related papers  (2020-03-12T02:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.