Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach

• URL: http://arxiv.org/abs/2408.07249v2
• Date: Sat, 08 Mar 2025 13:41:10 GMT
• Title: Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach
• Authors: Zechen Bai, Tianjun Xiao, Tong He, Pichao Wang, Zheng Zhang, Thomas Brox, Mike Zheng Shou,
• Abstract summary: A key challenge in text-video retrieval (TVR) is the information asymmetry between video and text.<n>This paper introduces a data-centric framework to bridge this gap by enriching textual representations to better match the richness of video content.<n>We propose a query selection mechanism that identifies the most relevant and diverse queries, reducing computational cost while improving accuracy.
• Score: 56.610806615527885
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As online video content rapidly grows, the task of text-video retrieval (TVR) becomes increasingly important. A key challenge in TVR is the information asymmetry between video and text: videos are inherently richer in information, while their textual descriptions often capture only fragments of this complexity. This paper introduces a novel, data-centric framework to bridge this gap by enriching textual representations to better match the richness of video content. During training, videos are segmented into event-level clips and captioned to ensure comprehensive coverage. During retrieval, a large language model (LLM) generates semantically diverse queries to capture a broader range of possible matches. To enhance retrieval efficiency, we propose a query selection mechanism that identifies the most relevant and diverse queries, reducing computational cost while improving accuracy. Our method achieves state-of-the-art results across multiple benchmarks, demonstrating the power of data-centric approaches in addressing information asymmetry in TVR. This work paves the way for new research focused on leveraging data to improve cross-modal retrieval.

Related papers

• Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment [0.0]
  Long Video Question Answering (LVQA) is challenging due to the need for temporal reasoning and large-scale multimodal data processing. We introduce UMaT, a retrieval-augmented generation framework that efficiently processes extremely long videos. We show that UMaT outperforms existing methods in multimodal integration, long-form video understanding, and sparse information retrieval.
  arXiv  Detail & Related papers  (2025-03-12T05:28:24Z)
• RAPID: Retrieval-Augmented Parallel Inference Drafting for Text-Based Video Event Retrieval [2.9927319356868436]
  Existing methods for text-based video event retrieval focus heavily on object-level descriptions, overlooking the crucial role of contextual information. We propose a novel system called RAPID, which leverages advancements in Large Language Models (LLMs) and prompt-based learning to semantically correct user queries. Our system was validated for both speed and accuracy through participation in the Ho Chi Minh City AI Challenge 2024, where it successfully retrieved events from over 300 hours of video.
  arXiv  Detail & Related papers  (2025-01-27T18:45:07Z)
• SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis [52.050036778325094]
  We introduce SALOVA: Segment-Augmented Video Assistant, a novel video-LLM framework designed to enhance the comprehension of lengthy video content. We present a high-quality collection of 87.8K long videos, each densely captioned at the segment level to enable models to capture scene continuity and maintain rich context. Our framework mitigates the limitations of current video-LMMs by allowing for precise identification and retrieval of relevant video segments in response to queries.
  arXiv  Detail & Related papers  (2024-11-25T08:04:47Z)
• MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval [57.891157692501345]
  $textbfMultiVENT 2.0$ is a large-scale, multilingual event-centric video retrieval benchmark. It features a collection of more than 218,000 news videos and 3,906 queries targeting specific world events. Preliminary results show that state-of-the-art vision-language models struggle significantly with this task.
  arXiv  Detail & Related papers  (2024-10-15T13:56:34Z)
• Personalized Video Summarization using Text-Based Queries and Conditional Modeling [3.4447129363520337]
  This thesis explores enhancing video summarization by integrating text-based queries and conditional modeling. Evaluation metrics such as accuracy and F1-score assess the quality of the generated summaries.
  arXiv  Detail & Related papers  (2024-08-27T02:43:40Z)
• EA-VTR: Event-Aware Video-Text Retrieval [97.30850809266725]
  Event-Aware Video-Text Retrieval model achieves powerful video-text retrieval ability through superior video event awareness. EA-VTR can efficiently encode frame-level and video-level visual representations simultaneously, enabling detailed event content and complex event temporal cross-modal alignment.
  arXiv  Detail & Related papers  (2024-07-10T09:09:58Z)
• Text-Video Retrieval with Global-Local Semantic Consistent Learning [122.15339128463715]
  We propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL) GLSCL capitalizes on latent shared semantics across modalities for text-video retrieval. Our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost.
  arXiv  Detail & Related papers  (2024-05-21T11:59:36Z)
• SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval [11.548061962976321]
  We propose a novel Syntax-Hierarchy-Enhanced text-video retrieval method (SHE-Net) First, to facilitate a more fine-grained integration of visual content, we employ the text syntax hierarchy, which reveals the grammatical structure of text descriptions. Second, to further enhance the multi-modal interaction and alignment, we also utilize the syntax hierarchy to guide the similarity calculation.
  arXiv  Detail & Related papers  (2024-04-22T10:23:59Z)
• iRAG: Advancing RAG for Videos with an Incremental Approach [3.486835161875852]
  One-time, upfront conversion of all content in large corpus of videos into text descriptions entails high processing times. We propose an incremental RAG system called iRAG, which augments RAG with a novel incremental workflow to enable interactive querying of video data. iRAG is the first system to augment RAG with an incremental workflow to support efficient interactive querying of a large corpus of videos.
  arXiv  Detail & Related papers  (2024-04-18T16:38:02Z)
• Scaling Up Video Summarization Pretraining with Large Language Models [73.74662411006426]
  We introduce an automated and scalable pipeline for generating a large-scale video summarization dataset. We analyze the limitations of existing approaches and propose a new video summarization model that effectively addresses them. Our work also presents a new benchmark dataset that contains 1200 long videos each with high-quality summaries annotated by professionals.
  arXiv  Detail & Related papers  (2024-04-04T11:59:06Z)
• Improving Video Corpus Moment Retrieval with Partial Relevance Enhancement [72.7576395034068]
  Video Corpus Moment Retrieval (VCMR) is a new video retrieval task aimed at retrieving a relevant moment from a large corpus of untrimmed videos using a text query. We argue that effectively capturing the partial relevance between the query and video is essential for the VCMR task. For video retrieval, we introduce a multi-modal collaborative video retriever, generating different query representations for the two modalities. For moment localization, we propose the focus-then-fuse moment localizer, utilizing modality-specific gates to capture essential content.
  arXiv  Detail & Related papers  (2024-02-21T07:16:06Z)
• Text-Video Retrieval via Variational Multi-Modal Hypergraph Networks [25.96897989272303]
  Main obstacle for text-video retrieval is the semantic gap between the textual nature of queries and the visual richness of video content. We propose chunk-level text-video matching, where the query chunks are extracted to describe a specific retrieval unit. We formulate the chunk-level matching as n-ary correlations modeling between words of the query and frames of the video.
  arXiv  Detail & Related papers  (2024-01-06T09:38:55Z)
• Zero-shot Audio Topic Reranking using Large Language Models [42.774019015099704]
  Multimodal Video Search by Examples (MVSE) investigates using video clips as the query term for information retrieval. This work aims to compensate for any performance loss from this rapid archive search by examining reranking approaches. Performance is evaluated for topic-based retrieval on a publicly available video archive, the BBC Rewind corpus.
  arXiv  Detail & Related papers  (2023-09-14T11:13:36Z)
• Zero-Shot Video Moment Retrieval from Frozen Vision-Language Models [58.17315970207874]
  We propose a zero-shot method for adapting generalisable visual-textual priors from arbitrary VLM to facilitate moment-text alignment. Experiments conducted on three VMR benchmark datasets demonstrate the notable performance advantages of our zero-shot algorithm.
  arXiv  Detail & Related papers  (2023-09-01T13:06:50Z)
• Deep Learning for Video-Text Retrieval: a Review [13.341694455581363]
  Video-Text Retrieval (VTR) aims to search for the most relevant video related to the semantics in a given sentence. In this survey, we review and summarize over 100 research papers related to VTR.
  arXiv  Detail & Related papers  (2023-02-24T10:14:35Z)
• Hybrid Contrastive Quantization for Efficient Cross-View Video Retrieval [55.088635195893325]
  We propose the first quantized representation learning method for cross-view video retrieval, namely Hybrid Contrastive Quantization (HCQ) HCQ learns both coarse-grained and fine-grained quantizations with transformers, which provide complementary understandings for texts and videos. Experiments on three Web video benchmark datasets demonstrate that HCQ achieves competitive performance with state-of-the-art non-compressed retrieval methods.
  arXiv  Detail & Related papers  (2022-02-07T18:04:10Z)
• Video Corpus Moment Retrieval with Contrastive Learning [56.249924768243375]
  Video corpus moment retrieval (VCMR) is to retrieve a temporal moment that semantically corresponds to a given text query. We propose a Retrieval and Localization Network with Contrastive Learning (ReLoCLNet) for VCMR. Experimental results show that ReLoCLNet encodes text and video separately for efficiency, its retrieval accuracy is comparable with baselines adopting cross-modal interaction learning.
  arXiv  Detail & Related papers  (2021-05-13T12:54:39Z)
• Straight to the Point: Fast-forwarding Videos via Reinforcement Learning Using Textual Data [1.004766879203303]
  We present a novel methodology based on a reinforcement learning formulation to accelerate instructional videos. Our approach can adaptively select frames that are not relevant to convey the information without creating gaps in the final video. We propose a novel network, called Visually-guided Document Attention Network (VDAN), able to generate a highly discriminative embedding space.
  arXiv  Detail & Related papers  (2020-03-31T14:07:45Z)
• Convolutional Hierarchical Attention Network for Query-Focused Video Summarization [74.48782934264094]
  This paper addresses the task of query-focused video summarization, which takes user's query and a long video as inputs. We propose a method, named Convolutional Hierarchical Attention Network (CHAN), which consists of two parts: feature encoding network and query-relevance computing module. In the encoding network, we employ a convolutional network with local self-attention mechanism and query-aware global attention mechanism to learns visual information of each shot.
  arXiv  Detail & Related papers  (2020-01-31T04:30:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.