Composed Video Retrieval via Enriched Context and Discriminative Embeddings

• URL: http://arxiv.org/abs/2403.16997v1
• Date: Mon, 25 Mar 2024 17:59:03 GMT
• Title: Composed Video Retrieval via Enriched Context and Discriminative Embeddings
• Authors: Omkar Thawakar, Muzammal Naseer, Rao Muhammad Anwer, Salman Khan, Michael Felsberg, Mubarak Shah, Fahad Shahbaz Khan,
• Abstract summary: Composed video retrieval (CoVR) is a challenging problem in computer vision. We introduce a novel CoVR framework that leverages detailed language descriptions to explicitly encode query-specific contextual information. Our approach achieves gains as high as around 7% in terms of recall@K=1 score.
• Score: 118.66322242183249
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Composed video retrieval (CoVR) is a challenging problem in computer vision which has recently highlighted the integration of modification text with visual queries for more sophisticated video search in large databases. Existing works predominantly rely on visual queries combined with modification text to distinguish relevant videos. However, such a strategy struggles to fully preserve the rich query-specific context in retrieved target videos and only represents the target video using visual embedding. We introduce a novel CoVR framework that leverages detailed language descriptions to explicitly encode query-specific contextual information and learns discriminative embeddings of vision only, text only and vision-text for better alignment to accurately retrieve matched target videos. Our proposed framework can be flexibly employed for both composed video (CoVR) and image (CoIR) retrieval tasks. Experiments on three datasets show that our approach obtains state-of-the-art performance for both CovR and zero-shot CoIR tasks, achieving gains as high as around 7% in terms of recall@K=1 score. Our code, models, detailed language descriptions for WebViD-CoVR dataset are available at \url{https://github.com/OmkarThawakar/composed-video-retrieval}

Related papers

• Vript: A Video Is Worth Thousands of Words [54.815686588378156]
  Vript is an annotated corpus of 12K high-resolution videos, offering detailed, dense, and script-like captions for over 420K clips. Each clip has a caption of 145 words, which is over 10x longer than most video-text datasets. Vript is a powerful model capable of end-to-end generation of dense and detailed captions for long videos.
  arXiv  Detail & Related papers  (2024-06-10T06:17:55Z)
• Videoprompter: an ensemble of foundational models for zero-shot video understanding [113.92958148574228]
  Vision-language models (VLMs) classify the query video by calculating a similarity score between the visual features and text-based class label representations. We propose a framework which combines pre-trained discrimi VLMs with pre-trained generative video-to-text and text-to-text models.
  arXiv  Detail & Related papers  (2023-10-23T19:45:46Z)
• CoVR-2: Automatic Data Construction for Composed Video Retrieval [59.854331104466254]
  Composed Image Retrieval (CoIR) has recently gained popularity as a task that considers both text and image queries together. We propose a scalable automatic dataset creation methodology that generates triplets given video-caption pairs. We also expand the scope of the task to include composed video retrieval (CoVR)
  arXiv  Detail & Related papers  (2023-08-28T17:55:33Z)
• ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Models [6.073813559982129]
  Video retrieval involves retrieving the ground truth video from the video database given a text caption or vice-versa. We evaluate the compositional and syntactic understanding of video retrieval models on standard benchmarks such as MSRVTT, MSVD and DIDEMO. Our experiments reveal that actions and syntax play a minor role compared to objects & attributes in video understanding.
  arXiv  Detail & Related papers  (2023-06-28T20:06:36Z)
• A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension [49.74647080936875]
  We introduce a large-scale and cross-modal Video Retrieval dataset with text reading comprehension, TextVR. The proposed TextVR requires one unified cross-modal model to recognize and comprehend texts, relate them to the visual context, and decide what text semantic information is vital for the video retrieval task.
  arXiv  Detail & Related papers  (2023-05-05T08:00:14Z)
• VLG: General Video Recognition with Web Textual Knowledge [47.3660792813967]
  We focus on the general video recognition (GVR) problem of solving different recognition tasks within a unified framework. By leveraging semantic knowledge from noisy text descriptions crawled from the Internet, we present a unified visual-linguistic framework (VLG) Our VLG is first pre-trained on video and language datasets to learn a shared feature space, and then devises a flexible bi-modal attention head to collaborate high-level semantic concepts under different settings.
  arXiv  Detail & Related papers  (2022-12-03T15:46:49Z)
• Are All Combinations Equal? Combining Textual and Visual Features with Multiple Space Learning for Text-Based Video Retrieval [9.537322316673617]
  We investigate how to optimally combine multiple diverse textual and visual features into feature pairs. To learn these representations our proposed network architecture is trained by following a multiple space learning procedure.
  arXiv  Detail & Related papers  (2022-11-21T11:08:13Z)
• Contrastive Video-Language Learning with Fine-grained Frame Sampling [54.542962813921214]
  FineCo is an approach to better learn video and language representations with a fine-grained contrastive objective operating on video frames. It helps distil a video by selecting the frames that are semantically equivalent to the text, improving cross-modal correspondence.
  arXiv  Detail & Related papers  (2022-10-10T22:48:08Z)
• Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval [41.420760047617506]
  Cross-modal representation learning projects both videos and sentences into common spaces for semantic similarity. Inspired by the reading strategy of humans, we propose a Reading-strategy Inspired Visual Representation Learning (RIVRL) to represent videos. Our model RIVRL achieves a new state-of-the-art on TGIF and VATEX.
  arXiv  Detail & Related papers  (2022-01-23T03:38:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.