Fine-grained Text-Video Retrieval with Frozen Image Encoders

• URL: http://arxiv.org/abs/2307.09972v1
• Date: Fri, 14 Jul 2023 02:57:00 GMT
• Title: Fine-grained Text-Video Retrieval with Frozen Image Encoders
• Authors: Zuozhuo Dai, Fangtao Shao, Qingkun Su, Zilong Dong, Siyu Zhu
• Abstract summary: We propose CrossTVR, a two-stage text-video retrieval architecture. In the first stage, we leverage existing TVR methods with cosine similarity network for efficient text/video candidate selection. In the second stage, we propose a novel decoupled video text cross attention module to capture fine-grained multimodal information in spatial and temporal dimensions.
• Score: 10.757101644990273
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State-of-the-art text-video retrieval (TVR) methods typically utilize CLIP and cosine similarity for efficient retrieval. Meanwhile, cross attention methods, which employ a transformer decoder to compute attention between each text query and all frames in a video, offer a more comprehensive interaction between text and videos. However, these methods lack important fine-grained spatial information as they directly compute attention between text and video-level tokens. To address this issue, we propose CrossTVR, a two-stage text-video retrieval architecture. In the first stage, we leverage existing TVR methods with cosine similarity network for efficient text/video candidate selection. In the second stage, we propose a novel decoupled video text cross attention module to capture fine-grained multimodal information in spatial and temporal dimensions. Additionally, we employ the frozen CLIP model strategy in fine-grained retrieval, enabling scalability to larger pre-trained vision models like ViT-G, resulting in improved retrieval performance. Experiments on text video retrieval datasets demonstrate the effectiveness and scalability of our proposed CrossTVR compared to state-of-the-art approaches.

Related papers

• RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter [77.0205013713008]
  Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries. To date, most state-of-the-art TVR methods learn image-to-video transfer learning based on large-scale pre-trained vision models. We propose a sparse-andcorrelated AdaPter (RAP) to fine-tune the pre-trained model with a few parameterized layers.
  arXiv  Detail & Related papers  (2024-05-29T19:23:53Z)
• An Empirical Study of Frame Selection for Text-to-Video Retrieval [62.28080029331507]
  Text-to-video retrieval (TVR) aims to find the most relevant video in a large video gallery given a query text. Existing methods typically select a subset of frames within a video to represent the video content for TVR. In this paper, we make the first empirical study of frame selection for TVR.
  arXiv  Detail & Related papers  (2023-11-01T05:03:48Z)
• Unified Coarse-to-Fine Alignment for Video-Text Retrieval [71.85966033484597]
  We propose a Unified Coarse-to-fine Alignment model, dubbed UCoFiA. Our model captures the cross-modal similarity information at different granularity levels. We apply the Sinkhorn-Knopp algorithm to normalize the similarities of each level before summing them.
  arXiv  Detail & Related papers  (2023-09-18T19:04:37Z)
• Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval [24.691270610091554]
  In this paper, we aim to learn semantically-enhanced representations purely from the video, so that the video representations can be computed offline and reused for different texts. We obtain state-of-the-art performances on three benchmark datasets, i.e., MSR-VTT, MSVD, and LSMDC.
  arXiv  Detail & Related papers  (2023-08-15T08:54:25Z)
• Video text tracking for dense and small text based on pp-yoloe-r and sort algorithm [0.9137554315375919]
  DSText is 1080 * 1920 and slicing the video frame into several areas will destroy the spatial correlation of text. For text detection, we adopt the PP-YOLOE-R which is proven effective in small object detection. For text detection, we use the sort algorithm for high inference speed.
  arXiv  Detail & Related papers  (2023-03-31T05:40:39Z)
• Contrastive Video-Language Learning with Fine-grained Frame Sampling [54.542962813921214]
  FineCo is an approach to better learn video and language representations with a fine-grained contrastive objective operating on video frames. It helps distil a video by selecting the frames that are semantically equivalent to the text, improving cross-modal correspondence.
  arXiv  Detail & Related papers  (2022-10-10T22:48:08Z)
• Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation [58.74791043631219]
  We propose a novel framework STswinCL that explores the complementary intra- and inter-video relations to boost segmentation performance. We extensively validate our approach on two public surgical video benchmarks, including EndoVis18 Challenge and CaDIS dataset. Experimental results demonstrate the promising performance of our method, which consistently exceeds previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-29T05:52:23Z)
• CLIP2Video: Mastering Video-Text Retrieval via Image CLIP [13.270902407320005]
  We present CLIP2Video network to transfer the image-language training model to video-text retrieval in an end-to-end manner. We conduct thorough ablation studies, and achieve state-of-the-art performance on text-to-video and video-to-text retrieval benchmarks.
  arXiv  Detail & Related papers  (2021-06-21T13:30:33Z)
• Video Corpus Moment Retrieval with Contrastive Learning [56.249924768243375]
  Video corpus moment retrieval (VCMR) is to retrieve a temporal moment that semantically corresponds to a given text query. We propose a Retrieval and Localization Network with Contrastive Learning (ReLoCLNet) for VCMR. Experimental results show that ReLoCLNet encodes text and video separately for efficiency, its retrieval accuracy is comparable with baselines adopting cross-modal interaction learning.
  arXiv  Detail & Related papers  (2021-05-13T12:54:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.