ViSeRet: A simple yet effective approach to moment retrieval via fine-grained video segmentation

• URL: http://arxiv.org/abs/2110.05146v2
• Date: Tue, 12 Oct 2021 10:29:37 GMT
• Title: ViSeRet: A simple yet effective approach to moment retrieval via fine-grained video segmentation
• Authors: Aiden Seungjoon Lee, Hanseok Oh, Minjoon Seo
• Abstract summary: This paper presents the 1st place solution to the video retrieval track of the ICCV VALUE Challenge 2021. We present a simple yet effective approach to jointly tackle two video-text retrieval tasks. We create an ensemble model that achieves the new state-of-the-art performance on all four datasets.
• Score: 6.544437737391409
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video-text retrieval has many real-world applications such as media analytics, surveillance, and robotics. This paper presents the 1st place solution to the video retrieval track of the ICCV VALUE Challenge 2021. We present a simple yet effective approach to jointly tackle two video-text retrieval tasks (video retrieval and video corpus moment retrieval) by leveraging the model trained only on the video retrieval task. In addition, we create an ensemble model that achieves the new state-of-the-art performance on all four datasets (TVr, How2r, YouCook2r, and VATEXr) presented in the VALUE Challenge.

Related papers

• MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrieval [57.891157692501345]
  $textbfMultiVENT 2.0$ is a large-scale, multilingual event-centric video retrieval benchmark. It features a collection of more than 218,000 news videos and 3,906 queries targeting specific world events. Preliminary results show that state-of-the-art vision-language models struggle significantly with this task.
  arXiv  Detail & Related papers  (2024-10-15T13:56:34Z)
• Scaling Up Video Summarization Pretraining with Large Language Models [73.74662411006426]
  We introduce an automated and scalable pipeline for generating a large-scale video summarization dataset. We analyze the limitations of existing approaches and propose a new video summarization model that effectively addresses them. Our work also presents a new benchmark dataset that contains 1200 long videos each with high-quality summaries annotated by professionals.
  arXiv  Detail & Related papers  (2024-04-04T11:59:06Z)
• Composed Video Retrieval via Enriched Context and Discriminative Embeddings [118.66322242183249]
  Composed video retrieval (CoVR) is a challenging problem in computer vision. We introduce a novel CoVR framework that leverages detailed language descriptions to explicitly encode query-specific contextual information. Our approach achieves gains as high as around 7% in terms of recall@K=1 score.
  arXiv  Detail & Related papers  (2024-03-25T17:59:03Z)
• Multi-event Video-Text Retrieval [33.470499262092105]
  Video-Text Retrieval (VTR) is a crucial multi-modal task in an era of massive video-text data on the Internet. We introduce the Multi-event Video-Text Retrieval (MeVTR) task, addressing scenarios in which each video contains multiple different events. We present a simple model, Me-Retriever, which incorporates key event video representation and a new MeVTR loss for the MeVTR task.
  arXiv  Detail & Related papers  (2023-08-22T16:32:46Z)
• VideoOFA: Two-Stage Pre-Training for Video-to-Text Generation [43.90887811621963]
  We propose a new two-stage pre-training framework for video-to-text generation tasks such as video captioning and question answering. A generative encoder-decoder model is first jointly pre-trained on massive image-language data to learn fundamental concepts. As a result, our VideoOFA model achieves new state-the-art performance on four Video Captioning benchmarks.
  arXiv  Detail & Related papers  (2023-05-04T23:27:21Z)
• Hierarchical Video-Moment Retrieval and Step-Captioning [68.4859260853096]
  HiREST consists of 3.4K text-video pairs from an instructional video dataset. Our hierarchical benchmark consists of video retrieval, moment retrieval, and two novel moment segmentation and step captioning tasks.
  arXiv  Detail & Related papers  (2023-03-29T02:33:54Z)
• Deep Learning for Video-Text Retrieval: a Review [13.341694455581363]
  Video-Text Retrieval (VTR) aims to search for the most relevant video related to the semantics in a given sentence. In this survey, we review and summarize over 100 research papers related to VTR.
  arXiv  Detail & Related papers  (2023-02-24T10:14:35Z)
• Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval [41.420760047617506]
  Cross-modal representation learning projects both videos and sentences into common spaces for semantic similarity. Inspired by the reading strategy of humans, we propose a Reading-strategy Inspired Visual Representation Learning (RIVRL) to represent videos. Our model RIVRL achieves a new state-of-the-art on TGIF and VATEX.
  arXiv  Detail & Related papers  (2022-01-23T03:38:37Z)
• BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions [38.843518809230524]
  We introduce a novel pretext task dubbed Multiple Choice Questions (MCQ) A module BridgeFormer is trained to answer the "questions" constructed by the text features via resorting to the video features. In the form of questions and answers, the semantic associations between local video-text features can be properly established. Our method outperforms state-of-the-art methods on the popular text-to-video retrieval task in five datasets.
  arXiv  Detail & Related papers  (2022-01-13T09:33:54Z)
• Video Corpus Moment Retrieval with Contrastive Learning [56.249924768243375]
  Video corpus moment retrieval (VCMR) is to retrieve a temporal moment that semantically corresponds to a given text query. We propose a Retrieval and Localization Network with Contrastive Learning (ReLoCLNet) for VCMR. Experimental results show that ReLoCLNet encodes text and video separately for efficiency, its retrieval accuracy is comparable with baselines adopting cross-modal interaction learning.
  arXiv  Detail & Related papers  (2021-05-13T12:54:39Z)
• Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval [80.7397409377659]
  We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. We show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks.
  arXiv  Detail & Related papers  (2021-04-01T17:48:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.