It's Just Another Day: Unique Video Captioning by Discriminative Prompting

• URL: http://arxiv.org/abs/2410.11702v1
• Date: Tue, 15 Oct 2024 15:41:49 GMT
• Title: It's Just Another Day: Unique Video Captioning by Discriminative Prompting
• Authors: Toby Perrett, Tengda Han, Dima Damen, Andrew Zisserman,
• Abstract summary: Given multiple clips with the same caption, we generate a new caption for each clip that uniquely identifies it. We demonstrate that captions generated by CDP improve text-to-video R@1 by 15% for egocentric videos and 10% in timeloop movies.
• Score: 70.99367779336256
• License:
• Abstract: Long videos contain many repeating actions, events and shots. These repetitions are frequently given identical captions, which makes it difficult to retrieve the exact desired clip using a text search. In this paper, we formulate the problem of unique captioning: Given multiple clips with the same caption, we generate a new caption for each clip that uniquely identifies it. We propose Captioning by Discriminative Prompting (CDP), which predicts a property that can separate identically captioned clips, and use it to generate unique captions. We introduce two benchmarks for unique captioning, based on egocentric footage and timeloop movies - where repeating actions are common. We demonstrate that captions generated by CDP improve text-to-video R@1 by 15% for egocentric videos and 10% in timeloop movies.

Related papers

• AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark [73.62572976072578]
  We propose AuroraCap, a video captioner based on a large multimodal model. We implement the token merging strategy, reducing the number of input visual tokens. AuroraCap shows superior performance on various video and image captioning benchmarks.
  arXiv  Detail & Related papers  (2024-10-04T00:13:54Z)
• MICap: A Unified Model for Identity-aware Movie Descriptions [16.287294191608893]
  We present a new single stage approach that can seamlessly switch between id-aware caption generation or FITB when given a caption with blanks. Our model, Movie-Identity Captioner (MICap), uses a shared auto-regressive decoder that benefits from training with FITB and full-caption generation objectives.
  arXiv  Detail & Related papers  (2024-05-19T08:54:12Z)
• HowToCaption: Prompting LLMs to Transform Video Annotations at Scale [72.69268311756082]
  We propose to leverage the capabilities of large language models (LLMs) to obtain high-quality video descriptions aligned with videos at scale. We introduce a prompting method that is able to take into account a longer text of subtitles, allowing us to capture the contextual information beyond one single sentence. We apply our method to the subtitles of the HowTo100M dataset, creating a new large-scale dataset, HowToCaption.
  arXiv  Detail & Related papers  (2023-10-07T19:32:55Z)
• Towards Generating Diverse Audio Captions via Adversarial Training [33.76154801580643]
  We propose a conditional generative adversarial network (C-GAN) to improve diversity of audio captioning systems. A caption generator and two hybrid discriminators compete and are learned jointly, where the caption generator can be any standard encoder-decoder captioning model used to generate captions. The results show that our proposed model can generate captions with better diversity as compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-12-05T05:06:19Z)
• Fine-grained Image Captioning with CLIP Reward [104.71533106301598]
  We propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function. We also propose a simple finetuning strategy of the CLIP text encoder to improve grammar that does not require extra text annotation. In experiments on text-to-image retrieval and FineCapEval, the proposed CLIP-guided model generates more distinctive captions than the CIDEr-optimized model.
  arXiv  Detail & Related papers  (2022-05-26T02:46:09Z)
• Learning Audio-Video Modalities from Image Captions [62.772232865072745]
  A major challenge in text-video and text-audio retrieval is the lack of large-scale training data. We propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort. We show that training a multimodal transformed based model on this data achieves competitive performance on video retrieval and video captioning, matching or even outperforming HowTo100M pretraining with 20x fewer clips.
  arXiv  Detail & Related papers  (2022-04-01T19:48:18Z)
• Syntax Customized Video Captioning by Imitating Exemplar Sentences [90.98221715705435]
  We introduce a new task of Syntax Customized Video Captioning (SCVC) SCVC aims to generate one caption which not only semantically describes the video contents but also syntactically imitates the given exemplar sentence. We demonstrate our model capability to generate syntax-varied and semantics-coherent video captions.
  arXiv  Detail & Related papers  (2021-12-02T09:08:09Z)
• Diverse and Styled Image Captioning Using SVD-Based Mixture of Recurrent Experts [5.859294565508523]
  A new captioning model is developed including an image encoder to extract the features, a mixture of recurrent networks to embed the set of extracted features to a set of words, and a sentence generator that combines the obtained words as a stylized sentence. We show that the proposed captioning model can generate a diverse and stylized image captions without the necessity of extra-labeling.
  arXiv  Detail & Related papers  (2020-07-07T11:00:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.