MovieCLIP: Visual Scene Recognition in Movies

• URL: http://arxiv.org/abs/2210.11065v2
• Date: Sun, 23 Oct 2022 01:25:13 GMT
• Title: MovieCLIP: Visual Scene Recognition in Movies
• Authors: Digbalay Bose, Rajat Hebbar, Krishna Somandepalli, Haoyang Zhang, Yin Cui, Kree Cole-McLaughlin, Huisheng Wang, Shrikanth Narayanan
• Abstract summary: Existing visual scene datasets in movies have limited and don't consider the visual scene transition within movie clips. In this work, we address the problem of visual scene recognition in movies by first automatically curating a new and extensive movie-centric taxonomy. Instead of manual annotations which can be expensive, we use CLIP to weakly label 1.12 million shots from 32K movie clips based on our proposed taxonomy.
• Score: 38.90153620199725
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Longform media such as movies have complex narrative structures, with events spanning a rich variety of ambient visual scenes. Domain specific challenges associated with visual scenes in movies include transitions, person coverage, and a wide array of real-life and fictional scenarios. Existing visual scene datasets in movies have limited taxonomies and don't consider the visual scene transition within movie clips. In this work, we address the problem of visual scene recognition in movies by first automatically curating a new and extensive movie-centric taxonomy of 179 scene labels derived from movie scripts and auxiliary web-based video datasets. Instead of manual annotations which can be expensive, we use CLIP to weakly label 1.12 million shots from 32K movie clips based on our proposed taxonomy. We provide baseline visual models trained on the weakly labeled dataset called MovieCLIP and evaluate them on an independent dataset verified by human raters. We show that leveraging features from models pretrained on MovieCLIP benefits downstream tasks such as multi-label scene and genre classification of web videos and movie trailers.

Related papers

• ScreenWriter: Automatic Screenplay Generation and Movie Summarisation [55.20132267309382]
  Video content has driven demand for textual descriptions or summaries that allow users to recall key plot points or get an overview without watching. We propose the task of automatic screenplay generation, and a method, ScreenWriter, that operates only on video and produces output which includes dialogue, speaker names, scene breaks, and visual descriptions. ScreenWriter introduces a novel algorithm to segment the video into scenes based on the sequence of visual vectors, and a novel method for the challenging problem of determining character names, based on a database of actors' faces.
  arXiv  Detail & Related papers  (2024-10-17T07:59:54Z)
• Movie101v2: Improved Movie Narration Benchmark [53.54176725112229]
  Automatic movie narration aims to generate video-aligned plot descriptions to assist visually impaired audiences. We introduce Movie101v2, a large-scale, bilingual dataset with enhanced data quality specifically designed for movie narration. Based on our new benchmark, we baseline a range of large vision-language models, including GPT-4V, and conduct an in-depth analysis of the challenges in narration generation.
  arXiv  Detail & Related papers  (2024-04-20T13:15:27Z)
• Select and Summarize: Scene Saliency for Movie Script Summarization [11.318175666743656]
  We introduce a scene saliency dataset that consists of human-annotated salient scenes for 100 movies. We propose a two-stage abstractive summarization approach which first identifies the salient scenes in script and then generates a summary using only those scenes.
  arXiv  Detail & Related papers  (2024-04-04T16:16:53Z)
• MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learning [54.73173491543553]
  MoviePuzzle is a novel challenge that targets visual narrative reasoning and holistic movie understanding. To tackle this quandary, we put forth MoviePuzzle task that amplifies the temporal feature learning and structure learning of video models. Our approach outperforms existing state-of-the-art methods on the MoviePuzzle benchmark.
  arXiv  Detail & Related papers  (2023-06-04T03:51:54Z)
• Movies2Scenes: Using Movie Metadata to Learn Scene Representation [8.708989357658501]
  We propose a novel contrastive learning approach that uses movie metadata to learn a general-purpose scene representation. Specifically, we use movie metadata to define a measure of movie similarity, and use it during contrastive learning to limit our search for positive scene-pairs. Our learned scene representation consistently outperforms existing state-of-the-art methods on a diverse set of tasks evaluated using multiple benchmark datasets.
  arXiv  Detail & Related papers  (2022-02-22T03:31:33Z)
• Multilevel profiling of situation and dialogue-based deep networks for movie genre classification using movie trailers [7.904790547594697]
  We propose a novel multi-modality: situation, dialogue, and metadata-based movie genre classification framework. We develop the English movie trailer dataset (EMTD), which contains 2000 Hollywood movie trailers belonging to five popular genres.
  arXiv  Detail & Related papers  (2021-09-14T07:33:56Z)
• Condensed Movies: Story Based Retrieval with Contextual Embeddings [83.73479493450009]
  We create the Condensed Movies dataset (CMD) consisting of the key scenes from over 3K movies. The dataset is scalable, obtained automatically from YouTube, and is freely available for anybody to download and use. We provide a deep network baseline for text-to-video retrieval on our dataset, combining character, speech and visual cues into a single video embedding.
  arXiv  Detail & Related papers  (2020-05-08T17:55:03Z)
• A Local-to-Global Approach to Multi-modal Movie Scene Segmentation [95.34033481442353]
  We build a large-scale video dataset MovieScenes, which contains 21K annotated scene segments from 150 movies. We propose a local-to-global scene segmentation framework, which integrates multi-modal information across three levels, i.e. clip, segment, and movie. Our experiments show that the proposed network is able to segment a movie into scenes with high accuracy, consistently outperforming previous methods.
  arXiv  Detail & Related papers  (2020-04-06T13:58:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.