Movie101v2: Improved Movie Narration Benchmark

• URL: http://arxiv.org/abs/2404.13370v2
• Date: Fri, 18 Oct 2024 16:44:05 GMT
• Title: Movie101v2: Improved Movie Narration Benchmark
• Authors: Zihao Yue, Yepeng Zhang, Ziheng Wang, Qin Jin,
• Abstract summary: Automatic movie narration aims to generate video-aligned plot descriptions to assist visually impaired audiences. We introduce Movie101v2, a large-scale, bilingual dataset with enhanced data quality specifically designed for movie narration. Based on our new benchmark, we baseline a range of large vision-language models, including GPT-4V, and conduct an in-depth analysis of the challenges in narration generation.
• Score: 53.54176725112229
• License:
• Abstract: Automatic movie narration aims to generate video-aligned plot descriptions to assist visually impaired audiences. Unlike standard video captioning, it involves not only describing key visual details but also inferring plots that unfold across multiple movie shots, presenting distinct and complex challenges. To advance this field, we introduce Movie101v2, a large-scale, bilingual dataset with enhanced data quality specifically designed for movie narration. Revisiting the task, we propose breaking down the ultimate goal of automatic movie narration into three progressive stages, offering a clear roadmap with corresponding evaluation metrics. Based on our new benchmark, we baseline a range of large vision-language models, including GPT-4V, and conduct an in-depth analysis of the challenges in narration generation. Our findings highlight that achieving applicable movie narration generation is a fascinating goal that requires significant research.

Related papers

• StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification [6.762705315042178]
  Long video description introduces new challenges, such as plot-level consistency across descriptions. We propose StoryTeller, a system for generating dense descriptions of long videos, incorporating both low-level visual concepts and high-level plot information.
  arXiv  Detail & Related papers  (2024-11-11T15:51:48Z)
• DiscoGraMS: Enhancing Movie Screen-Play Summarization using Movie Character-Aware Discourse Graph [6.980991481207376]
  We introduce DiscoGraMS, a novel resource that represents movie scripts as a movie character-aware discourse graph (CaD Graph) The model aims to preserve all salient information, offering a more comprehensive and faithful representation of the screenplay's content.
  arXiv  Detail & Related papers  (2024-10-18T17:56:11Z)
• ScreenWriter: Automatic Screenplay Generation and Movie Summarisation [55.20132267309382]
  Video content has driven demand for textual descriptions or summaries that allow users to recall key plot points or get an overview without watching. We propose the task of automatic screenplay generation, and a method, ScreenWriter, that operates only on video and produces output which includes dialogue, speaker names, scene breaks, and visual descriptions. ScreenWriter introduces a novel algorithm to segment the video into scenes based on the sequence of visual vectors, and a novel method for the challenging problem of determining character names, based on a database of actors' faces.
  arXiv  Detail & Related papers  (2024-10-17T07:59:54Z)
• MovieSum: An Abstractive Summarization Dataset for Movie Screenplays [11.318175666743656]
  We present a new dataset, MovieSum, for abstractive summarization of movie screenplays. This dataset comprises 2200 movie screenplays accompanied by their Wikipedia plot summaries.
  arXiv  Detail & Related papers  (2024-08-12T16:43:09Z)
• MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images [92.13079696503803]
  We present MovieFactory, a framework to generate cinematic-picture (3072$times$1280), film-style (multi-scene), and multi-modality (sounding) movies. Our approach empowers users to create captivating movies with smooth transitions using simple text inputs.
  arXiv  Detail & Related papers  (2023-06-12T17:31:23Z)
• Movie101: A New Movie Understanding Benchmark [47.24519006577205]
  We construct a large-scale Chinese movie benchmark, named Movie101. We propose a new metric called Movie Narration Score (MNScore) for movie narrating evaluation. For both two tasks, our proposed methods well leverage external knowledge and outperform carefully designed baselines.
  arXiv  Detail & Related papers  (2023-05-20T08:43:51Z)
• Movie Summarization via Sparse Graph Construction [65.16768855902268]
  We propose a model that identifies TP scenes by building a sparse movie graph that represents relations between scenes and is constructed using multimodal information. According to human judges, the summaries created by our approach are more informative and complete, and receive higher ratings, than the outputs of sequence-based models and general-purpose summarization algorithms.
  arXiv  Detail & Related papers  (2020-12-14T13:54:34Z)
• Condensed Movies: Story Based Retrieval with Contextual Embeddings [83.73479493450009]
  We create the Condensed Movies dataset (CMD) consisting of the key scenes from over 3K movies. The dataset is scalable, obtained automatically from YouTube, and is freely available for anybody to download and use. We provide a deep network baseline for text-to-video retrieval on our dataset, combining character, speech and visual cues into a single video embedding.
  arXiv  Detail & Related papers  (2020-05-08T17:55:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.