FocusedAD: Character-centric Movie Audio Description

• URL: http://arxiv.org/abs/2504.12157v2
• Date: Tue, 22 Apr 2025 12:25:51 GMT
• Title: FocusedAD: Character-centric Movie Audio Description
• Authors: Xiaojun Ye, Chun Wang, Yiren Song, Sheng Zhou, Liangcheng Li, Jiajun Bu,
• Abstract summary: Movie Audio Description (AD) aims to narrate visual content during dialogue-free segments, particularly benefiting blind and visually impaired (BVI) audiences.<n>We propose FocusedAD, a novel framework that delivers character-centric movie audio descriptions.
• Score: 20.257919582999133
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Movie Audio Description (AD) aims to narrate visual content during dialogue-free segments, particularly benefiting blind and visually impaired (BVI) audiences. Compared with general video captioning, AD demands plot-relevant narration with explicit character name references, posing unique challenges in movie understanding.To identify active main characters and focus on storyline-relevant regions, we propose FocusedAD, a novel framework that delivers character-centric movie audio descriptions. It includes: (i) a Character Perception Module(CPM) for tracking character regions and linking them to names; (ii) a Dynamic Prior Module(DPM) that injects contextual cues from prior ADs and subtitles via learnable soft prompts; and (iii) a Focused Caption Module(FCM) that generates narrations enriched with plot-relevant details and named characters. To overcome limitations in character identification, we also introduce an automated pipeline for building character query banks. FocusedAD achieves state-of-the-art performance on multiple benchmarks, including strong zero-shot results on MAD-eval-Named and our newly proposed Cinepile-AD dataset. Code and data will be released at https://github.com/Thorin215/FocusedAD .

Related papers

• DistinctAD: Distinctive Audio Description Generation in Contexts [62.58375366359421]
  We propose DistinctAD, a framework for generating Audio Descriptions that emphasize distinctiveness to produce better narratives. To address the domain gap, we introduce a CLIP-AD adaptation strategy that does not require additional AD corpora. In Stage-II, DistinctAD incorporates two key innovations: (i) a Contextual Expectation-Maximization Attention (EMA) module that reduces redundancy by extracting common bases from consecutive video clips, and (ii) an explicit distinctive word prediction loss that filters out repeated words in the context.
  arXiv  Detail & Related papers  (2024-11-27T09:54:59Z)
• AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description [92.72058446133468]
  Our objective is to generate Audio Descriptions (ADs) for both movies and TV series in a training-free manner. We use the power of off-the-shelf Visual-Language Models (VLMs) and Large Language Models (LLMs) Our approach, named AutoAD-Zero, demonstrates outstanding performance (even competitive with some models fine-tuned on ground truth ADs) in AD generation for both movies and TV series, achieving state-of-the-art CRITIC scores.
  arXiv  Detail & Related papers  (2024-07-22T17:59:56Z)
• Contextual AD Narration with Interleaved Multimodal Sequence [50.240534605090396]
  The task aims to generate descriptions of visual elements for visually impaired individuals to help them access long-form video content, like movies.<n>With video feature, text, character bank and context information as inputs, the generated ADs are able to correspond to the characters by name.<n>We propose to leverage pre-trained foundation models through a simple and unified framework to generate ADs.
  arXiv  Detail & Related papers  (2024-03-19T17:27:55Z)
• SOVC: Subject-Oriented Video Captioning [59.04029220586337]
  We propose a new video captioning task, Subject-Oriented Video Captioning (SOVC), which aims to allow users to specify the describing target via a bounding box. To support this task, we construct two subject-oriented video captioning datasets based on two widely used video captioning datasets.
  arXiv  Detail & Related papers  (2023-12-20T17:44:32Z)
• AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description [95.70092272297704]
  We develop a new model for automatically generating movie AD, given CLIP visual features of the frames, the cast list, and the temporal locations of the speech. We demonstrate how this improves over previous architectures for AD text generation in an apples-to-apples comparison.
  arXiv  Detail & Related papers  (2023-10-10T17:59:53Z)
• AutoAD: Movie Description in Context [91.98603496476215]
  This paper presents an automatic Audio Description (AD) model that ingests movies and outputs AD in text form. We leverage the power of pretrained foundation models, such as GPT and CLIP, and only train a mapping network that bridges the two models for visually-conditioned text generation.
  arXiv  Detail & Related papers  (2023-03-29T17:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.