REFRAMED: Towards Realistic Audio Description Generation for Movies

• URL: http://arxiv.org/abs/2608.09765v1
• Date: Mon, 10 Aug 2026 15:55:56 GMT
• Title: REFRAMED: Towards Realistic Audio Description Generation for Movies
• Abstract summary: Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences.<n>Existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified.<n>We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it.
• Score: 53.11641004249926
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences. Unlike standard video captioning, AD is a structured editorial task: descriptions must be inserted into gaps in dialogue and must convey only what is needed to understand the narrative being told. However, existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified, reducing the task to clip-level captioning. They further rely on noisy transcription and alignment pipelines, and lack the rich parallel data required for modeling narrative context. We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it. To support this, we present REFRAMED, a high-quality dataset of 2,023 videos that span 3,302 scenes from 206 movies, with professional AD transcripts (both American and British versions), professional subtitles and aligned screenplays. We also provide a manually curated challenge set that pairs full movies with multiple AD references, together with evaluation protocols that leverage dialogue gaps and multi-reference comparisons. Experiments with state-of-the-art AD systems and multimodal LLMs show that they outperform trivial baselines but fall far short of expert human performance. Our dataset and benchmark establish a new foundation for research on video understanding.

Related papers

• StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description [2.26752781534698]
  We propose a training-free framework for story-aware long-form audio descriptions.<n>StoryTeller maintains a verified narrative memory that carries forward story-relevant information across scenes.
  arXiv  Detail & Related papers  (2026-07-13T16:50:03Z)
• DANTE-AD: Dual-Vision Attention Network for Long-Term Audio Description [19.14915136673913]
  We introduce DANTE-AD, an enhanced video description model leveraging a dual-vision Transformer-based architecture.<n>We propose a novel, state-of-the-art method for sequential cross-attention to achieve contextual grounding for fine-grained audio description generation.
  arXiv  Detail & Related papers  (2025-03-31T13:49:43Z)
• DistinctAD: Distinctive Audio Description Generation in Contexts [62.58375366359421]
  We propose DistinctAD, a framework for generating Audio Descriptions that emphasize distinctiveness to produce better narratives.<n>To address the domain gap, we introduce a CLIP-AD adaptation strategy that does not require additional AD corpora.<n>In Stage-II, DistinctAD incorporates two key innovations: (i) a Contextual Expectation-Maximization Attention (EMA) module that reduces redundancy by extracting common bases from consecutive video clips, and (ii) an explicit distinctive word prediction loss that filters out repeated words in the context.
  arXiv  Detail & Related papers  (2024-11-27T09:54:59Z)
• AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description [92.72058446133468]
  Our objective is to generate Audio Descriptions (ADs) for both movies and TV series in a training-free manner. We use the power of off-the-shelf Visual-Language Models (VLMs) and Large Language Models (LLMs) Our approach, named AutoAD-Zero, demonstrates outstanding performance (even competitive with some models fine-tuned on ground truth ADs) in AD generation for both movies and TV series, achieving state-of-the-art CRITIC scores.
  arXiv  Detail & Related papers  (2024-07-22T17:59:56Z)
• Contextual AD Narration with Interleaved Multimodal Sequence [50.240534605090396]
  The task aims to generate descriptions of visual elements for visually impaired individuals to help them access long-form video content, like movies.<n>With video feature, text, character bank and context information as inputs, the generated ADs are able to correspond to the characters by name.<n>We propose to leverage pre-trained foundation models through a simple and unified framework to generate ADs.
  arXiv  Detail & Related papers  (2024-03-19T17:27:55Z)
• AutoAD: Movie Description in Context [91.98603496476215]
  This paper presents an automatic Audio Description (AD) model that ingests movies and outputs AD in text form. We leverage the power of pretrained foundation models, such as GPT and CLIP, and only train a mapping network that bridges the two models for visually-conditioned text generation.
  arXiv  Detail & Related papers  (2023-03-29T17:59:58Z)
• QuerYD: A video dataset with high-quality text and audio narrations [85.6468286746623]
  We introduce QuerYD, a new large-scale dataset for retrieval and event localisation in video. A unique feature of our dataset is the availability of two audio tracks for each video: the original audio, and a high-quality spoken description. The dataset is based on YouDescribe, a volunteer project that assists visually-impaired people by attaching voiced narrations to existing YouTube videos.
  arXiv  Detail & Related papers  (2020-11-22T17:33:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.