From Visual Cues to Spoken Narration: Rethinking Audio Description

• URL: http://arxiv.org/abs/2609.01725v1
• Date: Tue, 01 Sep 2026 18:00:50 GMT
• Title: From Visual Cues to Spoken Narration: Rethinking Audio Description
• Abstract summary: We propose Cue2Narrate, a two-stage pipeline that jointly predicts what and when to narrate in longer untrimmed movie clips.<n>A dual-head audio-visual localizer predicts two temporally distinct windows per AD utterance.<n>A LoRA-adapted VLM then generates concise ADs from the predicted visual evidence.
• Score: 56.77584893970582
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Audio Description (AD) provides spoken narration of visual events during dialogue gaps, making movies accessible to visually impaired audiences. The problem requires determining both what (which visual event) and when (position for inserting the AD) to narrate, to achieve the best user experience. Prior work has largely reduced the problem to video captioning of pre-segmented video clips, i.e., what is largely predefined and when is ignored entirely. We propose Cue2Narrate, a two-stage pipeline that jointly predicts what and when to narrate in longer untrimmed movie clips. A dual-head audio-visual localizer predicts two temporally distinct windows per AD utterance: a visual cue window and a spoken narration window. A LoRA-adapted VLM then generates concise ADs from the predicted visual evidence, trained with a Description Ranking Loss that ranks captions (negative samples) of the same frames lower than the GT AD. To benchmark this new problem statement, we introduce the LongLSMDC benchmark with up to 8-min movie clips (~6.5min on average). On LongLSMDC, Cue2Narrate outperforms video-only and audio-only localization baselines by 5--12 points in avg. mAP. Under both predicted- and GT-window evaluation, Cue2Narrate improves AD generation over the corresponding fine-tuned base VLM. These results establish the first benchmark for multi-segment AD generation on long-form clips. Data & Code: https://github.com/multimodal-ai-lab/Cue2Narrate

Related papers

• Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding [59.773752487686444]
  Long-video understanding on edge devices must reason over hours of content under tight compute and bandwidth budgets.<n>We propose Caption-once, Frames-onDemand, a budget-aware edge-cloud agentic framework.<n> Experiments on long-video benchmarks demonstrate strong accuracy-efficiency trade-offs while substantially reducing online visual processing.
  arXiv  Detail & Related papers  (2026-09-10T17:53:59Z)
• StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videos [16.585176367435682]
  We introduce StrAD, a benchmark for long-form AD generation on full-length videos spanning diverse genres such as movies, documentaries, short films, performances, and video games.<n>Our approach processes full-length videos with a sliding window, inserting ADs into existing transcripts without ground-truth timestamps.<n>On the segment-level task with given timestamps, our fine-tuned StrAD-FT sets the state of the art on CMD-AD with 36.3 CIDEr (+10.0 over Shot-by-shot), establishes a reference point on StrAD (51.0 CIDEr), and remains competitive on MAD-E
  arXiv  Detail & Related papers  (2026-08-12T19:39:17Z)
• REFRAMED: Towards Realistic Audio Description Generation for Movies [53.11641004249926]
  Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences.<n>Existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified.<n>We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it.
  arXiv  Detail & Related papers  (2026-08-10T15:55:56Z)
• What You See is What You Ask: Evaluating Audio Descriptions [27.76958202277314]
  We propose ADQA, a QA benchmark that evaluates ADs at the level of few-minute long, coherent video segments.<n>We show that current AD generation methods lag far behind human-authored ADs.
  arXiv  Detail & Related papers  (2025-10-01T12:14:15Z)
• Exploiting Temporal Audio-Visual Correlation Embedding for Audio-Driven One-Shot Talking Head Animation [62.218932509432314]
  Inherently, the temporal relationship of adjacent audio clips is highly correlated with that of the corresponding adjacent video frames.<n>We learn audio-visual correlations and integrate the correlations to help enhance feature representation and regularize final generation.
  arXiv  Detail & Related papers  (2025-04-08T07:23:28Z)
• Audio-centric Video Understanding Benchmark without Text Shortcut [49.01648001666229]
  Audio often serves as an auxiliary modality in video understanding tasks of audio-visual large language models (LLMs)<n>This paper proposes an audio-centric video understanding benchmark (AVUT) to evaluate the video comprehension capabilities of multimodal LLMs.
  arXiv  Detail & Related papers  (2025-03-25T16:28:24Z)
• AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? [65.49972312524724]
  multimodal large language models (MLLMs) have expanded their capabilities to include vision and audio modalities.<n>Our proposed DeafTest reveals that MLLMs often struggle with simple tasks humans find trivial.<n>We introduce AV-Odyssey Bench, a comprehensive audio-visual benchmark designed to assess whether those MLLMs can truly understand the audio-visual information.
  arXiv  Detail & Related papers  (2024-12-03T17:41:23Z)
• Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language [77.33458847943528]
  We present DenseAV, a novel dual encoder grounding architecture that learns high-resolution, semantically meaningful, and audio-visually aligned features solely through watching videos. We show that DenseAV can discover the meaning'' of words and the location'' of sounds without explicit localization supervision.
  arXiv  Detail & Related papers  (2024-06-09T03:38:21Z)
• Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models [25.660343393359565]
  This paper proposes a fine-grained audio-visual joint representation (FAVOR) learning framework for multimodal large language models (LLM) FAVOR simultaneously perceive speech and audio events in the audio input stream and images or videos in the visual input stream, at the frame level. An interactive demo of FAVOR is available at https://github.com/BriansIDP/AudioVisualLLM.git, and the training code and model checkpoints will be released soon.
  arXiv  Detail & Related papers  (2023-10-09T17:00:20Z)
• Dual-path Attention is All You Need for Audio-Visual Speech Extraction [34.7260610874298]
  We propose a new way to fuse audio-visual features. The proposed algorithm incorporates the visual features as an additional feature stream. Results show we achieve superior results compared with other time-domain based audio-visual fusion models.
  arXiv  Detail & Related papers  (2022-07-09T07:27:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.