StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videos

• URL: http://arxiv.org/abs/2608.12549v1
• Date: Wed, 12 Aug 2026 19:39:17 GMT
• Title: StrAD: A Streaming Method and Benchmark for Audio Description Generation for Long-form Videos
• Abstract summary: We introduce StrAD, a benchmark for long-form AD generation on full-length videos spanning diverse genres such as movies, documentaries, short films, performances, and video games.<n>Our approach processes full-length videos with a sliding window, inserting ADs into existing transcripts without ground-truth timestamps.<n>On the segment-level task with given timestamps, our fine-tuned StrAD-FT sets the state of the art on CMD-AD with 36.3 CIDEr (+10.0 over Shot-by-shot), establishes a reference point on StrAD (51.0 CIDEr), and remains competitive on MAD-E
• Score: 16.585176367435682
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual content is the dominant medium of communication, yet without audio descriptions (ADs), it remains inaccessible to blind and low-vision people. ADs narrate context-relevant visual events during natural audio pauses. Manually creating ADs is expensive, limiting coverage to a small fraction of available content. Most existing automatic AD generation methods frame the task as video clip captioning, requiring ground-truth timestamps and additional context cues such as character databases. Current benchmarks reinforce this framing, consisting of short video segments paired with automatic or task-mismatched annotations. We introduce StrAD, a benchmark for long-form AD generation on full-length videos spanning diverse genres such as movies, documentaries, short films, performances, and video games. We reformulate AD generation as streaming dense video captioning. Our approach processes full-length videos with a sliding window, inserting ADs into existing transcripts without ground-truth timestamps, and supports both fine-tuned models and zero-shot prompting of vision-language models. On the segment-level task with given timestamps, our fine-tuned StrAD-FT sets the state of the art on CMD-AD with 36.3 CIDEr (+10.0 over Shot-by-shot), establishes a reference point on StrAD (51.0 CIDEr), and remains competitive on MAD-Eval at 24.9 CIDEr. On the full-video streaming task, StrAD-FT reaches a SODA score of 2.4 against 1.1 for our zero-shot baseline StrAD-Zero, though both exhibit limitations in temporal localization and narrative coherence. While prior work has tackled full-video AD generation in an offline, multi-stage fashion, ours is the first streaming approach, generating ADs on the fly without ground-truth timestamps. StrAD makes progress on full-video AD generation measurable, a prerequisite for scaling accessibility.

Related papers

• From Visual Cues to Spoken Narration: Rethinking Audio Description [56.77584893970582]
  We propose Cue2Narrate, a two-stage pipeline that jointly predicts what and when to narrate in longer untrimmed movie clips.<n>A dual-head audio-visual localizer predicts two temporally distinct windows per AD utterance.<n>A LoRA-adapted VLM then generates concise ADs from the predicted visual evidence.
  arXiv  Detail & Related papers  (2026-09-01T18:00:50Z)
• REFRAMED: Towards Realistic Audio Description Generation for Movies [53.11641004249926]
  Audio Description (AD) is a verbal narration of key visual content in videos, enabling access for visually impaired audiences.<n>Existing approaches formulate AD generation in an artificial setting where both the content and timing of descriptions are pre-specified.<n>We introduce a new formulation of AD generation in which models must jointly decide what to describe and when to do it.
  arXiv  Detail & Related papers  (2026-08-10T15:55:56Z)
• OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains [50.186434778589415]
  Current automated pipelines for audio-visual Question Answering (QA) generally adopt a video-caption-QA'' paradigm.<n>We propose an automated data engine featuring two mechanisms: textbfEntity-Anchored Video Scripting' and textbfClue-Guided QA Generation'
  arXiv  Detail & Related papers  (2026-06-12T17:59:55Z)
• TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions [64.27159505605312]
  This paper proposes Omni Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps.<n>To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions.<n>Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro.
  arXiv  Detail & Related papers  (2026-02-09T14:21:58Z)
• MCAD: Multimodal Context-Aware Audio Description Generation For Soccer [8.83668236549788]
  We present an end-to-end pipeline, MCAD, that extends AD generation beyond movies to the domain of sports.<n>We fine-tune a Video Large Language Model on publicly available movie AD datasets so that it learns the narrative structure and conventions of AD.<n>We introduce a new evaluation metric, ARGE-AD, designed to accurately assess the quality of generated AD.
  arXiv  Detail & Related papers  (2025-11-12T16:05:05Z)
• Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generation [110.79299467093006]
  We propose a two-stage framework that leverages "shots" as the fundamental units of video understanding.<n>This includes extending temporal context to neighbouring shots and incorporating film grammar devices, such as shot scales and thread structures.<n>Our method is compatible with both open-source and proprietary Visual-Language Models.
  arXiv  Detail & Related papers  (2025-04-01T17:59:57Z)
• DANTE-AD: Dual-Vision Attention Network for Long-Term Audio Description [19.14915136673913]
  We introduce DANTE-AD, an enhanced video description model leveraging a dual-vision Transformer-based architecture.<n>We propose a novel, state-of-the-art method for sequential cross-attention to achieve contextual grounding for fine-grained audio description generation.
  arXiv  Detail & Related papers  (2025-03-31T13:49:43Z)
• DistinctAD: Distinctive Audio Description Generation in Contexts [62.58375366359421]
  We propose DistinctAD, a framework for generating Audio Descriptions that emphasize distinctiveness to produce better narratives.<n>To address the domain gap, we introduce a CLIP-AD adaptation strategy that does not require additional AD corpora.<n>In Stage-II, DistinctAD incorporates two key innovations: (i) a Contextual Expectation-Maximization Attention (EMA) module that reduces redundancy by extracting common bases from consecutive video clips, and (ii) an explicit distinctive word prediction loss that filters out repeated words in the context.
  arXiv  Detail & Related papers  (2024-11-27T09:54:59Z)
• AutoAD: Movie Description in Context [91.98603496476215]
  This paper presents an automatic Audio Description (AD) model that ingests movies and outputs AD in text form. We leverage the power of pretrained foundation models, such as GPT and CLIP, and only train a mapping network that bridges the two models for visually-conditioned text generation.
  arXiv  Detail & Related papers  (2023-03-29T17:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.