READ More than What You See: Reinforcement Learning for Accurate and Coherent Audio Description Generations

• URL: http://arxiv.org/abs/2606.22766v1
• Date: Mon, 22 Jun 2026 02:05:38 GMT
• Title: READ More than What You See: Reinforcement Learning for Accurate and Coherent Audio Description Generations
• Abstract summary: We present READ, the first reinforcement-learning framework for training-based AD generation.<n> READ formulates AD as sequence-level optimization with reference-matching, length, and format rewards.<n>Our results highlight RL as a promising paradigm for accurate and coherent AD generation.
• Score: 48.760748753984764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio Description aims to generate concise narrations of essential visual content in audio-visual media for blind and low-vision audiences. Existing methods either rely on prompting off-the-shelf multimodal models, which often mismatch AD style, or partially optimize training-based systems with next-token prediction, which under-explores model capacity and biases generation toward generic expressions. We present READ, the first reinforcement-learning (RL) framework for training-based AD generation. READ formulates AD as sequence-level optimization with reference-matching, length, and format rewards, and further introduces a dedicated coherence reward under context-aware supervision to promote narratively coherent descriptions. Experiments on MAD-Eval, CMD-AD, and TV-AD show that READ substantially outperforms prior methods across diverse evaluation metrics. Our results highlight RL as a promising paradigm for accurate and coherent AD generation. Our codes, models, and benchmark results will be publicly available.

Related papers

• A Unified and Reproducible Experimentation Framework for Speech Understanding [44.73703262229875]
  SURE is a unified experimentation framework that standardizes prediction formats, normalization, and scoring.<n>We present SURE, a unified experimentation framework that standardizes prediction formats, normalization, and scoring.
  arXiv  Detail & Related papers  (2026-05-29T06:33:36Z)
• REC-RL: Referring expression counting via Gaussian and range-based reward optimization [28.33789155610508]
  Referring expression counting (REC) is an intention-driven task that requires context-aware visual reasoning.<n>We propose REC-RL, a reinforcement learning framework that introduces a think-range-answer paradigm to explicitly optimize the visual reasoning process.
  arXiv  Detail & Related papers  (2026-05-15T07:37:45Z)
• More than a Moment: Towards Coherent Sequences of Audio Descriptions [88.14731697642098]
  Audio Descriptions (ADs) convey essential on-screen information, allowing visually impaired audiences to follow videos.<n>Most automatic methods generate each AD independently, often resulting in repetitive, incoherent descriptions.<n>We propose a training-free method, CoherentAD, that first generates multiple candidate descriptions for each AD time interval, and then performs auto-regressive selection across the sequence.
  arXiv  Detail & Related papers  (2025-10-29T12:06:42Z)
• AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation [113.75682363364004]
  AURORA is a framework designed to enhance genuine reasoning and language comprehension in reference audio-visual segmentation.<n>AURORA achieves state-of-the-art performance on Ref-AVS benchmarks and generalizes effectively to unreferenced segmentation.
  arXiv  Detail & Related papers  (2025-08-04T07:47:38Z)
• Benchmarking Representations for Speech, Music, and Acoustic Events [24.92641211471113]
  ARCH is a comprehensive benchmark for evaluating ARL methods on diverse audio classification domains. ARCH comprises 12 datasets, that allow us to thoroughly assess pre-trained SSL models of different sizes. To address the current lack of open-source, pre-trained models for non-speech audio, we also release new pre-trained models that demonstrate strong performance on non-speech datasets.
  arXiv  Detail & Related papers  (2024-05-02T01:24:53Z)
• Hierarchical Indexing for Retrieval-Augmented Opinion Summarization [60.5923941324953]
  We propose a method for unsupervised abstractive opinion summarization that combines the attributability and scalability of extractive approaches with the coherence and fluency of Large Language Models (LLMs) Our method, HIRO, learns an index structure that maps sentences to a path through a semantically organized discrete hierarchy. At inference time, we populate the index and use it to identify and retrieve clusters of sentences containing popular opinions from input reviews.
  arXiv  Detail & Related papers  (2024-03-01T10:38:07Z)
• MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning [120.95150400119705]
  We present MM-Narrator, a novel system leveraging GPT-4 with multimodal in-context learning for the generation of audio descriptions (AD) MM-Narrator excels in generating precise audio descriptions for videos of extensive lengths, even beyond hours, in an autoregressive manner. We introduce the first segment-based evaluator for recurrent text generation.
  arXiv  Detail & Related papers  (2023-11-29T08:27:00Z)
• Boosting Audio-visual Zero-shot Learning with Large Language Models [32.533844163120875]
  We introduce a framework called KnowleDge-Augmented audio-visual learning (KDA) Our proposed KDA can outperform state-of-the-art methods on three popular audio-visual zero-shot learning datasets.
  arXiv  Detail & Related papers  (2023-11-21T01:18:23Z)
• An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition [98.70304981174748]
  We focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models. We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
  arXiv  Detail & Related papers  (2021-10-09T15:06:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.