VideoA11y: Method and Dataset for Accessible Video Description

• URL: http://arxiv.org/abs/2502.20480v1
• Date: Thu, 27 Feb 2025 19:44:31 GMT
• Title: VideoA11y: Method and Dataset for Accessible Video Description
• Abstract summary: Video descriptions are crucial for blind and low vision (BLV) users to access visual content.<n>We introduce VideoA11y, an approach that leverages multimodal large language models (MLLMs) and video accessibility guidelines to generate descriptions tailored for BLV individuals.<n>Using this method, we have curated VideoA11y-40K, the largest and most comprehensive dataset of 40,000 videos described for BLV users.
• Score: 4.081116465356645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video descriptions are crucial for blind and low vision (BLV) users to access visual content. However, current artificial intelligence models for generating descriptions often fall short due to limitations in the quality of human annotations within training datasets, resulting in descriptions that do not fully meet BLV users' needs. To address this gap, we introduce VideoA11y, an approach that leverages multimodal large language models (MLLMs) and video accessibility guidelines to generate descriptions tailored for BLV individuals. Using this method, we have curated VideoA11y-40K, the largest and most comprehensive dataset of 40,000 videos described for BLV users. Rigorous experiments across 15 video categories, involving 347 sighted participants, 40 BLV participants, and seven professional describers, showed that VideoA11y descriptions outperform novice human annotations and are comparable to trained human annotations in clarity, accuracy, objectivity, descriptiveness, and user satisfaction. We evaluated models on VideoA11y-40K using both standard and custom metrics, demonstrating that MLLMs fine-tuned on this dataset produce high-quality accessible descriptions. Code and dataset are available at https://people-robots.github.io/VideoA11y.

Related papers

• Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios [61.926801508808616]
  We introduce Video-IFBench, a benchmark for evaluating instruction following in video understanding.<n>We develop an instruction taxonomy with four templates, including single-task, multi-task, selection, and nested instructions.<n>We build a semi-automatic data construction pipeline that combines MLLMs, programmatic processing, and human verification, resulting in 1.5K samples.
  arXiv  Detail & Related papers  (2026-08-26T08:35:42Z)
• InstructionBench: An Instructional Video Understanding Benchmark [14.71613140347162]
  We introduce InstructionBench, an instructional video understanding benchmark. We formulate Q&A pairs in open-ended and multiple-choice formats to assess both Coarse-Grained event-level and Fine-Grained object-level reasoning. The benchmark finally contains 5k questions across over 700 videos.
  arXiv  Detail & Related papers  (2025-04-07T13:05:09Z)
• GuideDog: A Real-World Egocentric Multimodal Dataset for Blind and Low-Vision Accessibility-Aware Guidance [18.467461615621872]
  Mobility remains a significant challenge for the 2.2 billion people worldwide affected by blindness and low vision (BLV)<n>We introduce GuideDog, a novel accessibility-aware guide dataset containing 22K image-description pairs.<n>We also develop GuideDogQA, a subset of 818 samples featuring multiple-choice questions designed to evaluate fine-grained visual perception capabilities.
  arXiv  Detail & Related papers  (2025-03-17T05:43:40Z)
• GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning [62.775721264492994]
  GRADEO is one of the first specifically designed video evaluation models. It grades AI-generated videos for explainable scores and assessments through multi-step reasoning. Experiments show that our method aligns better with human evaluations than existing methods.
  arXiv  Detail & Related papers  (2025-03-04T07:04:55Z)
• Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmark [5.76230561819199]
  We propose Repurpose-10K, an extensive dataset comprising over 10,000 videos with more than 120,000 annotated clips.<n>We propose a two-stage solution to obtain annotations from real-world user-generated content.<n>We offer a baseline model to address this challenging task by integrating audio, visual, and caption aspects.
  arXiv  Detail & Related papers  (2024-12-12T02:27:46Z)
• StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identification [6.762705315042178]
  Long video description introduces new challenges, such as plot-level consistency across descriptions. We propose StoryTeller, a system for generating dense descriptions of long videos, incorporating both low-level visual concepts and high-level plot information.
  arXiv  Detail & Related papers  (2024-11-11T15:51:48Z)
• Can LVLMs Describe Videos like Humans? A Five-in-One Video Annotations Benchmark for Better Human-Machine Comparison [15.363132825156477]
  Video description serves as a fundamental task for evaluating video comprehension, requiring a deep understanding of spatial and temporal dynamics. Current benchmarks for video comprehension have notable limitations, including short video durations, brief annotations, and reliance on a single annotator's perspective. We propose a novel benchmark, FIOVA, designed to evaluate the differences between LVLMs and human understanding more comprehensively.
  arXiv  Detail & Related papers  (2024-10-20T03:59:54Z)
• SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset [7.585772927643345]
  This dataset encompasses human preferences in text-to-video generation tasks along two primary dimensions: helpfulness and harmlessness. The SafeSora dataset includes 14,711 unique prompts, 57,333 unique videos generated by 4 distinct LVMs, and 51,691 pairs of preference annotations labeled by humans.
  arXiv  Detail & Related papers  (2024-06-20T16:38:56Z)
• Towards Event-oriented Long Video Understanding [101.48089908037888]
  Event-Bench is an event-oriented long video understanding benchmark built on existing datasets and human annotations. VIM is a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions.
  arXiv  Detail & Related papers  (2024-06-20T09:14:19Z)
• DeVAn: Dense Video Annotation for Video-Language Models [68.70692422636313]
  We present a novel human annotated dataset for evaluating the ability for visual-language models to generate descriptions for real-world video clips. The dataset contains 8.5K YouTube video clips of 20-60 seconds in duration and covers a wide range of topics and interests.
  arXiv  Detail & Related papers  (2023-10-08T08:02:43Z)
• InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation [90.71796406228265]
  InternVid is a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations. The InternVid dataset contains over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words.
  arXiv  Detail & Related papers  (2023-07-13T17:58:32Z)
• A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot [67.00455874279383]
  We propose verbalizing long videos to generate descriptions in natural language, then performing video-understanding tasks on the generated story as opposed to the original video. Our method, despite being zero-shot, achieves significantly better results than supervised baselines for video understanding. To alleviate a lack of story understanding benchmarks, we publicly release the first dataset on a crucial task in computational social science on persuasion strategy identification.
  arXiv  Detail & Related papers  (2023-05-16T19:13:11Z)
• Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions [75.77044856100349]
  We present the Spoken Moments dataset of 500k spoken captions each attributed to a unique short video depicting a broad range of different events. We show that our AMM approach consistently improves our results and that models trained on our Spoken Moments dataset generalize better than those trained on other video-caption datasets.
  arXiv  Detail & Related papers  (2021-05-10T16:30:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.