T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Models

• URL: http://arxiv.org/abs/2505.04946v1
• Date: Thu, 08 May 2025 04:49:52 GMT
• Title: T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Models
• Authors: Xuyang Guo, Jiayan Huo, Zhenmei Shi, Zhao Song, Jiahao Zhang, Jiale Zhao,
• Abstract summary: T2VTextBench is the first human-evaluation benchmark dedicated to evaluating on-screen text fidelity and temporal consistency in text-to-video models.<n>We evaluate ten state-of-the-art systems, ranging from open-source solutions to commercial offerings, and find that most struggle to generate legible, consistent text.
• Score: 12.120541052871486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Thanks to recent advancements in scalable deep architectures and large-scale pretraining, text-to-video generation has achieved unprecedented capabilities in producing high-fidelity, instruction-following content across a wide range of styles, enabling applications in advertising, entertainment, and education. However, these models' ability to render precise on-screen text, such as captions or mathematical formulas, remains largely untested, posing significant challenges for applications requiring exact textual accuracy. In this work, we introduce T2VTextBench, the first human-evaluation benchmark dedicated to evaluating on-screen text fidelity and temporal consistency in text-to-video models. Our suite of prompts integrates complex text strings with dynamic scene changes, testing each model's ability to maintain detailed instructions across frames. We evaluate ten state-of-the-art systems, ranging from open-source solutions to commercial offerings, and find that most struggle to generate legible, consistent text. These results highlight a critical gap in current video generators and provide a clear direction for future research aimed at enhancing textual manipulation in video synthesis.

Related papers

• VidText: Towards Comprehensive Evaluation for Video Text Understanding [54.15328647518558]
  VidText is a benchmark for comprehensive and in-depth evaluation of video text understanding.<n>It covers a wide range of real-world scenarios and supports multilingual content.<n>It introduces a hierarchical evaluation framework with video-level, clip-level, and instance-level tasks.
  arXiv  Detail & Related papers  (2025-05-28T19:39:35Z)
• Visual Text Processing: A Comprehensive Review and Unified Evaluation [99.57846940547171]
  We present a comprehensive, multi-perspective analysis of recent advancements in visual text processing.<n>Our aim is to establish this work as a fundamental resource that fosters future exploration and innovation in the dynamic field of visual text processing.
  arXiv  Detail & Related papers  (2025-04-30T14:19:29Z)
• Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models [19.51519289698524]
  We present T2VCountBench, a specialized benchmark aiming at evaluating the counting capability of SOTA text-to-video models as of 2025.<n>Our experiments reveal that all existing models struggle with basic numerical tasks, almost always failing to generate videos with an object count of 9 or fewer.<n>Our findings highlight important challenges in current text-to-video generation and provide insights for future research aimed at improving adherence to basic numerical constraints.
  arXiv  Detail & Related papers  (2025-04-05T04:13:06Z)
• TextInVision: Text and Prompt Complexity Driven Visual Text Generation Benchmark [61.412934963260724]
  Existing diffusion-based text-to-image models often struggle to accurately embed text within images.<n>We introduce TextInVision, a large-scale, text and prompt complexity driven benchmark to evaluate the ability of diffusion models to integrate visual text into images.
  arXiv  Detail & Related papers  (2025-03-17T21:36:31Z)
• T2VEval: Benchmark Dataset and Objective Evaluation Method for T2V-generated Videos [9.742383920787413]
  T2VEval is a multi-branch fusion scheme for text-to-video quality evaluation.<n>It assesses videos across three branches: text-video consistency, realness, and technical quality.<n>T2VEval achieves state-of-the-art performance across multiple metrics.
  arXiv  Detail & Related papers  (2025-01-15T03:11:33Z)
• T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation [55.57459883629706]
  We conduct the first systematic study on compositional text-to-video generation.<n>We propose T2V-CompBench, the first benchmark tailored for compositional text-to-video generation.
  arXiv  Detail & Related papers  (2024-07-19T17:58:36Z)
• Towards A Better Metric for Text-to-Video Generation [102.16250512265995]
  Generative models have demonstrated remarkable capability in synthesizing high-quality text, images, and videos. We introduce a novel evaluation pipeline, the Text-to-Video Score (T2VScore) This metric integrates two pivotal criteria: (1) Text-Video Alignment, which scrutinizes the fidelity of the video in representing the given text description, and (2) Video Quality, which evaluates the video's overall production caliber with a mixture of experts.
  arXiv  Detail & Related papers  (2024-01-15T15:42:39Z)
• CelebV-Text: A Large-Scale Facial Text-Video Dataset [91.22496444328151]
  CelebV-Text is a large-scale, diverse, and high-quality dataset of facial text-video pairs. CelebV-Text comprises 70,000 in-the-wild face video clips with diverse visual content, each paired with 20 texts generated using the proposed semi-automatic text generation strategy. The superiority of CelebV-Text over other datasets is demonstrated via comprehensive statistical analysis of the videos, texts, and text-video relevance.
  arXiv  Detail & Related papers  (2023-03-26T13:06:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.