Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicality

• URL: http://arxiv.org/abs/2406.08845v4
• Date: Thu, 31 Oct 2024 08:38:26 GMT
• Title: Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicality
• Authors: Tianle Zhang, Langtian Ma, Yuchen Yan, Yuchen Zhang, Kai Wang, Yue Yang, Ziyao Guo, Wenqi Shao, Yang You, Yu Qiao, Ping Luo, Kaipeng Zhang,
• Abstract summary: This paper introduces the Text-to-Video Human Evaluation (T2VHE) protocol, a comprehensive and standardized protocol for T2V models. The protocol includes well-defined metrics, thorough annotator training, and an effective dynamic evaluation module. We will open-source the entire setup of the T2VHE protocol, including the complete protocol workflow, the dynamic evaluation component details, and the annotation interface code.
• Score: 58.87422943009375
• License:
• Abstract: Recent text-to-video (T2V) technology advancements, as demonstrated by models such as Gen2, Pika, and Sora, have significantly broadened its applicability and popularity. Despite these strides, evaluating these models poses substantial challenges. Primarily, due to the limitations inherent in automatic metrics, manual evaluation is often considered a superior method for assessing T2V generation. However, existing manual evaluation protocols face reproducibility, reliability, and practicality issues. To address these challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE) protocol, a comprehensive and standardized protocol for T2V models. The T2VHE protocol includes well-defined metrics, thorough annotator training, and an effective dynamic evaluation module. Experimental results demonstrate that this protocol not only ensures high-quality annotations but can also reduce evaluation costs by nearly 50\%. We will open-source the entire setup of the T2VHE protocol, including the complete protocol workflow, the dynamic evaluation component details, and the annotation interface code. This will help communities establish more sophisticated human assessment protocols.

Related papers

• T2VEval: T2V-generated Videos Benchmark Dataset and Objective Evaluation Method [13.924105106722534]
  T2VEval is a multi-branch fusion scheme for text-to-video quality evaluation. It assesses videos across three branches: text-video consistency, realness, and technical quality. T2VEval achieves state-of-the-art performance across multiple metrics.
  arXiv  Detail & Related papers  (2025-01-15T03:11:33Z)
• EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation [29.176750442205325]
  In this study, we contribute an EvalMuse-40K benchmark, gathering 40K image-text pairs with fine-grained human annotations for image-text alignment-related tasks. We introduce two new methods to evaluate the image-text alignment capabilities of T2I models.
  arXiv  Detail & Related papers  (2024-12-24T04:08:25Z)
• LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment [15.11363628734519]
  This paper proposes LiFT, a novel fine-tuning method leveraging human feedback for T2V model alignment. We train a reward model LiFT-Critic to learn reward function effectively, which serves as a proxy for human judgment. Lastly, we leverage the learned reward function to align the T2V model by maximizing the reward-weighted likelihood.
  arXiv  Detail & Related papers  (2024-12-06T07:16:14Z)
• Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols [52.40622903199512]
  This paper introduces AI-Control Games, a formal decision-making model of the red-teaming exercise as a multi-objective, partially observable game. We apply our formalism to model, evaluate and synthesise protocols for deploying untrusted language models as programming assistants.
  arXiv  Detail & Related papers  (2024-09-12T12:30:07Z)
• Evaluation of Text-to-Video Generation Models: A Dynamics Perspective [94.2662603491163]
  Existing evaluation protocols primarily focus on temporal consistency and content continuity. We propose an effective evaluation protocol, termed DEVIL, which centers on the dynamics dimension to evaluate T2V models.
  arXiv  Detail & Related papers  (2024-07-01T08:51:22Z)
• Measuring the Quality of Text-to-Video Model Outputs: Metrics and Dataset [1.9685736810241874]
  The paper presents a dataset of more than 1,000 generated videos from 5 very recent T2V models on which some of those commonly used quality metrics are applied. We also include extensive human quality evaluations on those videos, allowing the relative strengths and weaknesses of metrics, including human assessment, to be compared. Our conclusion is that naturalness and semantic matching with the text prompt used to generate the T2V output are important but there is no single measure to capture these subtleties in assessing T2V model output.
  arXiv  Detail & Related papers  (2023-09-14T19:35:53Z)
• Visual Programming for Text-to-Image Generation and Evaluation [73.12069620086311]
  We propose two novel interpretable/explainable visual programming frameworks for text-to-image (T2I) generation and evaluation. First, we introduce VPGen, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. Second, we introduce VPEval, an interpretable and explainable evaluation framework for T2I generation based on visual programming.
  arXiv  Detail & Related papers  (2023-05-24T16:42:17Z)
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation [176.56131810249602]
  evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial. We introduce FACTSCORE, a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atomic facts supported by a reliable knowledge source.
  arXiv  Detail & Related papers  (2023-05-23T17:06:00Z)
• Evaluation of Test-Time Adaptation Under Computational Time Constraints [80.40939405129102]
  Test Time Adaptation (TTA) methods leverage unlabeled data at test time to adapt to distribution shifts. Current evaluation protocols overlook the effect of this extra cost, affecting their real-world applicability. We propose a more realistic evaluation protocol for TTA methods, where data is received in an online fashion from a constant-speed data stream.
  arXiv  Detail & Related papers  (2023-04-10T18:01:47Z)
• Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation [35.8129864412223]
  This paper proposes a standardized and well-defined human evaluation protocol. We experimentally show that the current automatic measures are incompatible with human perception. We provide insights for designing human evaluation experiments reliably and conclusively.
  arXiv  Detail & Related papers  (2023-04-04T14:14:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.