AIGVE-Tool: AI-Generated Video Evaluation Toolkit with Multifaceted Benchmark

• URL: http://arxiv.org/abs/2503.14064v1
• Date: Tue, 18 Mar 2025 09:36:33 GMT
• Title: AIGVE-Tool: AI-Generated Video Evaluation Toolkit with Multifaceted Benchmark
• Authors: Xinhao Xiang, Xiao Liu, Zizhong Li, Zhuosheng Liu, Jiawei Zhang,
• Abstract summary: Existing metrics lack a unified framework for systematically categorizing methodologies.<n>We introduce AIGVE-Tool, a unified framework that provides a structured taxonomy and evaluation pipeline for AI-generated video evaluation.<n>A large-scale benchmark dataset is created with five SOTA video generation models based on hand-crafted instructions and prompts.
• Score: 8.827755848017578
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid advancement in AI-generated video synthesis has led to a growth demand for standardized and effective evaluation metrics. Existing metrics lack a unified framework for systematically categorizing methodologies, limiting a holistic understanding of the evaluation landscape. Additionally, fragmented implementations and the absence of standardized interfaces lead to redundant processing overhead. Furthermore, many prior approaches are constrained by dataset-specific dependencies, limiting their applicability across diverse video domains. To address these challenges, we introduce AIGVE-Tool (AI-Generated Video Evaluation Toolkit), a unified framework that provides a structured and extensible evaluation pipeline for a comprehensive AI-generated video evaluation. Organized within a novel five-category taxonomy, AIGVE-Tool integrates multiple evaluation methodologies while allowing flexible customization through a modular configuration system. Additionally, we propose AIGVE-Bench, a large-scale benchmark dataset created with five SOTA video generation models based on hand-crafted instructions and prompts. This dataset systematically evaluates various video generation models across nine critical quality dimensions. Extensive experiments demonstrate the effectiveness of AIGVE-Tool in providing standardized and reliable evaluation results, highlighting specific strengths and limitations of current models and facilitating the advancements of next-generation AI-generated video techniques.

Related papers

• VideoGen-Eval: Agent-based System for Video Generation Evaluation [54.662739174367836]
  Video generation has rendered existing evaluation systems inadequate for assessing state-of-the-art models. We propose VideoGen-Eval, an agent evaluation system that integrates content structuring, MLLM-based content judgment, and patch tools for temporal-dense dimensions. We introduce a video generation benchmark to evaluate existing cutting-edge models and verify the effectiveness of our evaluation system.
  arXiv  Detail & Related papers  (2025-03-30T14:12:21Z)
• VACT: A Video Automatic Causal Testing System and a Benchmark [55.53300306960048]
  VACT is an **automated** framework for modeling, evaluating, and measuring the causal understanding of VGMs in real-world scenarios. We introduce multi-level causal evaluation metrics to provide a detailed analysis of the causal performance of VGMs.
  arXiv  Detail & Related papers  (2025-03-08T10:54:42Z)
• AnyAnomaly: Zero-Shot Customizable Video Anomaly Detection with LVLM [1.7051307941715268]
  Video anomaly detection (VAD) is crucial for video analysis and surveillance in computer vision. Existing VAD models rely on learned normal patterns, which makes them difficult to apply to diverse environments. This study proposes customizable video anomaly detection (C-VAD) technique and the AnyAnomaly model.
  arXiv  Detail & Related papers  (2025-03-06T14:52:34Z)
• Multilevel Semantic-Aware Model for AI-Generated Video Quality Assessment [15.529169236891532]
  We introduce MSA-VQA, a Multilevel Semantic-Aware Model for AI-Generated Video Quality Assessment.<n>Our hierarchical framework analyzes video content at three levels: frame, segment, and video.<n>We propose a Prompt Semantic Supervision Module using text encoder of CLIP to ensure semantic consistency between videos and conditional prompts.
  arXiv  Detail & Related papers  (2025-01-06T01:18:11Z)
• AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM [54.44479359918971]
  We first present AIGVQA-DB, a large-scale dataset comprising 36,576 AIGVs generated by 15 advanced text-to-video models using 1,048 prompts. We then introduce AIGV-Assessor, a novel VQA model that leverages intricate quality attributes to capture precise video quality scores and pair video preferences.
  arXiv  Detail & Related papers  (2024-11-26T08:43:15Z)
• Benchmarking Multi-dimensional AIGC Video Quality Assessment: A Dataset and Unified Model [56.03592388332793]
  We investigate the AIGC-VQA problem, considering both subjective and objective quality assessment perspectives. For the subjective perspective, we construct the Large-scale Generated Video Quality assessment (LGVQ) dataset, consisting of 2,808 AIGC videos. We evaluate the perceptual quality of AIGC videos from three critical dimensions: spatial quality, temporal quality, and text-video alignment. We propose the Unify Generated Video Quality assessment (UGVQ) model, designed to accurately evaluate the multi-dimensional quality of AIGC videos.
  arXiv  Detail & Related papers  (2024-07-31T07:54:26Z)
• AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI [1.1035305628305816]
  This paper introduces AIGCBench, a pioneering comprehensive benchmark designed to evaluate a variety of video generation tasks. A varied and open-domain image-text dataset that evaluates different state-of-the-art algorithms under equivalent conditions. We employ a novel text combiner and GPT-4 to create rich text prompts, which are then used to generate images via advanced Text-to-Image models.
  arXiv  Detail & Related papers  (2024-01-03T10:08:40Z)
• Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models [81.84810348214113]
  Video-based large language models (Video-LLMs) have been recently introduced, targeting both fundamental improvements in perception and comprehension, and a diverse range of user inquiries. To guide the development of such a model, the establishment of a robust and comprehensive evaluation system becomes crucial. This paper proposes textitVideo-Bench, a new comprehensive benchmark along with a toolkit specifically designed for evaluating Video-LLMs.
  arXiv  Detail & Related papers  (2023-11-27T18:59:58Z)
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models [70.19437817951673]
  We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Our approach involves generating a diverse and comprehensive list of 700 prompts for text-to-video generation. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmark, in terms of visual qualities, content qualities, motion qualities, and text-video alignment with 17 well-selected objective metrics.
  arXiv  Detail & Related papers  (2023-10-17T17:50:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.