GAIA: Rethinking Action Quality Assessment for AI-Generated Videos

• URL: http://arxiv.org/abs/2406.06087v1
• Date: Mon, 10 Jun 2024 08:18:07 GMT
• Title: GAIA: Rethinking Action Quality Assessment for AI-Generated Videos
• Authors: Zijian Chen, Wei Sun, Yuan Tian, Jun Jia, Zicheng Zhang, Jiarui Wang, Ru Huang, Xiongkuo Min, Guangtao Zhai, Wenjun Zhang,
• Abstract summary: Action quality assessment (AQA) algorithms predominantly focus on actions from real specific scenarios and are pre-trained with normative action features. We construct GAIA, a Generic AI-generated Action dataset, by conducting a large-scale subjective evaluation from a novel causal reasoning-based perspective. Based on GAIA, we evaluate a suite of popular text-to-video (T2V) models on their ability to generate visually rational actions. Results show that traditional AQA methods, action-related metrics in recent T2V benchmarks, and mainstream video quality methods correlate poorly with human opinions.
• Score: 56.047773400426486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Assessing action quality is both imperative and challenging due to its significant impact on the quality of AI-generated videos, further complicated by the inherently ambiguous nature of actions within AI-generated video (AIGV). Current action quality assessment (AQA) algorithms predominantly focus on actions from real specific scenarios and are pre-trained with normative action features, thus rendering them inapplicable in AIGVs. To address these problems, we construct GAIA, a Generic AI-generated Action dataset, by conducting a large-scale subjective evaluation from a novel causal reasoning-based perspective, resulting in 971,244 ratings among 9,180 video-action pairs. Based on GAIA, we evaluate a suite of popular text-to-video (T2V) models on their ability to generate visually rational actions, revealing their pros and cons on different categories of actions. We also extend GAIA as a testbed to benchmark the AQA capacity of existing automatic evaluation methods. Results show that traditional AQA methods, action-related metrics in recent T2V benchmarks, and mainstream video quality methods correlate poorly with human opinions, indicating a sizable gap between current models and human action perception patterns in AIGVs. Our findings underscore the significance of action quality as a unique perspective for studying AIGVs and can catalyze progress towards methods with enhanced capacities for AQA in AIGVs.

Related papers

• Advancing Video Quality Assessment for AIGC [17.23281750562252]
  We propose a novel loss function that combines mean absolute error with cross-entropy loss to mitigate inter-frame quality inconsistencies. We also introduce the innovative S2CNet technique to retain critical content, while leveraging adversarial training to enhance the model's generalization capabilities.
  arXiv  Detail & Related papers  (2024-09-23T10:36:22Z)
• Revisiting Video Quality Assessment from the Perspective of Generalization [17.695835285573807]
  Short video platforms such as YouTube Shorts, TikTok, and Kwai have led to a surge in User-Generated Content (UGC) These challenges not only affect performance on test sets but also impact the ability to generalize across different datasets. We show that adversarial weight perturbations can effectively smooth this landscape, significantly improving the generalization performance.
  arXiv  Detail & Related papers  (2024-09-23T09:24:55Z)
• Benchmarking AIGC Video Quality Assessment: A Dataset and Unified Model [54.69882562863726]
  We try to systemically investigate the AIGC-VQA problem from both subjective and objective quality assessment perspectives. We evaluate the perceptual quality of AIGC videos from three dimensions: spatial quality, temporal quality, and text-to-video alignment. We propose a Unify Generated Video Quality assessment (UGVQ) model to comprehensively and accurately evaluate the quality of AIGC videos.
  arXiv  Detail & Related papers  (2024-07-31T07:54:26Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessment [54.93996119324928]
  We create the largest AIGI subjective quality database to date with 20,000 AIGIs and 420,000 subjective ratings, known as AIGIQA-20K. We conduct benchmark experiments on this database to assess the correspondence between 16 mainstream AIGI quality models and human perception.
  arXiv  Detail & Related papers  (2024-04-04T12:12:24Z)
• AIGCOIQA2024: Perceptual Quality Assessment of AI Generated Omnidirectional Images [70.42666704072964]
  We establish a large-scale AI generated omnidirectional image IQA database named AIGCOIQA2024. A subjective IQA experiment is conducted to assess human visual preferences from three perspectives. We conduct a benchmark experiment to evaluate the performance of state-of-the-art IQA models on our database.
  arXiv  Detail & Related papers  (2024-04-01T10:08:23Z)
• Multi-Stage Contrastive Regression for Action Quality Assessment [31.763380011104015]
  We propose a novel Multi-stage Contrastive Regression (MCoRe) framework for the action quality assessment (AQA) task. Inspired by the graph contrastive learning, we propose a new stage-wise contrastive learning loss function to enhance performance. MCoRe demonstrates the state-of-the-art result so far on the widely-adopted fine-grained AQA dataset.
  arXiv  Detail & Related papers  (2024-01-05T14:48:19Z)
• Group-aware Contrastive Regression for Action Quality Assessment [85.43203180953076]
  We show that the relations among videos can provide important clues for more accurate action quality assessment. Our approach outperforms previous methods by a large margin and establishes new state-of-the-art on all three benchmarks.
  arXiv  Detail & Related papers  (2021-08-17T17:59:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.