Large Multi-modality Model Assisted AI-Generated Image Quality Assessment

• URL: http://arxiv.org/abs/2404.17762v2
• Date: Sat, 3 Aug 2024 02:49:36 GMT
• Title: Large Multi-modality Model Assisted AI-Generated Image Quality Assessment
• Authors: Puyi Wang, Wei Sun, Zicheng Zhang, Jun Jia, Yanwei Jiang, Zhichao Zhang, Xiongkuo Min, Guangtao Zhai,
• Abstract summary: We introduce a large Multi-modality model Assisted AI-Generated Image Quality Assessment (MA-AGIQA) model. It uses semantically informed guidance to sense semantic information and extract semantic vectors through carefully designed text prompts. It achieves state-of-the-art performance, and demonstrates its superior generalization capabilities on assessing the quality of AI-generated images.
• Score: 53.182136445844904
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional deep neural network (DNN)-based image quality assessment (IQA) models leverage convolutional neural networks (CNN) or Transformer to learn the quality-aware feature representation, achieving commendable performance on natural scene images. However, when applied to AI-Generated images (AGIs), these DNN-based IQA models exhibit subpar performance. This situation is largely due to the semantic inaccuracies inherent in certain AGIs caused by uncontrollable nature of the generation process. Thus, the capability to discern semantic content becomes crucial for assessing the quality of AGIs. Traditional DNN-based IQA models, constrained by limited parameter complexity and training data, struggle to capture complex fine-grained semantic features, making it challenging to grasp the existence and coherence of semantic content of the entire image. To address the shortfall in semantic content perception of current IQA models, we introduce a large Multi-modality model Assisted AI-Generated Image Quality Assessment (MA-AGIQA) model, which utilizes semantically informed guidance to sense semantic information and extract semantic vectors through carefully designed text prompts. Moreover, it employs a mixture of experts (MoE) structure to dynamically integrate the semantic information with the quality-aware features extracted by traditional DNN-based IQA models. Comprehensive experiments conducted on two AI-generated content datasets, AIGCQA-20k and AGIQA-3k show that MA-AGIQA achieves state-of-the-art performance, and demonstrate its superior generalization capabilities on assessing the quality of AGIs. Code is available at https://github.com/wangpuyi/MA-AGIQA.

Related papers

• GenzIQA: Generalized Image Quality Assessment using Prompt-Guided Latent Diffusion Models [7.291687946822539]
  A major drawback of state-of-the-art NR-IQA methods is their limited ability to generalize across diverse IQA settings. Recent text-to-image generative models generate meaningful visual concepts with fine details related to text concepts. In this work, we leverage the denoising process of such diffusion models for generalized IQA by understanding the degree of alignment between learnable quality-aware text prompts and images.
  arXiv  Detail & Related papers  (2024-06-07T05:46:39Z)
• ATTIQA: Generalizable Image Quality Feature Extractor using Attribute-aware Pretraining [25.680035174334886]
  In no-reference image quality assessment (NR-IQA), the challenge of limited dataset sizes hampers the development of robust and generalizable models. We propose a novel pretraining framework that constructs a generalizable representation for IQA by selectively extracting quality-related knowledge. Our approach achieves state-of-the-art performance on multiple IQA datasets and exhibits remarkable generalization capabilities.
  arXiv  Detail & Related papers  (2024-06-03T06:03:57Z)
• Understanding and Evaluating Human Preferences for AI Generated Images with Instruction Tuning [58.41087653543607]
  We first establish a novel Image Quality Assessment (IQA) database for AIGIs, termed AIGCIQA2023+. This paper presents a MINT-IQA model to evaluate and explain human preferences for AIGIs from Multi-perspectives with INstruction Tuning.
  arXiv  Detail & Related papers  (2024-05-12T17:45:11Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Diffusion Model Based Visual Compensation Guidance and Visual Difference Analysis for No-Reference Image Quality Assessment [82.13830107682232]
  We propose a novel class of state-of-the-art (SOTA) generative model, which exhibits the capability to model intricate relationships. We devise a new diffusion restoration network that leverages the produced enhanced image and noise-containing images. Two visual evaluation branches are designed to comprehensively analyze the obtained high-level feature information.
  arXiv  Detail & Related papers  (2024-02-22T09:39:46Z)
• Transformer-based No-Reference Image Quality Assessment via Supervised Contrastive Learning [36.695247860715874]
  We propose a novel Contrastive Learning (SCL) and Transformer-based NR-IQA model SaTQA. We first train a model on a large-scale synthetic dataset by SCL to extract degradation features of images with various distortion types and levels. To further extract distortion information from images, we propose a backbone network incorporating the Multi-Stream Block (MSB) by combining the CNN inductive bias and Transformer long-term dependence modeling capability. Experimental results on seven standard IQA datasets show that SaTQA outperforms the state-of-the-art methods for both synthetic and authentic datasets
  arXiv  Detail & Related papers  (2023-12-12T06:01:41Z)
• Image Quality Assessment using Contrastive Learning [50.265638572116984]
  We train a deep Convolutional Neural Network (CNN) using a contrastive pairwise objective to solve the auxiliary problem. We show through extensive experiments that CONTRIQUE achieves competitive performance when compared to state-of-the-art NR image quality models. Our results suggest that powerful quality representations with perceptual relevance can be obtained without requiring large labeled subjective image quality datasets.
  arXiv  Detail & Related papers  (2021-10-25T21:01:00Z)
• No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency [38.88541492121366]
  The goal of No-Reference Image Quality Assessment (NR-IQA) is to estimate the perceptual image quality in accordance with subjective evaluations. We propose a novel model to address the NR-IQA task by leveraging a hybrid approach that benefits from Convolutional Neural Networks (CNNs) and self-attention mechanism in Transformers.
  arXiv  Detail & Related papers  (2021-08-16T02:07:08Z)
• MetaIQA: Deep Meta-learning for No-Reference Image Quality Assessment [73.55944459902041]
  This paper presents a no-reference IQA metric based on deep meta-learning. We first collect a number of NR-IQA tasks for different distortions. Then meta-learning is adopted to learn the prior knowledge shared by diversified distortions. Extensive experiments demonstrate that the proposed metric outperforms the state-of-the-arts by a large margin.
  arXiv  Detail & Related papers  (2020-04-11T23:36:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.