IQA-Adapter: Exploring Knowledge Transfer from Image Quality Assessment to Diffusion-based Generative Models

• URL: http://arxiv.org/abs/2412.01794v1
• Date: Mon, 02 Dec 2024 18:40:19 GMT
• Title: IQA-Adapter: Exploring Knowledge Transfer from Image Quality Assessment to Diffusion-based Generative Models
• Authors: Khaled Abud, Sergey Lavrushkin, Alexey Kirillov, Dmitriy Vatolin,
• Abstract summary: We introduce IQA-Adapter, a novel architecture that conditions generation on target quality levels by learning the relationship between images and quality scores. IQA-Adapter achieves up to a 10% improvement across multiple objective metrics, as confirmed by a subjective study. Our quality-aware methods also provide insights into the adversarial robustness of IQA models.
• Score: 0.5356944479760104
• License:
• Abstract: Diffusion-based models have recently transformed conditional image generation, achieving unprecedented fidelity in generating photorealistic and semantically accurate images. However, consistently generating high-quality images remains challenging, partly due to the lack of mechanisms for conditioning outputs on perceptual quality. In this work, we propose methods to integrate image quality assessment (IQA) models into diffusion-based generators, enabling quality-aware image generation. First, we experiment with gradient-based guidance to optimize image quality directly and show this approach has limited generalizability. To address this, we introduce IQA-Adapter, a novel architecture that conditions generation on target quality levels by learning the relationship between images and quality scores. When conditioned on high target quality, IQA-Adapter shifts the distribution of generated images towards a higher-quality subdomain. This approach achieves up to a 10% improvement across multiple objective metrics, as confirmed by a subjective study, while preserving generative diversity and content. Additionally, IQA-Adapter can be used inversely as a degradation model, generating progressively more distorted images when conditioned on lower quality scores. Our quality-aware methods also provide insights into the adversarial robustness of IQA models, underscoring the potential of quality conditioning in generative modeling and the importance of robust IQA methods.

Related papers

• GenzIQA: Generalized Image Quality Assessment using Prompt-Guided Latent Diffusion Models [7.291687946822539]
  A major drawback of state-of-the-art NR-IQA methods is their limited ability to generalize across diverse IQA settings. Recent text-to-image generative models generate meaningful visual concepts with fine details related to text concepts. In this work, we leverage the denoising process of such diffusion models for generalized IQA by understanding the degree of alignment between learnable quality-aware text prompts and images.
  arXiv  Detail & Related papers  (2024-06-07T05:46:39Z)
• Dual-Branch Network for Portrait Image Quality Assessment [76.27716058987251]
  We introduce a dual-branch network for portrait image quality assessment (PIQA) We utilize two backbone networks (textiti.e., Swin Transformer-B) to extract the quality-aware features from the entire portrait image and the facial image cropped from it. We leverage LIQE, an image scene classification and quality assessment model, to capture the quality-aware and scene-specific features as the auxiliary features.
  arXiv  Detail & Related papers  (2024-05-14T12:43:43Z)
• Cross-IQA: Unsupervised Learning for Image Quality Assessment [3.2287957986061038]
  We propose a no-reference image quality assessment (NR-IQA) method termed Cross-IQA based on vision transformer(ViT) model. The proposed Cross-IQA method can learn image quality features from unlabeled image data. Experimental results show that Cross-IQA can achieve state-of-the-art performance in assessing the low-frequency degradation information.
  arXiv  Detail & Related papers  (2024-05-07T13:35:51Z)
• Large Multi-modality Model Assisted AI-Generated Image Quality Assessment [53.182136445844904]
  We introduce a large Multi-modality model Assisted AI-Generated Image Quality Assessment (MA-AGIQA) model. It uses semantically informed guidance to sense semantic information and extract semantic vectors through carefully designed text prompts. It achieves state-of-the-art performance, and demonstrates its superior generalization capabilities on assessing the quality of AI-generated images.
  arXiv  Detail & Related papers  (2024-04-27T02:40:36Z)
• When No-Reference Image Quality Models Meet MAP Estimation in Diffusion Latents [92.45867913876691]
  No-reference image quality assessment (NR-IQA) models can effectively quantify perceived image quality. We show that NR-IQA models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement.
  arXiv  Detail & Related papers  (2024-03-11T03:35:41Z)
• Q-Refine: A Perceptual Quality Refiner for AI-Generated Image [85.89840673640028]
  A quality-award refiner named Q-Refine is proposed. It uses the Image Quality Assessment (IQA) metric to guide the refining process for the first time. It can be a general refiner to optimize AIGIs from both fidelity and aesthetic quality levels.
  arXiv  Detail & Related papers  (2024-01-02T09:11:23Z)
• Conformer and Blind Noisy Students for Improved Image Quality Assessment [80.57006406834466]
  Learning-based approaches for perceptual image quality assessment (IQA) usually require both the distorted and reference image for measuring the perceptual quality accurately. In this work, we explore the performance of transformer-based full-reference IQA models. We also propose a method for IQA based on semi-supervised knowledge distillation from full-reference teacher models into blind student models.
  arXiv  Detail & Related papers  (2022-04-27T10:21:08Z)
• Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild [98.48284827503409]
  We develop a textitunified BIQA model and an approach of training it for both synthetic and realistic distortions. We employ the fidelity loss to optimize a deep neural network for BIQA over a large number of such image pairs. Experiments on six IQA databases show the promise of the learned method in blindly assessing image quality in the laboratory and wild.
  arXiv  Detail & Related papers  (2020-05-28T13:35:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.