Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models

• URL: http://arxiv.org/abs/2312.08962v3
• Date: Sun, 14 Jul 2024 12:33:05 GMT
• Title: Depicting Beyond Scores: Advancing Image Quality Assessment through Multi-modal Language Models
• Authors: Zhiyuan You, Zheyuan Li, Jinjin Gu, Zhenfei Yin, Tianfan Xue, Chao Dong,
• Abstract summary: We introduce a Depicted image Quality Assessment method (DepictQA), overcoming the constraints of traditional score-based methods. DepictQA allows for detailed, language-based, human-like evaluation of image quality by leveraging Multi-modal Large Language Models. These results showcase the research potential of multi-modal IQA methods.
• Score: 28.194638379354252
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a Depicted image Quality Assessment method (DepictQA), overcoming the constraints of traditional score-based methods. DepictQA allows for detailed, language-based, human-like evaluation of image quality by leveraging Multi-modal Large Language Models (MLLMs). Unlike conventional Image Quality Assessment (IQA) methods relying on scores, DepictQA interprets image content and distortions descriptively and comparatively, aligning closely with humans' reasoning process. To build the DepictQA model, we establish a hierarchical task framework, and collect a multi-modal IQA training dataset. To tackle the challenges of limited training data and multi-image processing, we propose to use multi-source training data and specialized image tags. These designs result in a better performance of DepictQA than score-based approaches on multiple benchmarks. Moreover, compared with general MLLMs, DepictQA can generate more accurate reasoning descriptive languages. We also demonstrate that our full-reference dataset can be extended to non-reference applications. These results showcase the research potential of multi-modal IQA methods. Codes and datasets are available in https://depictqa.github.io.

Related papers

• Grounding-IQA: Multimodal Language Grounding Model for Image Quality Assessment [69.07445098168344]
  We introduce a new image quality assessment (IQA) task paradigm, grounding-IQA. Grounding-IQA comprises two subtasks: grounding-IQA-description (GIQA-DES) and visual question answering (GIQA-VQA) To realize grounding-IQA, we construct a corresponding dataset, GIQA-160K, through our proposed automated annotation pipeline. Experiments demonstrate that our proposed task paradigm, dataset, and benchmark facilitate the more fine-grained IQA application.
  arXiv  Detail & Related papers  (2024-11-26T09:03:16Z)
• ATTIQA: Generalizable Image Quality Feature Extractor using Attribute-aware Pretraining [25.680035174334886]
  In no-reference image quality assessment (NR-IQA), the challenge of limited dataset sizes hampers the development of robust and generalizable models. We propose a novel pretraining framework that constructs a generalizable representation for IQA by selectively extracting quality-related knowledge. Our approach achieves state-of-the-art performance on multiple IQA datasets and exhibits remarkable generalization capabilities.
  arXiv  Detail & Related papers  (2024-06-03T06:03:57Z)
• Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare [99.57567498494448]
  We introduce Compare2Score, an all-around LMM-based no-reference IQA model. During training, we generate scaled-up comparative instructions by comparing images from the same IQA dataset. Experiments on nine IQA datasets validate that the Compare2Score effectively bridges text-defined comparative levels during training.
  arXiv  Detail & Related papers  (2024-05-29T17:26:09Z)
• Descriptive Image Quality Assessment in the Wild [25.503311093471076]
  VLM-based Image Quality Assessment (IQA) seeks to describe image quality linguistically to align with human expression. We introduce Depicted image Quality Assessment in the Wild (DepictQA-Wild) Our method includes a multi-functional IQA task paradigm that encompasses both assessment and comparison tasks, brief and detailed responses, full-reference and non-reference scenarios.
  arXiv  Detail & Related papers  (2024-05-29T07:49:15Z)
• Opinion-Unaware Blind Image Quality Assessment using Multi-Scale Deep Feature Statistics [54.08757792080732]
  We propose integrating deep features from pre-trained visual models with a statistical analysis model to achieve opinion-unaware BIQA (OU-BIQA) Our proposed model exhibits superior consistency with human visual perception compared to state-of-the-art BIQA models.
  arXiv  Detail & Related papers  (2024-05-29T06:09:34Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective [93.56647950778357]
  Blind image quality assessment (BIQA) predicts the human perception of image quality without any reference information. We develop a general and automated multitask learning scheme for BIQA to exploit auxiliary knowledge from other tasks.
  arXiv  Detail & Related papers  (2023-03-27T07:58:09Z)
• Blind Multimodal Quality Assessment: A Brief Survey and A Case Study of Low-light Images [73.27643795557778]
  Blind image quality assessment (BIQA) aims at automatically and accurately forecasting objective scores for visual signals. Recent developments in this field are dominated by unimodal solutions inconsistent with human subjective rating patterns. We present a unique blind multimodal quality assessment (BMQA) of low-light images from subjective evaluation to objective score.
  arXiv  Detail & Related papers  (2023-03-18T09:04:55Z)
• Training and challenging models for text-guided fashion image retrieval [1.4266272677701561]
  We introduce a new evaluation dataset, Challenging Fashion Queries (CFQ) CFQ complements existing benchmarks by including relative captions with positive and negative labels of caption accuracy and conditional image similarity. We demonstrate the importance of multimodal pretraining for the task and show that domain-specific weak supervision based on attribute labels can augment generic large-scale pretraining.
  arXiv  Detail & Related papers  (2022-04-23T06:24:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.