Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning

• URL: http://arxiv.org/abs/2412.11952v1
• Date: Mon, 16 Dec 2024 16:35:35 GMT
• Title: Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning
• Authors: Yuti Liu, Shice Liu, Junyuan Gao, Pengtao Jiang, Hao Zhang, Jinwei Chen, Bo Li,
• Abstract summary: Image Aesthetic Assessment (IAA) is a vital and intricate task that entails analyzing and assessing an image's aesthetic values. Traditional methods of IAA often concentrate on a single aesthetic task and suffer from inadequate labeled datasets. We propose a comprehensive aesthetic MLLM capable of nuanced aesthetic insight.
• Score: 14.405750888492735
• License:
• Abstract: Image Aesthetic Assessment (IAA) is a vital and intricate task that entails analyzing and assessing an image's aesthetic values, and identifying its highlights and areas for improvement. Traditional methods of IAA often concentrate on a single aesthetic task and suffer from inadequate labeled datasets, thus impairing in-depth aesthetic comprehension. Despite efforts to overcome this challenge through the application of Multi-modal Large Language Models (MLLMs), such models remain underdeveloped for IAA purposes. To address this, we propose a comprehensive aesthetic MLLM capable of nuanced aesthetic insight. Central to our approach is an innovative multi-scale text-guided self-supervised learning technique. This technique features a multi-scale feature alignment module and capitalizes on a wealth of unlabeled data in a self-supervised manner to structurally and functionally enhance aesthetic ability. The empirical evidence indicates that accompanied with extensive instruct-tuning, our model sets new state-of-the-art benchmarks across multiple tasks, including aesthetic scoring, aesthetic commenting, and personalized image aesthetic assessment. Remarkably, it also demonstrates zero-shot learning capabilities in the emerging task of aesthetic suggesting. Furthermore, for personalized image aesthetic assessment, we harness the potential of in-context learning and showcase its inherent advantages.

Related papers

• Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms [91.19304518033144]
  We aim to align vision models with human aesthetic standards in a retrieval system. We propose a preference-based reinforcement learning method that fine-tunes the vision models to better align the vision models with human aesthetics.
  arXiv  Detail & Related papers  (2024-06-13T17:59:20Z)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension [131.14381425260706]
  We introduce Self-Training on Image (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference for image descriptions using unlabeled images. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data.
  arXiv  Detail & Related papers  (2024-05-30T05:53:49Z)
• AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception [74.11069437400398]
  We develop a corpus-rich aesthetic critique database with 21,904 diverse-sourced images and 88K human natural language feedbacks. We fine-tune the open-sourced general foundation models, achieving multi-modality Aesthetic Expert models, dubbed AesExpert. Experiments demonstrate that the proposed AesExpert models deliver significantly better aesthetic perception performances than the state-of-the-art MLLMs.
  arXiv  Detail & Related papers  (2024-04-15T09:56:20Z)
• AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception [64.25808552299905]
  AesBench is an expert benchmark aiming to comprehensively evaluate the aesthetic perception capacities of MLLMs. We construct an Expert-labeled Aesthetics Perception Database (EAPD), which features diversified image contents and high-quality annotations provided by professional aesthetic experts. We propose a set of integrative criteria to measure the aesthetic perception abilities of MLLMs from four perspectives, including Perception (AesP), Empathy (AesE), Assessment (AesA) and Interpretation (AesI)
  arXiv  Detail & Related papers  (2024-01-16T10:58:07Z)
• Unveiling The Factors of Aesthetic Preferences with Explainable AI [0.0]
  In this study, we pioneer a novel perspective by utilizing several different machine learning (ML) models. Our models process these attributes as inputs to predict the aesthetic scores of images. Our aim is to shed light on the complex nature of aesthetic preferences in images through ML and to provide a deeper understanding of the attributes that influence aesthetic judgements.
  arXiv  Detail & Related papers  (2023-11-24T11:06:22Z)
• Image Aesthetics Assessment via Learnable Queries [59.313054821874864]
  We propose the Image Aesthetics Assessment via Learnable Queries (IAA-LQ) approach. It adapts learnable queries to extract aesthetic features from pre-trained image features obtained from a frozen image encoder. Experiments on real-world data demonstrate the advantages of IAA-LQ, beating the best state-of-the-art method by 2.2% and 2.1% in terms of SRCC and PLCC, respectively.
  arXiv  Detail & Related papers  (2023-09-06T09:42:16Z)
• ALL-E: Aesthetics-guided Low-light Image Enhancement [45.40896781156727]
  We propose a new paradigm, i.e. aesthetics-guided low-light image enhancement (ALL-E) It introduces aesthetic preferences to LLE and motivates training in a reinforcement learning framework with an aesthetic reward. Our results on various benchmarks demonstrate the superiority of ALL-E over state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-04-28T03:34:10Z)
• VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining [53.470662123170555]
  We propose learning image aesthetics from user comments, and exploring vision-language pretraining methods to learn multimodal aesthetic representations. Specifically, we pretrain an image-text encoder-decoder model with image-comment pairs, using contrastive and generative objectives to learn rich and generic aesthetic semantics without human labels. Our results show that our pretrained aesthetic vision-language model outperforms prior works on image aesthetic captioning over the AVA-Captions dataset.
  arXiv  Detail & Related papers  (2023-03-24T23:57:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.