CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment

• URL: http://arxiv.org/abs/2404.01123v1
• Date: Mon, 1 Apr 2024 13:57:46 GMT
• Title: CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment
• Authors: Hyeongmin Lee, Kyoungkook Kang, Jungseul Ok, Sunghyun Cho,
• Abstract summary: We propose an unsupervised learning-based approach for text-based image tone adjustment method, CLIPtone. Our approach's efficacy is demonstrated through comprehensive experiments, including a user study.
• Score: 23.36770607997754
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent image tone adjustment (or enhancement) approaches have predominantly adopted supervised learning for learning human-centric perceptual assessment. However, these approaches are constrained by intrinsic challenges of supervised learning. Primarily, the requirement for expertly-curated or retouched images escalates the data acquisition expenses. Moreover, their coverage of target style is confined to stylistic variants inferred from the training data. To surmount the above challenges, we propose an unsupervised learning-based approach for text-based image tone adjustment method, CLIPtone, that extends an existing image enhancement method to accommodate natural language descriptions. Specifically, we design a hyper-network to adaptively modulate the pretrained parameters of the backbone model based on text description. To assess whether the adjusted image aligns with the text description without ground truth image, we utilize CLIP, which is trained on a vast set of language-image pairs and thus encompasses knowledge of human perception. The major advantages of our approach are three fold: (i) minimal data collection expenses, (ii) support for a range of adjustments, and (iii) the ability to handle novel text descriptions unseen in training. Our approach's efficacy is demonstrated through comprehensive experiments, including a user study.

Related papers

• Debiasing Vison-Language Models with Text-Only Training [15.069736314663352]
  We propose a Text-Only Debiasing framework called TOD, leveraging a text-as-image training paradigm to mitigate visual biases. To address the limitations, we propose a Text-Only Debiasing framework called TOD, leveraging a text-as-image training paradigm to mitigate visual biases.
  arXiv  Detail & Related papers  (2024-10-12T04:34:46Z)
• Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control [73.6361029556484]
  Embodied AI agents require a fine-grained understanding of the physical world mediated through visual and language inputs. We consider pre-trained text-to-image diffusion models, which are explicitly optimized to generate images from text prompts. We show that Stable Control Representations enable learning policies that exhibit state-of-the-art performance on OVMM, a difficult open-vocabulary navigation benchmark.
  arXiv  Detail & Related papers  (2024-05-09T15:39:54Z)
• Enhancing Image Retrieval : A Comprehensive Study on Photo Search using the CLIP Mode [0.27195102129095]
  Photo search has witnessed significant advancements with the introduction of CLIP (Contrastive Language-Image Pretraining) model. This abstract summarizes the foundational principles of CLIP and highlights its potential impact on advancing the field of photo search.
  arXiv  Detail & Related papers  (2024-01-24T17:35:38Z)
• Training-free Zero-shot Composed Image Retrieval with Local Concept Reranking [34.31345844296072]
  Composed image retrieval attempts to retrieve an image of interest from gallery images through a composed query of a reference image and its corresponding modified text. Most current composed image retrieval methods follow a supervised learning approach to training on a costly triplet dataset composed of a reference image, modified text, and a corresponding target image. We present a new training-free zero-shot composed image retrieval method which translates the query into explicit human-understandable text.
  arXiv  Detail & Related papers  (2023-12-14T13:31:01Z)
• Adapt and Align to Improve Zero-Shot Sketch-Based Image Retrieval [85.39613457282107]
  Cross-domain nature of sketch-based image retrieval is challenging. We present an effective Adapt and Align'' approach to address the key challenges. Inspired by recent advances in image-text foundation models (e.g., CLIP) on zero-shot scenarios, we explicitly align the learned image embedding with a more semantic text embedding to achieve the desired knowledge transfer from seen to unseen classes.
  arXiv  Detail & Related papers  (2023-05-09T03:10:15Z)
• CPL: Counterfactual Prompt Learning for Vision and Language Models [76.18024920393245]
  This paper presents a novel underlinetextbfCounterfactual underlinetextbfPrompt underlinetextbfLearning (CPL) method for vision and language models. CPL simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework. Experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks.
  arXiv  Detail & Related papers  (2022-10-19T08:06:39Z)
• Prompt-based Learning for Unpaired Image Captioning [86.44188293709307]
  Unpaired Image Captioning (UIC) has been developed to learn image descriptions from unaligned vision-language sample pairs. Recent successes of Vision-Language Pre-Trained Models (VL-PTMs) have triggered the development of prompt-based learning. We present in this paper a novel scheme based on prompt to train the UIC model, making best use of the powerful generalization ability.
  arXiv  Detail & Related papers  (2022-05-26T03:13:43Z)
• Deep Learning Approaches on Image Captioning: A Review [0.5852077003870417]
  Image captioning aims to generate natural language descriptions for visual content in the form of still images. Deep learning and vision-language pre-training techniques have revolutionized the field, leading to more sophisticated methods and improved performance. We address the challenges faced in this field by emphasizing issues such as object hallucination, missing context, illumination conditions, contextual understanding, and referring expressions. We identify several potential future directions for research in this area, which include tackling the information misalignment problem between image and text modalities, mitigating dataset bias, incorporating vision-language pre-training methods to enhance caption generation, and developing improved evaluation tools to accurately
  arXiv  Detail & Related papers  (2022-01-31T00:39:37Z)
• Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model [168.04947140367258]
  We propose a novel framework, i.e., Predict, Prevent, and Evaluate (PPE), for disentangled text-driven image manipulation. Our method approaches the targets by exploiting the power of the large scale pre-trained vision-language model CLIP. Extensive experiments show that the proposed PPE framework achieves much better quantitative and qualitative results than the up-to-date StyleCLIP baseline.
  arXiv  Detail & Related papers  (2021-11-26T06:49:26Z)
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters [79.52844563138493]
  We show that there is an alternative path to achieve better vision-language models other than prompt tuning. In this paper, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2021-10-09T11:39:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.