Automatic Controllable Colorization via Imagination

• URL: http://arxiv.org/abs/2404.05661v1
• Date: Mon, 8 Apr 2024 16:46:07 GMT
• Title: Automatic Controllable Colorization via Imagination
• Authors: Xiaoyan Cong, Yue Wu, Qifeng Chen, Chenyang Lei,
• Abstract summary: We propose a framework for automatic colorization that allows for iterative editing and modifications. By understanding the content within a grayscale image, we utilize a pre-trained image generation model to generate multiple images that contain the same content. These images serve as references for coloring, mimicking the process of human experts.
• Score: 55.489416987587305
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a framework for automatic colorization that allows for iterative editing and modifications. The core of our framework lies in an imagination module: by understanding the content within a grayscale image, we utilize a pre-trained image generation model to generate multiple images that contain the same content. These images serve as references for coloring, mimicking the process of human experts. As the synthesized images can be imperfect or different from the original grayscale image, we propose a Reference Refinement Module to select the optimal reference composition. Unlike most previous end-to-end automatic colorization algorithms, our framework allows for iterative and localized modifications of the colorization results because we explicitly model the coloring samples. Extensive experiments demonstrate the superiority of our framework over existing automatic colorization algorithms in editability and flexibility. Project page: https://xy-cong.github.io/imagine-colorization.

Related papers

• Paint Bucket Colorization Using Anime Character Color Design Sheets [72.66788521378864]
  We introduce inclusion matching, which allows the network to understand the relationships between segments. Our network's training pipeline significantly improves performance in both colorization and consecutive frame colorization. To support our network's training, we have developed a unique dataset named PaintBucket-Character.
  arXiv  Detail & Related papers  (2024-10-25T09:33:27Z)
• Palette-based Color Transfer between Images [9.471264982229508]
  We propose a new palette-based color transfer method that can automatically generate a new color scheme. With a redesigned palette-based clustering method, pixels can be classified into different segments according to color distribution. Our method exhibits significant advantages over peer methods in terms of natural realism, color consistency, generality, and robustness.
  arXiv  Detail & Related papers  (2024-05-14T01:41:19Z)
• Control Color: Multimodal Diffusion-based Interactive Image Colorization [81.68817300796644]
  Control Color (Ctrl Color) is a multi-modal colorization method that leverages the pre-trained Stable Diffusion (SD) model. We present an effective way to encode user strokes to enable precise local color manipulation. We also introduce a novel module based on self-attention and a content-guided deformable autoencoder to address the long-standing issues of color overflow and inaccurate coloring.
  arXiv  Detail & Related papers  (2024-02-16T17:51:13Z)
• Fine-Tuning InstructPix2Pix for Advanced Image Colorization [3.4975669723257035]
  This paper presents a novel approach to human image colorization by fine-tuning the InstructPix2Pix model. We fine-tune the model using the IMDB-WIKI dataset, pairing black-and-white images with a diverse set of colorization prompts generated by ChatGPT. After finetuning, our model outperforms the original InstructPix2Pix model on multiple metrics quantitatively.
  arXiv  Detail & Related papers  (2023-12-08T01:36:49Z)
• DiffColor: Toward High Fidelity Text-Guided Image Colorization with Diffusion Models [12.897939032560537]
  We propose a new method called DiffColor to recover vivid colors conditioned on a prompt text. We first fine-tune a pre-trained text-to-image model to generate colorized images using a CLIP-based contrastive loss. Then we try to obtain an optimized text embedding aligning the colorized image and the text prompt, and a fine-tuned diffusion model enabling high-quality image reconstruction. Our method can produce vivid and diverse colors with a few iterations, and keep the structure and background intact while having colors well-aligned with the target language guidance.
  arXiv  Detail & Related papers  (2023-08-03T09:38:35Z)
• BiSTNet: Semantic Image Prior Guided Bidirectional Temporal Feature Fusion for Deep Exemplar-based Video Colorization [70.14893481468525]
  We present an effective BiSTNet to explore colors of reference exemplars and utilize them to help video colorization. We first establish the semantic correspondence between each frame and the reference exemplars in deep feature space to explore color information from reference exemplars. We develop a mixed expert block to extract semantic information for modeling the object boundaries of frames so that the semantic image prior can better guide the colorization process.
  arXiv  Detail & Related papers  (2022-12-05T13:47:15Z)
• UniColor: A Unified Framework for Multi-Modal Colorization with Transformer [23.581502129504287]
  We introduce a two-stage colorization framework for incorporating various conditions into a single model. In the first stage, multi-modal conditions are converted into a common representation of hint points. In the second stage, we propose a Transformer-based network composed of Chroma-VQGAN and Hybrid-Transformer to generate diverse and high-quality colorization results conditioned on hint points.
  arXiv  Detail & Related papers  (2022-09-22T17:59:09Z)
• Exemplar-Based Image Colorization with A Learning Framework [7.793461393970992]
  We propose an automatic colorization method with a learning framework. It decouples the colorization process and learning process so as to generate various color styles for the same gray image. It achieves comparable performance against the state-of-the-art colorization algorithms.
  arXiv  Detail & Related papers  (2022-09-13T07:15:25Z)
• Towards Photorealistic Colorization by Imagination [48.82757902812846]
  We present a novel approach to automatic image colorization by imitating the imagination process of human experts. Our work produces more colorful and diverse results than state-of-the-art image colorization methods.
  arXiv  Detail & Related papers  (2021-08-20T14:28:37Z)
• Instance-aware Image Colorization [51.12040118366072]
  In this paper, we propose a method for achieving instance-aware colorization. Our network architecture leverages an off-the-shelf object detector to obtain cropped object images. We use a similar network to extract the full-image features and apply a fusion module to predict the final colors.
  arXiv  Detail & Related papers  (2020-05-21T17:59:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.