ConStyle v2: A Strong Prompter for All-in-One Image Restoration

• URL: http://arxiv.org/abs/2406.18242v1
• Date: Wed, 26 Jun 2024 10:46:44 GMT
• Title: ConStyle v2: A Strong Prompter for All-in-One Image Restoration
• Authors: Dongqi Fan, Junhao Zhang, Liang Chang,
• Abstract summary: This paper introduces ConStyle v2, a strong plug-and-play prompter for U-Net Image Restoration models. Experiments show that ConStyle v2 can enhance any U-Net style Image Restoration models to all-in-one Image Restoration models.
• Score: 5.693207891187567
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces ConStyle v2, a strong plug-and-play prompter designed to output clean visual prompts and assist U-Net Image Restoration models in handling multiple degradations. The joint training process of IRConStyle, an Image Restoration framework consisting of ConStyle and a general restoration network, is divided into two stages: first, pre-training ConStyle alone, and then freezing its weights to guide the training of the general restoration network. Three improvements are proposed in the pre-training stage to train ConStyle: unsupervised pre-training, adding a pretext task (i.e. classification), and adopting knowledge distillation. Without bells and whistles, we can get ConStyle v2, a strong prompter for all-in-one Image Restoration, in less than two GPU days and doesn't require any fine-tuning. Extensive experiments on Restormer (transformer-based), NAFNet (CNN-based), MAXIM-1S (MLP-based), and a vanilla CNN network demonstrate that ConStyle v2 can enhance any U-Net style Image Restoration models to all-in-one Image Restoration models. Furthermore, models guided by the well-trained ConStyle v2 exhibit superior performance in some specific degradation compared to ConStyle.

Related papers

• Dual Prompting Image Restoration with Diffusion Transformers [45.159373436771]
  DPIR (Dual Prompting Image Restoration) is a novel image restoration method that effectivly extracts conditional information of low-quality images from multiple perspectives. The extracted global-local visual prompts as extra conditional control, alongside textual prompts to form dual prompts, greatly enhance the quality of the restoration.
  arXiv  Detail & Related papers  (2025-04-24T02:34:44Z)
• Review Learning: Advancing All-in-One Ultra-High-Definition Image Restoration Training Method [7.487270862599671]
  We propose a new training paradigm for general image restoration models, which we name bfReview Learning. This approach begins with sequential training of an image restoration model on several degraded datasets, combined with a review mechanism. We design a lightweight all-purpose image restoration network that can efficiently reason about degraded images with 4K resolution on a single consumer-grade GPU.
  arXiv  Detail & Related papers  (2024-08-13T08:08:45Z)
• MuseumMaker: Continual Style Customization without Catastrophic Forgetting [50.12727620780213]
  We propose MuseumMaker, a method that enables the synthesis of images by following a set of customized styles in a never-end manner. When facing with a new customization style, we develop a style distillation loss module to extract and learn the styles of the training data for new image generation. It can minimize the learning biases caused by content of new training images, and address the catastrophic overfitting issue induced by few-shot images.
  arXiv  Detail & Related papers  (2024-04-25T13:51:38Z)
• MoreStyle: Relax Low-frequency Constraint of Fourier-based Image Reconstruction in Generalizable Medical Image Segmentation [53.24011398381715]
  We introduce a Plug-and-Play module for data augmentation called MoreStyle. MoreStyle diversifies image styles by relaxing low-frequency constraints in Fourier space. With the help of adversarial learning, MoreStyle pinpoints the most intricate style combinations within latent features.
  arXiv  Detail & Related papers  (2024-03-18T11:38:47Z)
• IRConStyle: Image Restoration Framework Using Contrastive Learning and Style Transfer [5.361977985410345]
  We propose a novel module for image restoration called textbfConStyle, which can be efficiently integrated into any U-Net structure network. We perform extensive experiments on various image restoration tasks, including denoising, deraining, and dehazing. The results on 19 benchmarks demonstrate that ConStyle can be integrated with any U-Net-based network and significantly enhance performance.
  arXiv  Detail & Related papers  (2024-02-24T10:52:50Z)
• InstructIR: High-Quality Image Restoration Following Human Instructions [61.1546287323136]
  We present the first approach that uses human-written instructions to guide the image restoration model. Our method, InstructIR, achieves state-of-the-art results on several restoration tasks.
  arXiv  Detail & Related papers  (2024-01-29T18:53:33Z)
• Controlling Vision-Language Models for Multi-Task Image Restoration [6.239038964461397]
  We present a degradation-aware vision-language model (DA-CLIP) to better transfer pretrained vision-language models to low-level vision tasks. Our approach advances state-of-the-art performance on both emphdegradation-specific and emphunified image restoration tasks.
  arXiv  Detail & Related papers  (2023-10-02T09:10:16Z)
• StyleAdapter: A Unified Stylized Image Generation Model [97.24936247688824]
  StyleAdapter is a unified stylized image generation model capable of producing a variety of stylized images. It can be integrated with existing controllable synthesis methods, such as T2I-adapter and ControlNet.
  arXiv  Detail & Related papers  (2023-09-04T19:16:46Z)
• PromptIR: Prompting for All-in-One Blind Image Restoration [64.02374293256001]
  We present a prompt-based learning approach, PromptIR, for All-In-One image restoration. Our method uses prompts to encode degradation-specific information, which is then used to dynamically guide the restoration network. PromptIR offers a generic and efficient plugin module with few lightweight prompts.
  arXiv  Detail & Related papers  (2023-06-22T17:59:52Z)
• Third Time's the Charm? Image and Video Editing with StyleGAN3 [70.36056009463738]
  StyleGAN is arguably one of the most intriguing and well-studied generative models. We explore the recent StyleGAN3 architecture, compare it to its predecessor, and investigate its unique advantages, as well as drawbacks.
  arXiv  Detail & Related papers  (2022-01-31T18:44:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.