GenDeg: Diffusion-Based Degradation Synthesis for Generalizable All-in-One Image Restoration

• URL: http://arxiv.org/abs/2411.17687v1
• Date: Tue, 26 Nov 2024 18:55:49 GMT
• Title: GenDeg: Diffusion-Based Degradation Synthesis for Generalizable All-in-One Image Restoration
• Authors: Sudarshan Rajagopalan, Nithin Gopalakrishnan Nair, Jay N. Paranjape, Vishal M. Patel,
• Abstract summary: We introduce GenDeg, a conditional diffusion model capable of producing diverse degradation patterns on clean images. We synthesize over 550k samples across six degradation types: haze, rain, snow, motion blur, low-light, and raindrops. Our experiments reveal that image restoration models trained on the GenDS dataset exhibit significant improvements in out-of-distribution performance.
• Score: 26.434359848151978
• License:
• Abstract: Deep learning-based models for All-In-One Image Restoration (AIOR) have achieved significant advancements in recent years. However, their practical applicability is limited by poor generalization to samples outside the training distribution. This limitation arises primarily from insufficient diversity in degradation variations and scenes within existing datasets, resulting in inadequate representations of real-world scenarios. Additionally, capturing large-scale real-world paired data for degradations such as haze, low-light, and raindrops is often cumbersome and sometimes infeasible. In this paper, we leverage the generative capabilities of latent diffusion models to synthesize high-quality degraded images from their clean counterparts. Specifically, we introduce GenDeg, a degradation and intensity-aware conditional diffusion model capable of producing diverse degradation patterns on clean images. Using GenDeg, we synthesize over 550k samples across six degradation types: haze, rain, snow, motion blur, low-light, and raindrops. These generated samples are integrated with existing datasets to form the GenDS dataset, comprising over 750k samples. Our experiments reveal that image restoration models trained on the GenDS dataset exhibit significant improvements in out-of-distribution performance compared to those trained solely on existing datasets. Furthermore, we provide comprehensive analyses on the implications of diffusion model-based synthetic degradations for AIOR. The code will be made publicly available.

Related papers

• Data-free Distillation with Degradation-prompt Diffusion for Multi-weather Image Restoration [29.731089599252954]
  We propose a novel Data-free Distillation with Degradation-prompt Diffusion framework for multi-weather Image Restoration (D4IR) It replaces GANs with pre-trained diffusion models to avoid model collapse and incorporates a degradation-aware prompt adapter. Our proposal achieves comparable performance to the model distilled with original training data, and is even superior to other mainstream unsupervised methods.
  arXiv  Detail & Related papers  (2024-09-05T12:07:17Z)
• Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modeling [10.159932782892865]
  generative models have reached a unique threshold where their outputs are indistinguishable from real data. Severe degradation in performance has been observed when iterative loops of training and generation occur. We propose Reusable Diffusion Finetuning (ReDiFine), a simple yet effective strategy inspired by genetic mutations.
  arXiv  Detail & Related papers  (2024-07-04T13:41:54Z)
• Photo-Realistic Image Restoration in the Wild with Controlled Vision-Language Models [14.25759541950917]
  This work leverages a capable vision-language model and a synthetic degradation pipeline to learn image restoration in the wild (wild IR) Our base diffusion model is the image restoration SDE (IR-SDE)
  arXiv  Detail & Related papers  (2024-04-15T12:34:21Z)
• Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control [54.132297393662654]
  Diffusion models excel at capturing complex data distributions, such as those of natural images and proteins. While diffusion models are trained to represent the distribution in the training dataset, we often are more concerned with other properties, such as the aesthetic quality of the generated images. We present theoretical and empirical evidence that demonstrates our framework is capable of efficiently generating diverse samples with high genuine rewards.
  arXiv  Detail & Related papers  (2024-02-23T08:54:42Z)
• Learned representation-guided diffusion models for large-image generation [58.192263311786824]
  We introduce a novel approach that trains diffusion models conditioned on embeddings from self-supervised learning (SSL) Our diffusion models successfully project these features back to high-quality histopathology and remote sensing images. Augmenting real data by generating variations of real images improves downstream accuracy for patch-level and larger, image-scale classification tasks.
  arXiv  Detail & Related papers  (2023-12-12T14:45:45Z)
• Improving the Effectiveness of Deep Generative Data [5.856292656853396]
  Training a model on purely synthetic images for downstream image processing tasks results in an undesired performance drop compared to training on real data. We propose a new taxonomy to describe factors contributing to this commonly observed phenomenon and investigate it on the popular CIFAR-10 dataset. Our method outperforms baselines on downstream classification tasks both in case of training on synthetic only (Synthetic-to-Real) and training on a mix of real and synthetic data.
  arXiv  Detail & Related papers  (2023-11-07T12:57:58Z)
• Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis [62.07413805483241]
  Steered Diffusion is a framework for zero-shot conditional image generation using a diffusion model trained for unconditional generation. We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution.
  arXiv  Detail & Related papers  (2023-09-30T02:03:22Z)
• DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration [66.01846902242355]
  Blind face restoration usually synthesizes degraded low-quality data with a pre-defined degradation model for training. It is expensive and infeasible to include every type of degradation to cover real-world cases in the training data. We propose Robust Degradation Remover (DR2) to first transform the degraded image to a coarse but degradation-invariant prediction, then employ an enhancement module to restore the coarse prediction to a high-quality image.
  arXiv  Detail & Related papers  (2023-03-13T06:05:18Z)
• Rethinking Real-world Image Deraining via An Unpaired Degradation-Conditioned Diffusion Model [51.49854435403139]
  We propose RainDiff, the first real-world image deraining paradigm based on diffusion models. We introduce a stable and non-adversarial unpaired cycle-consistent architecture that can be trained, end-to-end, with only unpaired data for supervision. We also propose a degradation-conditioned diffusion model that refines the desired output via a diffusive generative process conditioned by learned priors of multiple rain degradations.
  arXiv  Detail & Related papers  (2023-01-23T13:34:01Z)
• Improved Autoregressive Modeling with Distribution Smoothing [106.14646411432823]
  Autoregressive models excel at image compression, but their sample quality is often lacking. Inspired by a successful adversarial defense method, we incorporate randomized smoothing into autoregressive generative modeling.
  arXiv  Detail & Related papers  (2021-03-28T09:21:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.