Removing Rain Streaks via Task Transfer Learning
- URL: http://arxiv.org/abs/2208.13133v1
- Date: Sun, 28 Aug 2022 03:32:17 GMT
- Title: Removing Rain Streaks via Task Transfer Learning
- Authors: Yinglong Wang and Chao Ma and Jianzhuang Liu
- Abstract summary: We first statistically explore why the supervised deraining models cannot generalize well to real rainy cases.
In connected tasks, the label for real data can be easily obtained.
Our core idea is to learn representations from real data through task transfer to improve deraining generalization.
- Score: 39.511454098771026
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Due to the difficulty in collecting paired real-world training data, image
deraining is currently dominated by supervised learning with synthesized data
generated by e.g., Photoshop rendering. However, the generalization to real
rainy scenes is usually limited due to the gap between synthetic and real-world
data. In this paper, we first statistically explore why the supervised
deraining models cannot generalize well to real rainy cases, and find the
substantial difference of synthetic and real rainy data. Inspired by our
studies, we propose to remove rain by learning favorable deraining
representations from other connected tasks. In connected tasks, the label for
real data can be easily obtained. Hence, our core idea is to learn
representations from real data through task transfer to improve deraining
generalization. We thus term our learning strategy as \textit{task transfer
learning}. If there are more than one connected tasks, we propose to reduce
model size by knowledge distillation. The pretrained models for the connected
tasks are treated as teachers, all their knowledge is distilled to a student
network, so that we reduce the model size, meanwhile preserve effective prior
representations from all the connected tasks. At last, the student network is
fine-tuned with minority of paired synthetic rainy data to guide the pretrained
prior representations to remove rain. Extensive experiments demonstrate that
proposed task transfer learning strategy is surprisingly successful and
compares favorably with state-of-the-art supervised learning methods and
apparently surpass other semi-supervised deraining methods on synthetic data.
Particularly, it shows superior generalization over them to real-world scenes.
Related papers
- Behaviour Distillation [10.437472004180883]
We formalize behaviour distillation, a setting that aims to discover and condense information required for training an expert policy into a synthetic dataset.
We then introduce Hallucinating datasets with Evolution Strategies (HaDES), a method for behaviour distillation that can discover datasets of just four state-action pairs.
We show that these datasets generalize out of distribution to training policies with a wide range of architectures.
We also demonstrate application to a downstream task, namely training multi-task agents in a zero-shot fashion.
arXiv Detail & Related papers (2024-06-21T10:45:43Z) - Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Data [2.6016285265085526]
Student models show a significant drop in accuracy compared to models trained on real data.
By training these layers using either real or synthetic data, we reveal that the drop mainly stems from the model's final layers.
Our results suggest an improved trade-off between the amount of real training data used and the model's accuracy.
arXiv Detail & Related papers (2024-05-06T07:51:13Z) - Rethinking Real-world Image Deraining via An Unpaired Degradation-Conditioned Diffusion Model [51.49854435403139]
We propose RainDiff, the first real-world image deraining paradigm based on diffusion models.
We introduce a stable and non-adversarial unpaired cycle-consistent architecture that can be trained, end-to-end, with only unpaired data for supervision.
We also propose a degradation-conditioned diffusion model that refines the desired output via a diffusive generative process conditioned by learned priors of multiple rain degradations.
arXiv Detail & Related papers (2023-01-23T13:34:01Z) - Not Just Streaks: Towards Ground Truth for Single Image Deraining [42.15398478201746]
We propose a large-scale dataset of real-world rainy and clean image pairs.
We propose a deep neural network that reconstructs the underlying scene by minimizing a rain-robust loss between rainy and clean images.
arXiv Detail & Related papers (2022-06-22T00:10:06Z) - Unsupervised Image Deraining: Optimization Model Driven Deep CNN [24.06511813683977]
We propose a unified unsupervised learning framework which inherits the generalization and representation merits for real rain removal.
We design an optimization model-driven deep CNN in which the unsupervised loss function of the optimization model is enforced on the proposed network.
The architecture of the network mimics the main role of the optimization models with better feature representation.
arXiv Detail & Related papers (2022-03-25T15:13:52Z) - A Scaling Law for Synthetic-to-Real Transfer: A Measure of Pre-Training [52.93808218720784]
Synthetic-to-real transfer learning is a framework in which we pre-train models with synthetically generated images and ground-truth annotations for real tasks.
Although synthetic images overcome the data scarcity issue, it remains unclear how the fine-tuning performance scales with pre-trained models.
We observe a simple and general scaling law that consistently describes learning curves in various tasks, models, and complexities of synthesized pre-training data.
arXiv Detail & Related papers (2021-08-25T02:29:28Z) - Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote
Sensing Data [64.40187171234838]
Seasonal Contrast (SeCo) is an effective pipeline to leverage unlabeled data for in-domain pre-training of re-mote sensing representations.
SeCo will be made public to facilitate transfer learning and enable rapid progress in re-mote sensing applications.
arXiv Detail & Related papers (2021-03-30T18:26:39Z) - Semi-Supervised Video Deraining with Dynamic Rain Generator [59.71640025072209]
This paper proposes a new semi-supervised video deraining method, in which a dynamic rain generator is employed to fit the rain layer.
Specifically, such dynamic generator consists of one emission model and one transition model to simultaneously encode the spatially physical structure and temporally continuous changes of rain streaks.
Various prior formats are designed for the labeled synthetic and unlabeled real data, so as to fully exploit the common knowledge underlying them.
arXiv Detail & Related papers (2021-03-14T14:28:57Z) - Syn2Real Transfer Learning for Image Deraining using Gaussian Processes [92.15895515035795]
CNN-based methods for image deraining have achieved excellent performance in terms of reconstruction error as well as visual quality.
Due to challenges in obtaining real world fully-labeled image deraining datasets, existing methods are trained only on synthetically generated data.
We propose a Gaussian Process-based semi-supervised learning framework which enables the network in learning to derain using synthetic dataset.
arXiv Detail & Related papers (2020-06-10T00:33:18Z) - Semi-DerainGAN: A New Semi-supervised Single Image Deraining Network [45.78251508028359]
We propose a new semi-supervised GAN-based deraining network termed Semi-DerainGAN.
It can use both synthetic and real rainy images in a uniform network using two supervised and unsupervised processes.
To deliver better deraining results, we design a paired discriminator for distinguishing the real pairs from fake pairs.
arXiv Detail & Related papers (2020-01-23T07:01:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.