Rain rendering for evaluating and improving robustness to bad weather

• URL: http://arxiv.org/abs/2009.03683v1
• Date: Sun, 6 Sep 2020 21:08:41 GMT
• Title: Rain rendering for evaluating and improving robustness to bad weather
• Authors: Maxime Tremblay, Shirsendu Sukanta Halder, Raoul de Charette, Jean-Fran\c{c}ois Lalonde
• Abstract summary: We present a rain rendering pipeline that enables the systematic evaluation of common computer vision algorithms to controlled amounts of rain. The physic-based rain augmentation combines a physical particle simulator and accurate rain photometric modeling. Using our generated rain-augmented KITTI, Cityscapes, and nuScenes datasets, we conduct a thorough evaluation of object detection, semantic segmentation, and depth estimation algorithms.
• Score: 20.43775471447035
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rain fills the atmosphere with water particles, which breaks the common assumption that light travels unaltered from the scene to the camera. While it is well-known that rain affects computer vision algorithms, quantifying its impact is difficult. In this context, we present a rain rendering pipeline that enables the systematic evaluation of common computer vision algorithms to controlled amounts of rain. We present three different ways to add synthetic rain to existing images datasets: completely physic-based; completely data-driven; and a combination of both. The physic-based rain augmentation combines a physical particle simulator and accurate rain photometric modeling. We validate our rendering methods with a user study, demonstrating our rain is judged as much as 73% more realistic than the state-of-theart. Using our generated rain-augmented KITTI, Cityscapes, and nuScenes datasets, we conduct a thorough evaluation of object detection, semantic segmentation, and depth estimation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection, 60% for semantic segmentation, and 6-fold increase in depth estimation error. Finetuning on our augmented synthetic data results in improvements of 21% on object detection, 37% on semantic segmentation, and 8% on depth estimation.

Related papers

• RainyScape: Unsupervised Rainy Scene Reconstruction using Decoupled Neural Rendering [50.14860376758962]
  We propose RainyScape, an unsupervised framework for reconstructing clean scenes from a collection of multi-view rainy images. Based on the spectral bias property of neural networks, we first optimize the neural rendering pipeline to obtain a low-frequency scene representation. We jointly optimize the two modules, driven by the proposed adaptive direction-sensitive gradient-based reconstruction loss.
  arXiv  Detail & Related papers  (2024-04-17T14:07:22Z)
• Why current rain denoising models fail on CycleGAN created rain images in autonomous driving [1.4831974871130875]
  Rain is artificially added to a set of clear-weather condition images using a Generative Adversarial Network (GAN) This artificial generation of rain images is sufficiently realistic as in 7 out of 10 cases, human test subjects believed the generated rain images to be real. In a second step, this paired good/bad weather image data is used to train two rain denoising models, one based primarily on a Convolutional Neural Network (CNN) and the other using a Vision Transformer.
  arXiv  Detail & Related papers  (2023-05-22T12:42:32Z)
• Not Just Streaks: Towards Ground Truth for Single Image Deraining [42.15398478201746]
  We propose a large-scale dataset of real-world rainy and clean image pairs. We propose a deep neural network that reconstructs the underlying scene by minimizing a rain-robust loss between rainy and clean images.
  arXiv  Detail & Related papers  (2022-06-22T00:10:06Z)
• Semi-MoreGAN: A New Semi-supervised Generative Adversarial Network for Mixture of Rain Removal [18.04268933542476]
  We propose a new SEMI-supervised Mixture Of rain REmoval Generative Adversarial Network (Semi-MoreGAN) Semi-MoreGAN consists of four key modules: (I) a novel attentional depth prediction network to provide precise depth estimation; (ii) a context feature prediction network composed of several well-designed detailed residual blocks to produce detailed image context features; (iii) a pyramid depth-guided non-local network to effectively integrate the image context with the depth information, and produce the final rain-free images; and (iv) a comprehensive semi-supervised loss function to make the model not limited
  arXiv  Detail & Related papers  (2022-04-28T11:35:26Z)
• Towards Robust Rain Removal Against Adversarial Attacks: A Comprehensive Benchmark Analysis and Beyond [85.06231315901505]
  Rain removal aims to remove rain streaks from images/videos and reduce the disruptive effects caused by rain. This paper makes the first attempt to conduct a comprehensive study on the robustness of deep learning-based rain removal methods against adversarial attacks.
  arXiv  Detail & Related papers  (2022-03-31T10:22:24Z)
• UnfairGAN: An Enhanced Generative Adversarial Network for Raindrop Removal from A Single Image [8.642603456626391]
  UnfairGAN is an enhanced generative adversarial network that can utilize prior high-level information, such as edges and rain estimation, to boost deraining performance. We show that our proposed method is superior to other state-of-the-art approaches of deraining raindrops regarding quantitative metrics and visual quality.
  arXiv  Detail & Related papers  (2021-10-11T18:02:43Z)
• Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
  Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars. They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
  arXiv  Detail & Related papers  (2021-07-14T21:10:47Z)
• RCDNet: An Interpretable Rain Convolutional Dictionary Network for Single Image Deraining [49.99207211126791]
  We specifically build a novel deep architecture, called rain convolutional dictionary network (RCDNet) RCDNet embeds the intrinsic priors of rain streaks and has clear interpretability. By end-to-end training such an interpretable network, all involved rain kernels and proximal operators can be automatically extracted.
  arXiv  Detail & Related papers  (2021-07-14T16:08:11Z)
• Raindrops on Windshield: Dataset and Lightweight Gradient-Based Detection Algorithm [0.0]
  We present a new dataset for training and assessing vision algorithms' performance for different tasks of image artifacts detection on either camera lens or windshield. To augment the data, we also propose an algorithm for data augmentation which allows the generation of synthetic raindrops on images. The proposed algorithm showed a higher quality of raindrop presence detection and image processing speed, making it applicable for the self-check procedure of real autonomous systems.
  arXiv  Detail & Related papers  (2021-04-11T19:04:59Z)
• From Rain Generation to Rain Removal [67.71728610434698]
  We build a full Bayesian generative model for rainy image where the rain layer is parameterized as a generator. We employ the variational inference framework to approximate the expected statistical distribution of rainy image. Comprehensive experiments substantiate that the proposed model can faithfully extract the complex rain distribution.
  arXiv  Detail & Related papers  (2020-08-08T18:56:51Z)
• Structural Residual Learning for Single Image Rain Removal [48.87977695398587]
  This study proposes a new network architecture by enforcing the output residual of the network possess intrinsic rain structures. Such a structural residual setting guarantees the rain layer extracted by the network finely comply with the prior knowledge of general rain streaks.
  arXiv  Detail & Related papers  (2020-05-19T05:52:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.