Learning from Rendering: Realistic and Controllable Extreme Rainy Image Synthesis for Autonomous Driving Simulation

• URL: http://arxiv.org/abs/2502.16421v2
• Date: Thu, 31 Jul 2025 16:08:34 GMT
• Title: Learning from Rendering: Realistic and Controllable Extreme Rainy Image Synthesis for Autonomous Driving Simulation
• Authors: Kaibin Zhou, Kaifeng Huang, Hao Deng, Zelin Tao, Ziniu Liu, Lin Zhang, Shengjie Zhao,
• Abstract summary: Extreme weather conditions, particularly extreme rainfalls, are rare and costly to capture in real-world settings.<n>Existing rainy image synthesizers often suffer from poor controllability over illumination and limited realism.<n>We propose a learning-from-rendering rainy image synthesizer, which combines the benefits of rendering-based methods and the controllability of learning-based methods.
• Score: 8.83009075528098
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous driving simulators provide an effective and low-cost alternative for evaluating or enhancing visual perception models. However, the reliability of evaluation depends on the diversity and realism of the generated scenes. Extreme weather conditions, particularly extreme rainfalls, are rare and costly to capture in real-world settings. While simulated environments can help address this limitation, existing rainy image synthesizers often suffer from poor controllability over illumination and limited realism, which significantly undermines the effectiveness of the model evaluation. To that end, we propose a learning-from-rendering rainy image synthesizer, which combines the benefits of the realism of rendering-based methods and the controllability of learning-based methods. To validate the effectiveness of our extreme rainy image synthesizer on semantic segmentation task, we require a continuous set of well-labeled extreme rainy images. By integrating the proposed synthesizer with the CARLA driving simulator, we develop CARLARain an extreme rainy street scene simulator which can obtain paired rainy-clean images and labels under complex illumination conditions. Qualitative and quantitative experiments validate that CARLARain can effectively improve the accuracy of semantic segmentation models in extreme rainy scenes, with the models' accuracy (mIoU) improved by 5% - 8% on the synthetic dataset and significantly enhanced in real extreme rainy scenarios under complex illuminations. Our source code and datasets are available at https://github.com/kb824999404/CARLARain/.

Related papers

• Simulating Refractive Distortions and Weather-Induced Artifacts for Resource-Constrained Autonomous Perception [1.8463472137156713]
  We present a procedural augmentation pipeline that enhances low-cost monocular dashcam footage with realistic refractive distortions and weather-induced artifacts.<n>Our refractive module simulates optical effects from low-quality lenses and air turbulence, including lens distortion, Perlin noise, Thin-Plate Spline (TPS), and divergence-free (incompressible) warps.<n>To support perception research in underrepresented African contexts, without costly data collection, labeling, or simulation, we release our distortion toolkit, augmented dataset splits, and benchmark results.
  arXiv  Detail & Related papers  (2025-07-07T23:21:19Z)
• Semi-Supervised State-Space Model with Dynamic Stacking Filter for Real-World Video Deraining [73.5575992346396]
  We propose a dual-branch-temporal state-space model to enhance rain streak removal in video sequences.<n>To improve multi-frame feature fusion, we derive a dynamic filter stacking, which adaptively approximates statistical filters for pixel-wise feature refinement.<n>To further explore the capacity of deraining models in supporting other vision-based tasks in rainy environments, we introduce a novel real-world benchmark.
  arXiv  Detail & Related papers  (2025-05-22T15:50:00Z)
• Controllable Weather Synthesis and Removal with Video Diffusion Models [61.56193902622901]
  WeatherWeaver is a video diffusion model that synthesizes diverse weather effects directly into any input video.<n>Our model provides precise control over weather effect intensity and supports blending various weather types, ensuring both realism and adaptability.
  arXiv  Detail & Related papers  (2025-05-01T17:59:57Z)
• Enhancing autonomous vehicle safety in rain: a data-centric approach for clear vision [0.0]
  We developed a vision model that processes live vehicle camera feeds to eliminate rain-induced visual hindrances.<n>We employed a classic encoder-decoder architecture with skip connections and concatenation operations.<n>The results demonstrated notable improvements in steering accuracy, underscoring the model's potential to enhance navigation safety and reliability in rainy weather conditions.
  arXiv  Detail & Related papers  (2024-12-29T20:27:12Z)
• UltraPixel: Advancing Ultra-High-Resolution Image Synthesis to New Peaks [36.61645124563195]
  We present UltraPixel, a novel architecture utilizing cascade diffusion models to generate high-quality images at multiple resolutions. We use semantics-rich representations of lower-resolution images in the later denoising stage to guide the whole generation of highly detailed high-resolution images. Our model achieves fast training with reduced data requirements, producing photo-realistic high-resolution images.
  arXiv  Detail & Related papers  (2024-07-02T11:02:19Z)
• MDeRainNet: An Efficient Neural Network for Rain Streak Removal from Macro-pixel Images [44.83349966064718]
  We propose an efficient network, called MDeRainNet, for rain streak removal from LF images. The proposed network adopts a multi-scale encoder-decoder architecture, which directly works on Macro-pixel images (MPIs) to improve the rain removal performance. To improve the performance of our network on real-world rainy scenes, we propose a novel semi-supervised learning framework for our MDeRainNet.
  arXiv  Detail & Related papers  (2024-06-15T14:47:02Z)
• SynFog: A Photo-realistic Synthetic Fog Dataset based on End-to-end Imaging Simulation for Advancing Real-World Defogging in Autonomous Driving [48.27575423606407]
  We introduce an end-to-end simulation pipeline designed to generate photo-realistic foggy images. We present a new synthetic fog dataset named SynFog, which features both sky light and active lighting conditions. Experimental results demonstrate that models trained on SynFog exhibit superior performance in visual perception and detection accuracy.
  arXiv  Detail & Related papers  (2024-03-25T18:32:41Z)
• Exposure Bracketing Is All You Need For A High-Quality Image [50.822601495422916]
  Multi-exposure images are complementary in denoising, deblurring, high dynamic range imaging, and super-resolution.<n>We propose to utilize exposure bracketing photography to get a high-quality image by combining these tasks in this work.<n>In particular, a temporally modulated recurrent network (TMRNet) and self-supervised adaptation method are proposed.
  arXiv  Detail & Related papers  (2024-01-01T14:14:35Z)
• RSDiff: Remote Sensing Image Generation from Text Using Diffusion Model [0.8747606955991705]
  This research introduces a two-stage diffusion model methodology for synthesizing high-resolution satellite images from textual prompts. The pipeline comprises a Low-Resolution Diffusion Model (LRDM) that generates initial images based on text inputs and a Super-Resolution Diffusion Model (SRDM) that refines these images into high-resolution outputs.
  arXiv  Detail & Related papers  (2023-09-03T09:34:49Z)
• Contrastive Learning Based Recursive Dynamic Multi-Scale Network for Image Deraining [47.764883957379745]
  Rain streaks significantly decrease the visibility of captured images. Existing deep learning-based image deraining methods employ manually crafted networks and learn a straightforward projection from rainy images to clear images. We propose a contrastive learning-based image deraining method that investigates the correlation between rainy and clear images.
  arXiv  Detail & Related papers  (2023-05-29T13:51:41Z)
• Dual Degradation Representation for Joint Deraining and Low-Light Enhancement in the Dark [57.85378202032541]
  Rain in the dark poses a significant challenge to deploying real-world applications such as autonomous driving, surveillance systems, and night photography. Existing low-light enhancement or deraining methods struggle to brighten low-light conditions and remove rain simultaneously. We introduce an end-to-end model called L$2$RIRNet, designed to manage both low-light enhancement and deraining in real-world settings.
  arXiv  Detail & Related papers  (2023-05-06T10:17:42Z)
• TensoIR: Tensorial Inverse Rendering [51.57268311847087]
  TensoIR is a novel inverse rendering approach based on tensor factorization and neural fields. TensoRF is a state-of-the-art approach for radiance field modeling.
  arXiv  Detail & Related papers  (2023-04-24T21:39:13Z)
• Learning to Simulate Realistic LiDARs [66.7519667383175]
  We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
  arXiv  Detail & Related papers  (2022-09-22T13:12:54Z)
• Toward Real-world Single Image Deraining: A New Benchmark and Beyond [79.5893880599847]
  Single image deraining (SID) in real scenarios attracts increasing attention in recent years. Previous real datasets suffer from low-resolution images, homogeneous rain streaks, limited background variation, and even misalignment of image pairs. We establish a new high-quality dataset named RealRain-1k, consisting of $1,120$ high-resolution paired clean and rainy images with low- and high-density rain streaks, respectively.
  arXiv  Detail & Related papers  (2022-06-11T12:26:59Z)
• Lidar Light Scattering Augmentation (LISA): Physics-based Simulation of Adverse Weather Conditions for 3D Object Detection [60.89616629421904]
  Lidar-based object detectors are critical parts of the 3D perception pipeline in autonomous navigation systems such as self-driving cars. They are sensitive to adverse weather conditions such as rain, snow and fog due to reduced signal-to-noise ratio (SNR) and signal-to-background ratio (SBR)
  arXiv  Detail & Related papers  (2021-07-14T21:10:47Z)
• From Rain Generation to Rain Removal [67.71728610434698]
  We build a full Bayesian generative model for rainy image where the rain layer is parameterized as a generator. We employ the variational inference framework to approximate the expected statistical distribution of rainy image. Comprehensive experiments substantiate that the proposed model can faithfully extract the complex rain distribution.
  arXiv  Detail & Related papers  (2020-08-08T18:56:51Z)
• Rainy screens: Collecting rainy datasets, indoors [19.71705192452036]
  We present a simple method for generating diverse rainy images from existing clear ground-truth images. This setup allows us to leverage the diversity of existing datasets with auxiliary task ground-truth data. We generate rainy images with real adherent droplets and rain streaks based on Cityscapes and BDD, and train a de-raining model.
  arXiv  Detail & Related papers  (2020-03-10T13:57:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.