RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models

• URL: http://arxiv.org/abs/2608.02953v2
• Date: Mon, 10 Aug 2026 09:30:05 GMT
• Title: RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models
• Abstract summary: RealWeather is a driving world model for both realistic and scene-faithful weather translation.<n>Our key idea is to learn authentic weather dynamics directly from real-world videos.<n>Experiments demonstrate that RealWeather significantly outperforms existing methods in visual realism and structural preservation.
• Score: 42.84298567152532
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Realistic weather translation is valuable for developing and evaluating autonomous driving systems, yet collecting paired videos of the same scenes under different weather conditions at scale is impractical. Existing methods therefore rely on synthetic data, 3D weather editing, or geometry-conditioned generation, often compromising weather realism or scene fidelity. We propose RealWeather, a driving world model for both realistic and scene-faithful weather translation. Our key idea is to learn authentic weather dynamics directly from real-world videos. Specifically, RealWeather employs Progressive Realism Bootstrapping, an iterative data-refinement strategy. Assisted by an auxiliary Pseudo-Clear Generation pipeline, training initially starts with pseudo-style conditioning videos. As training proceeds, these inputs are progressively replaced with increasingly realistic videos generated by the model itself. This strategy bridges the pseudo-to-real domain gap, allowing the model to adapt seamlessly to real-world input distributions and naturally support bidirectional clear adverse translation. Furthermore, to strictly enforce structural integrity and suppress hallucinations, we introduce Scene-Fidelity RL Optimization, a reward-driven policy optimization strategy that explicitly penalizes alterations to safety-critical driving elements. Extensive experiments demonstrate that RealWeather significantly outperforms existing methods in visual realism and structural preservation, while enabling robust long-tail weather scenario generation and strong zero-shot out-of-distribution generalization. Our video demos can be found at https://hust-umi.github.io/RealWeather/.

Related papers

• Cyclone: Diffusion Model for Cycle-Consistent Weather Editing from Unpaired Driving Data [6.193129698981512]
  We present Cyclone, a unified framework for weather editing based on latent diffusion.<n>We show that our approach produces more realistic, structure-preserving outputs than existing baselines.
  arXiv  Detail & Related papers  (2026-07-15T15:08:45Z)
• Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis [81.61856788730198]
  Weather synthesis aims to add weather effects to input videos while preserving scene identity, structure, and motion.<n>Most approaches rely on text prompts, which are inherently underspecified and often fail to produce detailed weather characteristics.<n>We propose a Semantic-Aware, Physics-Informed, and Geometry-Grounded framework that steers an off-the-shelf video editor to synthesize diverse global appearances and detailed particle dynamics.
  arXiv  Detail & Related papers  (2026-06-27T17:38:21Z)
• DiffusionHarmonizer: Bridging Neural Reconstruction and Photorealistic Simulation with Online Diffusion Enhancer [62.18680935878919]
  We introduce DiffusionHarmonizer, an online generative enhancement framework that transforms renderings into temporally consistent outputs.<n>At its core is a single-step temporally-conditioned enhancer capable of running in online simulators on a single GPU.
  arXiv  Detail & Related papers  (2026-02-27T15:35:30Z)
• ClimaOoD: Improving Anomaly Segmentation via Physically Realistic Synthetic Data [16.145130650604344]
  We present a semantics-guided image-to-image framework for synthesizing semantically coherent, weather-diverse, and physically plausible OoD driving data.<n>Based on this framework, we construct ClimaOoD, a large-scale benchmark spanning six representative driving scenarios under both clear and adverse weather conditions.
  arXiv  Detail & Related papers  (2025-12-02T12:14:19Z)
• Real-World Adverse Weather Image Restoration via Dual-Level Reinforcement Learning with High-Quality Cold Start [12.423595039024597]
  Adverse weather severely impairs real-world visual perception, while existing vision models struggle to generalize to complex degradations.<n>We first construct HFLS-Weather, a physics-driven dataset that simulates diverse weather phenomena, and then design a dual-level reinforcement learning framework with HFLS-Weather for cold-start training.<n>This framework enables continuous adaptation to real-world conditions and achieves state-of-the-art performance across a wide range of adverse weather scenarios.
  arXiv  Detail & Related papers  (2025-11-07T09:22:53Z)
• Semi-Supervised State-Space Model with Dynamic Stacking Filter for Real-World Video Deraining [73.5575992346396]
  We propose a dual-branch-temporal state-space model to enhance rain streak removal in video sequences.<n>To improve multi-frame feature fusion, we derive a dynamic filter stacking, which adaptively approximates statistical filters for pixel-wise feature refinement.<n>To further explore the capacity of deraining models in supporting other vision-based tasks in rainy environments, we introduce a novel real-world benchmark.
  arXiv  Detail & Related papers  (2025-05-22T15:50:00Z)
• Controllable Weather Synthesis and Removal with Video Diffusion Models [61.56193902622901]
  WeatherWeaver is a video diffusion model that synthesizes diverse weather effects directly into any input video.<n>Our model provides precise control over weather effect intensity and supports blending various weather types, ensuring both realism and adaptability.
  arXiv  Detail & Related papers  (2025-05-01T17:59:57Z)
• Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal [53.15046196592023]
  We introduce test-time adaptation into adverse weather removal in videos. We propose the first framework that integrates test-time adaptation into the iterative diffusion reverse process.
  arXiv  Detail & Related papers  (2024-03-12T14:21:30Z)
• Semi-Supervised Video Deraining with Dynamic Rain Generator [59.71640025072209]
  This paper proposes a new semi-supervised video deraining method, in which a dynamic rain generator is employed to fit the rain layer. Specifically, such dynamic generator consists of one emission model and one transition model to simultaneously encode the spatially physical structure and temporally continuous changes of rain streaks. Various prior formats are designed for the labeled synthetic and unlabeled real data, so as to fully exploit the common knowledge underlying them.
  arXiv  Detail & Related papers  (2021-03-14T14:28:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.