SpikeDerain: Unveiling Clear Videos from Rainy Sequences Using Color Spike Streams

• URL: http://arxiv.org/abs/2503.20315v1
• Date: Wed, 26 Mar 2025 08:28:28 GMT
• Title: SpikeDerain: Unveiling Clear Videos from Rainy Sequences Using Color Spike Streams
• Authors: Hanwen Liang, Xian Zhong, Wenxuan Liu, Yajing Zheng, Wenxin Huang, Zhaofei Yu, Tiejun Huang,
• Abstract summary: Restoring clear frames from rainy videos presents a significant challenge due to the rapid motion of rain streaks.<n>Traditional frame-based visual sensors, which capture scene content synchronously, struggle to capture the fast-moving details of rain accurately.<n>We propose a Color Spike Stream Deraining Network (SpikeDerain), capable of reconstructing spike streams of dynamic scenes and accurately removing rain streaks.
• Score: 49.34425133546994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Restoring clear frames from rainy videos presents a significant challenge due to the rapid motion of rain streaks. Traditional frame-based visual sensors, which capture scene content synchronously, struggle to capture the fast-moving details of rain accurately. In recent years, neuromorphic sensors have introduced a new paradigm for dynamic scene perception, offering microsecond temporal resolution and high dynamic range. However, existing multimodal methods that fuse event streams with RGB images face difficulties in handling the complex spatiotemporal interference of raindrops in real scenes, primarily due to hardware synchronization errors and computational redundancy. In this paper, we propose a Color Spike Stream Deraining Network (SpikeDerain), capable of reconstructing spike streams of dynamic scenes and accurately removing rain streaks. To address the challenges of data scarcity in real continuous rainfall scenes, we design a physically interpretable rain streak synthesis model that generates parameterized continuous rain patterns based on arbitrary background images. Experimental results demonstrate that the network, trained with this synthetic data, remains highly robust even under extreme rainfall conditions. These findings highlight the effectiveness and robustness of our method across varying rainfall levels and datasets, setting new standards for video deraining tasks. The code will be released soon.

Related papers

• RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining [14.025870185802463]
  We present an improved SSMs-based video deraining network (RainMamba) with a novel Hilbert mechanism to better capture sequence-level local information. We also introduce a difference-guided dynamic contrastive locality learning strategy to enhance the patch-level self-similarity learning ability of the proposed network.
  arXiv  Detail & Related papers  (2024-07-31T17:48:22Z)
• NiteDR: Nighttime Image De-Raining with Cross-View Sensor Cooperative Learning for Dynamic Driving Scenes [49.92839157944134]
  In nighttime driving scenes, insufficient and uneven lighting shrouds the scenes in darkness, resulting degradation of image quality and visibility. We develop an image de-raining framework tailored for rainy nighttime driving scenes. It aims to remove rain artifacts, enrich scene representation, and restore useful information.
  arXiv  Detail & Related papers  (2024-02-28T09:02:33Z)
• EGVD: Event-Guided Video Deraining [57.59935209162314]
  We propose an end-to-end learning-based network to unlock the potential of the event camera for video deraining. We build a real-world dataset consisting of rainy videos and temporally synchronized event streams.
  arXiv  Detail & Related papers  (2023-09-29T13:47:53Z)
• Towards General and Fast Video Derain via Knowledge Distillation [10.614356931086267]
  We propose a Rain Review-based General video derain Network via knowledge distillation (named RRGNet) We design a frame grouping-based encoder-decoder network that makes full use of the temporal information of the video. To consolidate the network's ability to derain, we design a rain review module to play back data from old tasks for the current model.
  arXiv  Detail & Related papers  (2023-08-10T05:27:43Z)
• Dual Degradation Representation for Joint Deraining and Low-Light Enhancement in the Dark [57.85378202032541]
  Rain in the dark poses a significant challenge to deploying real-world applications such as autonomous driving, surveillance systems, and night photography. Existing low-light enhancement or deraining methods struggle to brighten low-light conditions and remove rain simultaneously. We introduce an end-to-end model called L$2$RIRNet, designed to manage both low-light enhancement and deraining in real-world settings.
  arXiv  Detail & Related papers  (2023-05-06T10:17:42Z)
• Not Just Streaks: Towards Ground Truth for Single Image Deraining [42.15398478201746]
  We propose a large-scale dataset of real-world rainy and clean image pairs. We propose a deep neural network that reconstructs the underlying scene by minimizing a rain-robust loss between rainy and clean images.
  arXiv  Detail & Related papers  (2022-06-22T00:10:06Z)
• Semi-MoreGAN: A New Semi-supervised Generative Adversarial Network for Mixture of Rain Removal [18.04268933542476]
  We propose a new SEMI-supervised Mixture Of rain REmoval Generative Adversarial Network (Semi-MoreGAN) Semi-MoreGAN consists of four key modules: (I) a novel attentional depth prediction network to provide precise depth estimation; (ii) a context feature prediction network composed of several well-designed detailed residual blocks to produce detailed image context features; (iii) a pyramid depth-guided non-local network to effectively integrate the image context with the depth information, and produce the final rain-free images; and (iv) a comprehensive semi-supervised loss function to make the model not limited
  arXiv  Detail & Related papers  (2022-04-28T11:35:26Z)
• RCDNet: An Interpretable Rain Convolutional Dictionary Network for Single Image Deraining [49.99207211126791]
  We specifically build a novel deep architecture, called rain convolutional dictionary network (RCDNet) RCDNet embeds the intrinsic priors of rain streaks and has clear interpretability. By end-to-end training such an interpretable network, all involved rain kernels and proximal operators can be automatically extracted.
  arXiv  Detail & Related papers  (2021-07-14T16:08:11Z)
• Semi-Supervised Video Deraining with Dynamic Rain Generator [59.71640025072209]
  This paper proposes a new semi-supervised video deraining method, in which a dynamic rain generator is employed to fit the rain layer. Specifically, such dynamic generator consists of one emission model and one transition model to simultaneously encode the spatially physical structure and temporally continuous changes of rain streaks. Various prior formats are designed for the labeled synthetic and unlabeled real data, so as to fully exploit the common knowledge underlying them.
  arXiv  Detail & Related papers  (2021-03-14T14:28:57Z)
• Structural Residual Learning for Single Image Rain Removal [48.87977695398587]
  This study proposes a new network architecture by enforcing the output residual of the network possess intrinsic rain structures. Such a structural residual setting guarantees the rain layer extracted by the network finely comply with the prior knowledge of general rain streaks.
  arXiv  Detail & Related papers  (2020-05-19T05:52:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.