Recurrent Spike-based Image Restoration under General Illumination

• URL: http://arxiv.org/abs/2308.03018v1
• Date: Sun, 6 Aug 2023 04:24:28 GMT
• Title: Recurrent Spike-based Image Restoration under General Illumination
• Authors: Lin Zhu, Yunlong Zheng, Mengyue Geng, Lizhi Wang, Hua Huang
• Abstract summary: Spike camera is a new type of bio-inspired vision sensor that records light intensity in the form of a spike array with high temporal resolution (20,000 Hz) Existing spike-based approaches typically assume that the scenes are with sufficient light intensity, which is usually unavailable in many real-world scenarios such as rainy days or dusk scenes. We propose a Recurrent Spike-based Image Restoration (RSIR) network, which is the first work towards restoring clear images from spike arrays under general illumination.
• Score: 21.630646894529065
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spike camera is a new type of bio-inspired vision sensor that records light intensity in the form of a spike array with high temporal resolution (20,000 Hz). This new paradigm of vision sensor offers significant advantages for many vision tasks such as high speed image reconstruction. However, existing spike-based approaches typically assume that the scenes are with sufficient light intensity, which is usually unavailable in many real-world scenarios such as rainy days or dusk scenes. To unlock more spike-based application scenarios, we propose a Recurrent Spike-based Image Restoration (RSIR) network, which is the first work towards restoring clear images from spike arrays under general illumination. Specifically, to accurately describe the noise distribution under different illuminations, we build a physical-based spike noise model according to the sampling process of the spike camera. Based on the noise model, we design our RSIR network which consists of an adaptive spike transformation module, a recurrent temporal feature fusion module, and a frequency-based spike denoising module. Our RSIR can process the spike array in a recursive manner to ensure that the spike temporal information is well utilized. In the training process, we generate the simulated spike data based on our noise model to train our network. Extensive experiments on real-world datasets with different illuminations demonstrate the effectiveness of the proposed network. The code and dataset are released at https://github.com/BIT-Vision/RSIR.

Related papers

• SwinSF: Image Reconstruction from Spatial-Temporal Spike Streams [2.609896297570564]
  We introduce Swin Spikeformer (SwinSF), a novel model for dynamic scene reconstruction from spike streams. SwinSF combines shifted window self-attention and proposed temporal spike attention, ensuring a comprehensive feature extraction. We build a new synthesized dataset for spike image reconstruction which matches the resolution of the latest spike camera.
  arXiv  Detail & Related papers  (2024-07-22T15:17:39Z)
• SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream [26.165424006344267]
  Spike cameras offer distinct advantages over standard cameras. Existing approaches reliant on spike cameras often assume optimal illumination. We introduce SpikeNeRF, the first work that derives a NeRF-based volumetric scene representation from spike camera data.
  arXiv  Detail & Related papers  (2024-03-17T13:51:25Z)
• Finding Visual Saliency in Continuous Spike Stream [23.591309376586835]
  In this paper, we investigate the visual saliency in the continuous spike stream for the first time. We propose a Recurrent Spiking Transformer framework, which is based on a full spiking neural network. Our framework exhibits a substantial margin of improvement in highlighting and capturing visual saliency in the spike stream.
  arXiv  Detail & Related papers  (2024-03-10T15:15:35Z)
• Learning to Robustly Reconstruct Low-light Dynamic Scenes from Spike Streams [28.258022350623023]
  As a neuromorphic sensor, spike camera can generate continuous binary spike streams to capture per-pixel light intensity. We propose a bidirectional recurrent-based reconstruction framework, including a Light-Robust Representation (LR-Rep) and a fusion module. We have developed a reconstruction benchmark for high-speed low-light scenes.
  arXiv  Detail & Related papers  (2024-01-19T03:01:07Z)
• LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models [54.93010869546011]
  We propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images. Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules. We observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance.
  arXiv  Detail & Related papers  (2023-12-02T04:31:51Z)
• Spike Stream Denoising via Spike Camera Simulation [64.11994763727631]
  We propose a systematic noise model for spike camera based on its unique circuit. The first benchmark for spike stream denoising is proposed which includes clear (noisy) spike stream. Experiments show that DnSS has promising performance on the proposed benchmark.
  arXiv  Detail & Related papers  (2023-04-06T14:59:48Z)
• SpikeCV: Open a Continuous Computer Vision Era [56.0388584615134]
  SpikeCV is a new open-source computer vision platform for the spike camera. The spike camera is a neuromorphic visual sensor that has developed rapidly in recent years. SpikeCV provides a variety of ultra-high-speed scene datasets, hardware interfaces, and an easy-to-use modules library.
  arXiv  Detail & Related papers  (2023-03-21T09:00:12Z)
• Seeing Through The Noisy Dark: Toward Real-world Low-Light Image Enhancement and Denoising [125.56062454927755]
  Real-world low-light environment usually suffer from lower visibility and heavier noise, due to insufficient light or hardware limitation. We propose a novel end-to-end method termed Real-world Low-light Enhancement & Denoising Network (RLED-Net)
  arXiv  Detail & Related papers  (2022-10-02T14:57:23Z)
• Wavelet-Based Network For High Dynamic Range Imaging [64.66969585951207]
  Existing methods, such as optical flow based and end-to-end deep learning based solutions, are error-prone either in detail restoration or ghosting artifacts removal. In this work, we propose a novel frequency-guided end-to-end deep neural network (FNet) to conduct HDR fusion in the frequency domain, and Wavelet Transform (DWT) is used to decompose inputs into different frequency bands. The low-frequency signals are used to avoid specific ghosting artifacts, while the high-frequency signals are used for preserving details.
  arXiv  Detail & Related papers  (2021-08-03T12:26:33Z)
• EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning [75.17497166510083]
  Event cameras sense intensity changes and have many advantages over conventional cameras. Some methods have been proposed to reconstruct intensity images from event streams. The outputs are still in low resolution (LR), noisy, and unrealistic. We propose a novel end-to-end pipeline that reconstructs LR images from event streams, enhances the image qualities and upsamples the enhanced images, called EventSR.
  arXiv  Detail & Related papers  (2020-03-17T10:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.