Rethinking High-speed Image Reconstruction Framework with Spike Camera

• URL: http://arxiv.org/abs/2501.04477v1
• Date: Wed, 08 Jan 2025 13:00:17 GMT
• Title: Rethinking High-speed Image Reconstruction Framework with Spike Camera
• Authors: Kang Chen, Yajing Zheng, Tiejun Huang, Zhaofei Yu,
• Abstract summary: Spike cameras generate continuous spike streams to capture high-speed scenes with lower bandwidth and higher dynamic range than traditional RGB cameras. We introduce a novel spike-to-image reconstruction framework SpikeCLIP that goes beyond traditional training paradigms. Our experiments on real-world low-light datasets demonstrate that SpikeCLIP significantly enhances texture details and the luminance balance of recovered images.
• Score: 48.627095354244204
• License:
• Abstract: Spike cameras, as innovative neuromorphic devices, generate continuous spike streams to capture high-speed scenes with lower bandwidth and higher dynamic range than traditional RGB cameras. However, reconstructing high-quality images from the spike input under low-light conditions remains challenging. Conventional learning-based methods often rely on the synthetic dataset as the supervision for training. Still, these approaches falter when dealing with noisy spikes fired under the low-light environment, leading to further performance degradation in the real-world dataset. This phenomenon is primarily due to inadequate noise modelling and the domain gap between synthetic and real datasets, resulting in recovered images with unclear textures, excessive noise, and diminished brightness. To address these challenges, we introduce a novel spike-to-image reconstruction framework SpikeCLIP that goes beyond traditional training paradigms. Leveraging the CLIP model's powerful capability to align text and images, we incorporate the textual description of the captured scene and unpaired high-quality datasets as the supervision. Our experiments on real-world low-light datasets U-CALTECH and U-CIFAR demonstrate that SpikeCLIP significantly enhances texture details and the luminance balance of recovered images. Furthermore, the reconstructed images are well-aligned with the broader visual features needed for downstream tasks, ensuring more robust and versatile performance in challenging environments.

Related papers

• SwinSF: Image Reconstruction from Spatial-Temporal Spike Streams [2.609896297570564]
  We introduce Swin Spikeformer (SwinSF), a novel model for dynamic scene reconstruction from spike streams. SwinSF combines shifted window self-attention and proposed temporal spike attention, ensuring a comprehensive feature extraction. We build a new synthesized dataset for spike image reconstruction which matches the resolution of the latest spike camera.
  arXiv  Detail & Related papers  (2024-07-22T15:17:39Z)
• CodeEnhance: A Codebook-Driven Approach for Low-Light Image Enhancement [97.95330185793358]
  Low-light image enhancement (LLIE) aims to improve low-illumination images. Existing methods face two challenges: uncertainty in restoration from diverse brightness degradations and loss of texture and color information. We propose a novel enhancement approach, CodeEnhance, by leveraging quantized priors and image refinement.
  arXiv  Detail & Related papers  (2024-04-08T07:34:39Z)
• SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream [26.165424006344267]
  Spike cameras offer distinct advantages over standard cameras. Existing approaches reliant on spike cameras often assume optimal illumination. We introduce SpikeNeRF, the first work that derives a NeRF-based volumetric scene representation from spike camera data.
  arXiv  Detail & Related papers  (2024-03-17T13:51:25Z)
• SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams [44.02794438687478]
  Spike cameras have proven effective in capturing motion features and beneficial for solving this ill-posed problem. Existing methods fall into the supervised learning paradigm, which suffers from notable performance degradation when applied to real-world scenarios. We propose the first self-supervised framework for the task of spike-guided motion deblurring.
  arXiv  Detail & Related papers  (2024-03-14T15:29:09Z)
• LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models [54.93010869546011]
  We propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images. Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules. We observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance.
  arXiv  Detail & Related papers  (2023-12-02T04:31:51Z)
• SCRNet: a Retinex Structure-based Low-light Enhancement Model Guided by Spatial Consistency [22.54951703413469]
  We present a novel low-light image enhancement model, termed Spatial Consistency Retinex Network (SCRNet) Our proposed model incorporates three levels of consistency: channel level, semantic level, and texture level, inspired by the principle of spatial consistency. Extensive evaluations on various low-light image datasets demonstrate that our proposed SCRNet outshines existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-14T03:32:19Z)
• Seeing Through The Noisy Dark: Toward Real-world Low-Light Image Enhancement and Denoising [125.56062454927755]
  Real-world low-light environment usually suffer from lower visibility and heavier noise, due to insufficient light or hardware limitation. We propose a novel end-to-end method termed Real-world Low-light Enhancement & Denoising Network (RLED-Net)
  arXiv  Detail & Related papers  (2022-10-02T14:57:23Z)
• INFWIDE: Image and Feature Space Wiener Deconvolution Network for Non-blind Image Deblurring in Low-Light Conditions [32.35378513394865]
  We propose a novel non-blind deblurring method dubbed image and feature space Wiener deconvolution network (INFWIDE) INFWIDE removes noise and hallucinates saturated regions in the image space and suppresses ringing artifacts in the feature space. Experiments on synthetic data and real data demonstrate the superior performance of the proposed approach.
  arXiv  Detail & Related papers  (2022-07-17T15:22:31Z)
• Enhancing Low-Light Images in Real World via Cross-Image Disentanglement [58.754943762945864]
  We propose a new low-light image enhancement dataset consisting of misaligned training images with real-world corruptions. Our model achieves state-of-the-art performances on both the newly proposed dataset and other popular low-light datasets.
  arXiv  Detail & Related papers  (2022-01-10T03:12:52Z)
• Unsupervised Low-light Image Enhancement with Decoupled Networks [103.74355338972123]
  We learn a two-stage GAN-based framework to enhance the real-world low-light images in a fully unsupervised fashion. Our proposed method outperforms the state-of-the-art unsupervised image enhancement methods in terms of both illumination enhancement and noise reduction.
  arXiv  Detail & Related papers  (2020-05-06T13:37:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.