Unveiling the Potential of Spike Streams for Foreground Occlusion Removal from Densely Continuous Views

• URL: http://arxiv.org/abs/2307.00821v1
• Date: Mon, 3 Jul 2023 08:01:43 GMT
• Title: Unveiling the Potential of Spike Streams for Foreground Occlusion Removal from Densely Continuous Views
• Authors: Jiyuan Zhang, Shiyan Chen, Yajing Zheng, Zhaofei Yu, Tiejun Huang
• Abstract summary: We propose an innovative solution for tackling the de-occlusion problem through continuous multi-view imaging using only one spike camera. By rapidly moving the spike camera, we continually capture the dense stream of spikes from the occluded scene. To process the spikes, we build a novel model textbfSpkOccNet, in which we integrate information of spikes from continuous viewpoints.
• Score: 23.10251947174782
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The extraction of a clean background image by removing foreground occlusion holds immense practical significance, but it also presents several challenges. Presently, the majority of de-occlusion research focuses on addressing this issue through the extraction and synthesis of discrete images from calibrated camera arrays. Nonetheless, the restoration quality tends to suffer when faced with dense occlusions or high-speed motions due to limited perspectives and motion blur. To successfully remove dense foreground occlusion, an effective multi-view visual information integration approach is required. Introducing the spike camera as a novel type of neuromorphic sensor offers promising capabilities with its ultra-high temporal resolution and high dynamic range. In this paper, we propose an innovative solution for tackling the de-occlusion problem through continuous multi-view imaging using only one spike camera without any prior knowledge of camera intrinsic parameters and camera poses. By rapidly moving the spike camera, we continually capture the dense stream of spikes from the occluded scene. To process the spikes, we build a novel model \textbf{SpkOccNet}, in which we integrate information of spikes from continuous viewpoints within multi-windows, and propose a novel cross-view mutual attention mechanism for effective fusion and refinement. In addition, we contribute the first real-world spike-based dataset \textbf{S-OCC} for occlusion removal. The experimental results demonstrate that our proposed model efficiently removes dense occlusions in diverse scenes while exhibiting strong generalization.

Related papers

• Deep Generative Adversarial Network for Occlusion Removal from a Single Image [3.5639148953570845]
  We propose a fully automatic, two-stage convolutional neural network for fence segmentation and occlusion completion. We leverage generative adversarial networks (GANs) to synthesize realistic content, including both structure and texture, in a single shot for inpainting.
  arXiv  Detail & Related papers  (2024-09-20T06:00:45Z)
• SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream [26.165424006344267]
  Spike cameras offer distinct advantages over standard cameras. Existing approaches reliant on spike cameras often assume optimal illumination. We introduce SpikeNeRF, the first work that derives a NeRF-based volumetric scene representation from spike camera data.
  arXiv  Detail & Related papers  (2024-03-17T13:51:25Z)
• SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams [44.02794438687478]
  Spike cameras have proven effective in capturing motion features and beneficial for solving this ill-posed problem. Existing methods fall into the supervised learning paradigm, which suffers from notable performance degradation when applied to real-world scenarios. We propose the first self-supervised framework for the task of spike-guided motion deblurring.
  arXiv  Detail & Related papers  (2024-03-14T15:29:09Z)
• Finding Visual Saliency in Continuous Spike Stream [23.591309376586835]
  In this paper, we investigate the visual saliency in the continuous spike stream for the first time. We propose a Recurrent Spiking Transformer framework, which is based on a full spiking neural network. Our framework exhibits a substantial margin of improvement in highlighting and capturing visual saliency in the spike stream.
  arXiv  Detail & Related papers  (2024-03-10T15:15:35Z)
• LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry [52.131996528655094]
  We present the Long-term Effective Any Point Tracking (LEAP) module. LEAP innovatively combines visual, inter-track, and temporal cues with mindfully selected anchors for dynamic track estimation. Based on these traits, we develop LEAP-VO, a robust visual odometry system adept at handling occlusions and dynamic scenes.
  arXiv  Detail & Related papers  (2024-01-03T18:57:27Z)
• Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images [65.41966114373373]
  We present an improved solution to the neural image-based rendering problem in computer vision. The proposed approach could synthesize a realistic image of the scene from a novel viewpoint at test time.
  arXiv  Detail & Related papers  (2023-11-08T08:18:23Z)
• Spike Stream Denoising via Spike Camera Simulation [64.11994763727631]
  We propose a systematic noise model for spike camera based on its unique circuit. The first benchmark for spike stream denoising is proposed which includes clear (noisy) spike stream. Experiments show that DnSS has promising performance on the proposed benchmark.
  arXiv  Detail & Related papers  (2023-04-06T14:59:48Z)
• Locality-aware Channel-wise Dropout for Occluded Face Recognition [116.2355331029041]
  Face recognition is a challenging task in unconstrained scenarios, especially when faces are partially occluded. We propose a novel and elegant occlusion-simulation method via dropping the activations of a group of neurons in some elaborately selected channel. Experiments on various benchmarks show that the proposed method outperforms state-of-the-art methods with a remarkable improvement.
  arXiv  Detail & Related papers  (2021-07-20T05:53:14Z)
• DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras [63.186486240525554]
  DeepMultiCap is a novel method for multi-person performance capture using sparse multi-view cameras. Our method can capture time varying surface details without the need of using pre-scanned template models.
  arXiv  Detail & Related papers  (2021-05-01T14:32:13Z)
• Aliasing is your Ally: End-to-End Super-Resolution from Raw Image Bursts [70.80220990106467]
  This presentation addresses the problem of reconstructing a high-resolution image from multiple lower-resolution snapshots captured from slightly different viewpoints in space and time. Key challenges for solving this problem include (i) aligning the input pictures with sub-pixel accuracy, (ii) handling raw (noisy) images for maximal faithfulness to native camera data, and (iii) designing/learning an image prior (regularizer) well suited to the task.
  arXiv  Detail & Related papers  (2021-04-13T13:39:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.