Event-based Synthetic Aperture Imaging with a Hybrid Network

• URL: http://arxiv.org/abs/2103.02376v2
• Date: Thu, 4 Mar 2021 16:39:37 GMT
• Title: Event-based Synthetic Aperture Imaging with a Hybrid Network
• Authors: Xiang Zhang, Liao Wei, Lei Yu, Wen Yang and Gui-Song Xia
• Abstract summary: We propose a novel SAI system based on the event camera which can produce asynchronous events with extremely low latency and high dynamic range. To reconstruct the occluded targets, we propose a hybrid encoder-decoder network composed of spiking neural networks (SNNs) and convolutional neural networks (CNNs)
• Score: 30.178111153441666
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Synthetic aperture imaging (SAI) is able to achieve the see through effect by blurring out the off-focus foreground occlusions and reconstructing the in-focus occluded targets from multi-view images. However, very dense occlusions and extreme lighting conditions may bring significant disturbances to SAI based on conventional frame-based cameras, leading to performance degeneration. To address these problems, we propose a novel SAI system based on the event camera which can produce asynchronous events with extremely low latency and high dynamic range. Thus, it can eliminate the interference of dense occlusions by measuring with almost continuous views, and simultaneously tackle the over/under exposure problems. To reconstruct the occluded targets, we propose a hybrid encoder-decoder network composed of spiking neural networks (SNNs) and convolutional neural networks (CNNs). In the hybrid network, the spatio-temporal information of the collected events is first encoded by SNN layers, and then transformed to the visual image of the occluded targets by a style-transfer CNN decoder. Through experiments, the proposed method shows remarkable performance in dealing with very dense occlusions and extreme lighting conditions, and high quality visual images can be reconstructed using pure event data.

Related papers

• A Novel Spike Transformer Network for Depth Estimation from Event Cameras via Cross-modality Knowledge Distillation [3.355813093377501]
  Event cameras operate differently from traditional digital cameras, continuously capturing data and generating binary spikes that encode time, location, and light intensity. This necessitates the development of innovative, spike-aware algorithms tailored for event cameras. We propose a purely spike-driven spike transformer network for depth estimation from spiking camera data.
  arXiv  Detail & Related papers  (2024-04-26T11:32:53Z)
• TBSN: Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising [94.09442506816724]
  Blind-spot networks (BSN) have been prevalent network architectures in self-supervised image denoising (SSID) We present a transformer-based blind-spot network (TBSN) by analyzing and redesigning the transformer operators that meet the blind-spot requirement. For spatial self-attention, an elaborate mask is applied to the attention matrix to restrict its receptive field, thus mimicking the dilated convolution. For channel self-attention, we observe that it may leak the blind-spot information when the channel number is greater than spatial size in the deep layers of multi-scale architectures.
  arXiv  Detail & Related papers  (2024-04-11T15:39:10Z)
• LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models [54.93010869546011]
  We propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images. Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules. We observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance.
  arXiv  Detail & Related papers  (2023-12-02T04:31:51Z)
• Learning to See Through with Events [37.19232535463858]
  This paper presents an Event-based SAI (E-SAI) method by relying on asynchronous events with extremely low latency and high dynamic range. The collected events are first refocused by a Re-focus-Net module to align in-focus events while scattering out off-focus ones. A hybrid network composed of spiking neural networks (SNNs) and convolutional neural networks (CNNs) is proposed to encode the foreground-temporal information from the refocused events and reconstruct a visual image of the occluded scenes.
  arXiv  Detail & Related papers  (2022-12-05T12:51:22Z)
• INFWIDE: Image and Feature Space Wiener Deconvolution Network for Non-blind Image Deblurring in Low-Light Conditions [32.35378513394865]
  We propose a novel non-blind deblurring method dubbed image and feature space Wiener deconvolution network (INFWIDE) INFWIDE removes noise and hallucinates saturated regions in the image space and suppresses ringing artifacts in the feature space. Experiments on synthetic data and real data demonstrate the superior performance of the proposed approach.
  arXiv  Detail & Related papers  (2022-07-17T15:22:31Z)
• Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision [64.71260357476602]
  Event-based vision sensors encode local pixel-wise brightness changes in streams of events rather than image frames. Recent progress in object recognition from event-based sensors has come from conversions of deep neural networks. We propose a hybrid architecture for end-to-end training of deep neural networks for event-based pattern recognition and object detection.
  arXiv  Detail & Related papers  (2021-12-06T23:45:58Z)
• MEFNet: Multi-scale Event Fusion Network for Motion Deblurring [62.60878284671317]
  Traditional frame-based cameras inevitably suffer from motion blur due to long exposure times. As a kind of bio-inspired camera, the event camera records the intensity changes in an asynchronous way with high temporal resolution. In this paper, we rethink the event-based image deblurring problem and unfold it into an end-to-end two-stage image restoration network.
  arXiv  Detail & Related papers  (2021-11-30T23:18:35Z)
• Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction [51.072733683919246]
  We introduce Recurrent Asynchronous Multimodal (RAM) networks to handle asynchronous and irregular data from multiple sensors. Inspired by traditional RNNs, RAM networks maintain a hidden state that is updated asynchronously and can be queried at any time to generate a prediction. We show an improvement over state-of-the-art methods by up to 30% in terms of mean depth absolute error.
  arXiv  Detail & Related papers  (2021-02-18T13:24:35Z)
• Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy [0.0]
  Event cameras are novel vision sensors that sample, in an asynchronous fashion, brightness increments with low latency and high temporal resolution. We propose a novel, lightweight neural network for optical flow estimation that achieves high speed inference with only a minor drop in performance. Results across multiple datasets show that the performance of the proposed self-supervised approach is in line with the state-of-the-art.
  arXiv  Detail & Related papers  (2020-09-17T13:30:05Z)
• Event-based Asynchronous Sparse Convolutional Networks [54.094244806123235]
  Event cameras are bio-inspired sensors that respond to per-pixel brightness changes in the form of asynchronous and sparse "events" We present a general framework for converting models trained on synchronous image-like event representations into asynchronous models with identical output. We show both theoretically and experimentally that this drastically reduces the computational complexity and latency of high-capacity, synchronous neural networks.
  arXiv  Detail & Related papers  (2020-03-20T08:39:49Z)
• Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks [40.44712305614071]
  We present Spike-FlowNet, a deep hybrid neural network architecture integrating SNNs and ANNs for efficiently estimating optical flow from sparse event camera outputs. The network is end-to-end trained with self-supervised learning on Multi-Vehicle Stereo Event Camera (MVSEC) dataset.
  arXiv  Detail & Related papers  (2020-03-14T20:37:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.