E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations

• URL: http://arxiv.org/abs/2206.07578v1
• Date: Wed, 15 Jun 2022 15:05:10 GMT
• Title: E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations
• Authors: Jongwan Kim, DongJin Lee, Byunggook Na, Seongsik Park, Jeonghee Jo, Sungroh Yoon
• Abstract summary: Event cameras respond to brightness changes in the scene asynchronously and independently for every pixel. E2V-SDE can rapidly reconstruct images at arbitrary time steps and make realistic predictions on unseen data. In terms of image quality, the LPIPS score improves by up to 12% and the reconstruction speed is 87% higher than that of ET-Net.
• Score: 23.866475611205736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Event cameras respond to brightness changes in the scene asynchronously and independently for every pixel. Due to the properties, these cameras have distinct features: high dynamic range (HDR), high temporal resolution, and low power consumption. However, the results of event cameras should be processed into an alternative representation for computer vision tasks. Also, they are usually noisy and cause poor performance in areas with few events. In recent years, numerous researchers have attempted to reconstruct videos from events. However, they do not provide good quality videos due to a lack of temporal information from irregular and discontinuous data. To overcome these difficulties, we introduce an E2V-SDE whose dynamics are governed in a latent space by Stochastic differential equations (SDE). Therefore, E2V-SDE can rapidly reconstruct images at arbitrary time steps and make realistic predictions on unseen data. In addition, we successfully adopted a variety of image composition techniques for improving image clarity and temporal consistency. By conducting extensive experiments on simulated and real-scene datasets, we verify that our model outperforms state-of-the-art approaches under various video reconstruction settings. In terms of image quality, the LPIPS score improves by up to 12% and the reconstruction speed is 87% higher than that of ET-Net.

Related papers

• LaSe-E2V: Towards Language-guided Semantic-Aware Event-to-Video Reconstruction [8.163356555241322]
  We propose a novel framework, called LaSe-E2V, that can achieve semantic-aware high-quality E2V reconstruction. We first propose an Event-guided Spatiotemporal Attention (ESA) module to condition the event data to the denoising pipeline effectively. We then introduce an event-aware mask loss to ensure temporal coherence and a noise strategy to enhance spatial consistency.
  arXiv  Detail & Related papers  (2024-07-08T01:40:32Z)
• E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning [53.63364311738552]
  Bio-inspired event cameras or dynamic vision sensors are capable of capturing per-pixel brightness changes (called event-streams) in high temporal resolution and high dynamic range. It calls for events-to-video (E2V) solutions which take event-streams as input and generate high quality video frames for intuitive visualization. We propose textbfE2HQV, a novel E2V paradigm designed to produce high-quality video frames from events.
  arXiv  Detail & Related papers  (2024-01-16T05:10:50Z)
• EventAid: Benchmarking Event-aided Image/Video Enhancement Algorithms with Real-captured Hybrid Dataset [55.12137324648253]
  Event cameras are emerging imaging technology that offers advantages over conventional frame-based imaging sensors in dynamic range and sensing speed. This paper focuses on five event-aided image and video enhancement tasks.
  arXiv  Detail & Related papers  (2023-12-13T15:42:04Z)
• Event-based Continuous Color Video Decompression from Single Frames [38.59798259847563]
  We present ContinuityCam, a novel approach to generate a continuous video from a single static RGB image, using an event camera. Our approach combines continuous long-range motion modeling with a feature-plane-based neural integration model, enabling frame prediction at arbitrary times within the events.
  arXiv  Detail & Related papers  (2023-11-30T18:59:23Z)
• Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion [67.15935067326662]
  Event cameras offer low power, low latency, high temporal resolution and high dynamic range. NeRF is seen as the leading candidate for efficient and effective scene representation. We propose Robust e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving event cameras.
  arXiv  Detail & Related papers  (2023-09-15T17:52:08Z)
• An Asynchronous Linear Filter Architecture for Hybrid Event-Frame Cameras [9.69495347826584]
  We present an asynchronous linear filter architecture, fusing event and frame camera data, for HDR video reconstruction and spatial convolution. The proposed AKF pipeline outperforms other state-of-the-art methods in both absolute intensity error (69.4% reduction) and image similarity indexes (average 35.5% improvement)
  arXiv  Detail & Related papers  (2023-09-03T12:37:59Z)
• Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography [54.36608424943729]
  We show that in a ''long-burst'', forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. We devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion.
  arXiv  Detail & Related papers  (2022-12-22T18:54:34Z)
• An Asynchronous Kalman Filter for Hybrid Event Cameras [13.600773150848543]
  Event cameras are ideally suited to capture HDR visual information without blur. conventional image sensors measure absolute intensity of slowly changing scenes effectively but do poorly on high dynamic range or quickly changing scenes. We present an event-based video reconstruction pipeline for High Dynamic Range scenarios.
  arXiv  Detail & Related papers  (2020-12-10T11:24:07Z)
• Reducing the Sim-to-Real Gap for Event Cameras [64.89183456212069]
  Event cameras are paradigm-shifting novel sensors that report asynchronous, per-pixel brightness changes called 'events' with unparalleled low latency. Recent work has demonstrated impressive results using Convolutional Neural Networks (CNNs) for video reconstruction and optic flow with events. We present strategies for improving training data for event based CNNs that result in 20-40% boost in performance of existing video reconstruction networks.
  arXiv  Detail & Related papers  (2020-03-20T02:44:29Z)
• EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning [75.17497166510083]
  Event cameras sense intensity changes and have many advantages over conventional cameras. Some methods have been proposed to reconstruct intensity images from event streams. The outputs are still in low resolution (LR), noisy, and unrealistic. We propose a novel end-to-end pipeline that reconstructs LR images from event streams, enhances the image qualities and upsamples the enhanced images, called EventSR.
  arXiv  Detail & Related papers  (2020-03-17T10:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.