E2VIDiff: Perceptual Events-to-Video Reconstruction using Diffusion Priors

• URL: http://arxiv.org/abs/2407.08231v1
• Date: Thu, 11 Jul 2024 07:10:58 GMT
• Title: E2VIDiff: Perceptual Events-to-Video Reconstruction using Diffusion Priors
• Authors: Jinxiu Liang, Bohan Yu, Yixin Yang, Yiming Han, Boxin Shi,
• Abstract summary: We introduce diffusion models to events-to-video reconstruction, achieving colorful, realistic, and perceptually superior video generation from achromatic events. Our approach can produce diverse, realistic frames with faithfulness to the given events.
• Score: 44.430588804079555
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Event cameras, mimicking the human retina, capture brightness changes with unparalleled temporal resolution and dynamic range. Integrating events into intensities poses a highly ill-posed challenge, marred by initial condition ambiguities. Traditional regression-based deep learning methods fall short in perceptual quality, offering deterministic and often unrealistic reconstructions. In this paper, we introduce diffusion models to events-to-video reconstruction, achieving colorful, realistic, and perceptually superior video generation from achromatic events. Powered by the image generation ability and knowledge of pretrained diffusion models, the proposed method can achieve a better trade-off between the perception and distortion of the reconstructed frame compared to previous solutions. Extensive experiments on benchmark datasets demonstrate that our approach can produce diverse, realistic frames with faithfulness to the given events.

Related papers

• E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning [53.63364311738552]
  Bio-inspired event cameras or dynamic vision sensors are capable of capturing per-pixel brightness changes (called event-streams) in high temporal resolution and high dynamic range. It calls for events-to-video (E2V) solutions which take event-streams as input and generate high quality video frames for intuitive visualization. We propose textbfE2HQV, a novel E2V paradigm designed to produce high-quality video frames from events.
  arXiv  Detail & Related papers  (2024-01-16T05:10:50Z)
• Diffusion Priors for Dynamic View Synthesis from Monocular Videos [59.42406064983643]
  Dynamic novel view synthesis aims to capture the temporal evolution of visual content within videos. We first finetune a pretrained RGB-D diffusion model on the video frames using a customization technique. We distill the knowledge from the finetuned model to a 4D representations encompassing both dynamic and static Neural Radiance Fields.
  arXiv  Detail & Related papers  (2024-01-10T23:26:41Z)
• EventAid: Benchmarking Event-aided Image/Video Enhancement Algorithms with Real-captured Hybrid Dataset [55.12137324648253]
  Event cameras are emerging imaging technology that offers advantages over conventional frame-based imaging sensors in dynamic range and sensing speed. This paper focuses on five event-aided image and video enhancement tasks.
  arXiv  Detail & Related papers  (2023-12-13T15:42:04Z)
• Revisiting Event-based Video Frame Interpolation [49.27404719898305]
  Dynamic vision sensors or event cameras provide rich complementary information for video frame. estimating optical flow from events is arguably more difficult than from RGB information. We propose a divide-and-conquer strategy in which event-based intermediate frame synthesis happens incrementally in multiple simplified stages.
  arXiv  Detail & Related papers  (2023-07-24T06:51:07Z)
• EVREAL: Towards a Comprehensive Benchmark and Analysis Suite for Event-based Video Reconstruction [16.432164340779266]
  Event cameras offer advantages over traditional frame-based cameras such as high dynamic range and minimal motion blur. Their output is not easily understandable by humans, making reconstruction of intensity images from event streams a fundamental task in event-based vision. Recent deep learning-based methods have shown promise in video reconstruction from events, but this problem is not completely solved yet.
  arXiv  Detail & Related papers  (2023-04-30T09:28:38Z)
• Event-based Non-Rigid Reconstruction from Contours [17.049602518532847]
  We propose a novel approach for reconstructing such deformations using measurements from event-based cameras. Under the assumption of a static background, where all events are generated by the motion, our approach estimates the deformation of objects from events generated at the object contour. It associates events to mesh faces on the contour and maximizes the alignment of the line of sight through the event pixel with the associated face.
  arXiv  Detail & Related papers  (2022-10-12T14:53:11Z)
• Learning Monocular Dense Depth from Events [53.078665310545745]
  Event cameras produce brightness changes in the form of a stream of asynchronous events instead of intensity frames. Recent learning-based approaches have been applied to event-based data, such as monocular depth prediction. We propose a recurrent architecture to solve this task and show significant improvement over standard feed-forward methods.
  arXiv  Detail & Related papers  (2020-10-16T12:36:23Z)
• Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy [0.0]
  Event cameras are novel vision sensors that sample, in an asynchronous fashion, brightness increments with low latency and high temporal resolution. We propose a novel, lightweight neural network for optical flow estimation that achieves high speed inference with only a minor drop in performance. Results across multiple datasets show that the performance of the proposed self-supervised approach is in line with the state-of-the-art.
  arXiv  Detail & Related papers  (2020-09-17T13:30:05Z)
• Learning Event-Based Motion Deblurring [39.16921854492941]
  Fast motion can be captured as events at high time rate for event-based cameras. We show how its optimization can be unfolded with a novel end-to-end deep architecture. The proposed approach achieves state-of-the-art reconstruction quality, and generalizes better to handling real-world motion blur.
  arXiv  Detail & Related papers  (2020-04-13T07:01:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.