EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning

• URL: http://arxiv.org/abs/2003.07640v1
• Date: Tue, 17 Mar 2020 10:58:10 GMT
• Title: EventSR: From Asynchronous Events to Image Reconstruction, Restoration, and Super-Resolution via End-to-End Adversarial Learning
• Authors: Lin Wang, Tae-Kyun Kim, Kuk-Jin Yoon
• Abstract summary: Event cameras sense intensity changes and have many advantages over conventional cameras. Some methods have been proposed to reconstruct intensity images from event streams. The outputs are still in low resolution (LR), noisy, and unrealistic. We propose a novel end-to-end pipeline that reconstructs LR images from event streams, enhances the image qualities and upsamples the enhanced images, called EventSR.
• Score: 75.17497166510083
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Event cameras sense intensity changes and have many advantages over conventional cameras. To take advantage of event cameras, some methods have been proposed to reconstruct intensity images from event streams. However, the outputs are still in low resolution (LR), noisy, and unrealistic. The low-quality outputs stem broader applications of event cameras, where high spatial resolution (HR) is needed as well as high temporal resolution, dynamic range, and no motion blur. We consider the problem of reconstructing and super-resolving intensity images from LR events, when no ground truth (GT) HR images and down-sampling kernels are available. To tackle the challenges, we propose a novel end-to-end pipeline that reconstructs LR images from event streams, enhances the image qualities and upsamples the enhanced images, called EventSR. For the absence of real GT images, our method is primarily unsupervised, deploying adversarial learning. To train EventSR, we create an open dataset including both real-world and simulated scenes. The use of both datasets boosts up the network performance, and the network architectures and various loss functions in each phase help improve the image qualities. The whole pipeline is trained in three phases. While each phase is mainly for one of the three tasks, the networks in earlier phases are fine-tuned by respective loss functions in an end-to-end manner. Experimental results show that EventSR reconstructs high-quality SR images from events for both simulated and real-world data.

Related papers

• E$^3$NeRF: Efficient Event-Enhanced Neural Radiance Fields from Blurry Images [25.304680391243537]
  We propose a novel Efficient Event-Enhanced NeRF (E$3$NeRF) We leverage spatial-temporal information from the event stream to evenly distribute learning attention over temporal blur. Experiments on both synthetic data and real-world data demonstrate that E$3$NeRF can effectively learn a sharp NeRF from blurry images.
  arXiv  Detail & Related papers  (2024-08-03T18:47:31Z)
• Learning Many-to-Many Mapping for Unpaired Real-World Image Super-resolution and Downscaling [60.80788144261183]
  We propose an image downscaling and SR model dubbed as SDFlow, which simultaneously learns a bidirectional many-to-many mapping between real-world LR and HR images unsupervisedly. Experimental results on real-world image SR datasets indicate that SDFlow can generate diverse realistic LR and SR images both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-10-08T01:48:34Z)
• Learning to Super-Resolve Blurry Images with Events [62.61911224564196]
  Super-Resolution from a single motion Blurred image (SRB) is a severely ill-posed problem due to the joint degradation of motion blurs and low spatial resolution. We employ events to alleviate the burden of SRB and propose an Event-enhanced SRB (E-SRB) algorithm. We show that the proposed eSL-Net++ outperforms state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2023-02-27T13:46:42Z)
• HDR Reconstruction from Bracketed Exposures and Events [12.565039752529797]
  Reconstruction of high-quality HDR images is at the core of modern computational photography. We present a multi-modal end-to-end learning-based HDR imaging system that fuses bracketed images and event in the feature domain. Our framework exploits the higher temporal resolution of events by sub-sampling the input event streams using a sliding window.
  arXiv  Detail & Related papers  (2022-03-28T15:04:41Z)
• RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network [23.451063587138393]
  We propose a Raw Burst Super-Resolution Iterative Convolutional Neural Network (RBSRICNN) The proposed network produces the final output by an iterative refinement of the intermediate SR estimates. We demonstrate the effectiveness of our proposed approach in quantitative and qualitative experiments.
  arXiv  Detail & Related papers  (2021-10-25T19:01:28Z)
• Blind Motion Deblurring Super-Resolution: When Dynamic Spatio-Temporal Learning Meets Static Image Understanding [87.5799910153545]
  Single-image super-resolution (SR) and multi-frame SR are two ways to super resolve low-resolution images. Blind Motion Deblurring Super-Reslution Networks is proposed to learn dynamic-temporal information from single static motion-blurred images.
  arXiv  Detail & Related papers  (2021-05-27T11:52:45Z)
• Frequency Consistent Adaptation for Real World Super Resolution [64.91914552787668]
  We propose a novel Frequency Consistent Adaptation (FCA) that ensures the frequency domain consistency when applying Super-Resolution (SR) methods to the real scene. We estimate degradation kernels from unsupervised images and generate the corresponding Low-Resolution (LR) images. Based on the domain-consistent LR-HR pairs, we train easy-implemented Convolutional Neural Network (CNN) SR models.
  arXiv  Detail & Related papers  (2020-12-18T08:25:39Z)
• DDet: Dual-path Dynamic Enhancement Network for Real-World Image Super-Resolution [69.2432352477966]
  Real image super-resolution(Real-SR) focus on the relationship between real-world high-resolution(HR) and low-resolution(LR) image. In this article, we propose a Dual-path Dynamic Enhancement Network(DDet) for Real-SR. Unlike conventional methods which stack up massive convolutional blocks for feature representation, we introduce a content-aware framework to study non-inherently aligned image pair.
  arXiv  Detail & Related papers  (2020-02-25T18:24:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.