Time Lens++: Event-based Frame Interpolation with Parametric Non-linear
Flow and Multi-scale Fusion
- URL: http://arxiv.org/abs/2203.17191v1
- Date: Thu, 31 Mar 2022 17:14:58 GMT
- Title: Time Lens++: Event-based Frame Interpolation with Parametric Non-linear
Flow and Multi-scale Fusion
- Authors: Stepan Tulyakov, Alfredo Bochicchio, Daniel Gehrig, Stamatios
Georgoulis, Yuanyou Li, and Davide Scaramuzza
- Abstract summary: We introduce multi-scale feature-level fusion and computing one-shot non-linear inter-frame motion from events and images.
We show that our method improves the reconstruction quality by up to 0.2 dB in terms of PSNR and up to 15% in LPIPS score.
- Score: 47.57998625129672
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, video frame interpolation using a combination of frame- and
event-based cameras has surpassed traditional image-based methods both in terms
of performance and memory efficiency. However, current methods still suffer
from (i) brittle image-level fusion of complementary interpolation results,
that fails in the presence of artifacts in the fused image, (ii) potentially
temporally inconsistent and inefficient motion estimation procedures, that run
for every inserted frame and (iii) low contrast regions that do not trigger
events, and thus cause events-only motion estimation to generate artifacts.
Moreover, previous methods were only tested on datasets consisting of planar
and faraway scenes, which do not capture the full complexity of the real world.
In this work, we address the above problems by introducing multi-scale
feature-level fusion and computing one-shot non-linear inter-frame motion from
events and images, which can be efficiently sampled for image warping. We also
collect the first large-scale events and frames dataset consisting of more than
100 challenging scenes with depth variations, captured with a new experimental
setup based on a beamsplitter. We show that our method improves the
reconstruction quality by up to 0.2 dB in terms of PSNR and up to 15% in LPIPS
score.
Related papers
- Joint Video Multi-Frame Interpolation and Deblurring under Unknown
Exposure Time [101.91824315554682]
In this work, we aim ambitiously for a more realistic and challenging task - joint video multi-frame and deblurring under unknown exposure time.
We first adopt a variant of supervised contrastive learning to construct an exposure-aware representation from input blurred frames.
We then build our video reconstruction network upon the exposure and motion representation by progressive exposure-adaptive convolution and motion refinement.
arXiv Detail & Related papers (2023-03-27T09:43:42Z) - Event-Based Frame Interpolation with Ad-hoc Deblurring [68.97825675372354]
We propose a general method for event-based frame that performs deblurring ad-hoc on input videos.
Our network consistently outperforms state-of-the-art methods on frame, single image deblurring and the joint task of deblurring.
Our code and dataset will be made publicly available.
arXiv Detail & Related papers (2023-01-12T18:19:00Z) - Blur Interpolation Transformer for Real-World Motion from Blur [52.10523711510876]
We propose a encoded blur transformer (BiT) to unravel the underlying temporal correlation in blur.
Based on multi-scale residual Swin transformer blocks, we introduce dual-end temporal supervision and temporally symmetric ensembling strategies.
In addition, we design a hybrid camera system to collect the first real-world dataset of one-to-many blur-sharp video pairs.
arXiv Detail & Related papers (2022-11-21T13:10:10Z) - Event-based Image Deblurring with Dynamic Motion Awareness [10.81953574179206]
We introduce the first dataset containing pairs of real RGB blur images and related events during the exposure time.
Our results show better robustness overall when using events, with improvements in PSNR by up to 1.57dB on synthetic data and 1.08 dB on real event data.
arXiv Detail & Related papers (2022-08-24T09:39:55Z) - Unifying Motion Deblurring and Frame Interpolation with Events [11.173687810873433]
Slow shutter speed and long exposure time of frame-based cameras often cause visual blur and loss of inter-frame information, degenerating the overall quality of captured videos.
We present a unified framework of event-based motion deblurring and frame enhancement for blurry video enhancement, where the extremely low latency of events is leveraged to alleviate motion blur and facilitate intermediate frame prediction.
By exploring the mutual constraints among blurry frames, latent images, and event streams, we further propose a self-supervised learning framework to enable network training with real-world blurry videos and events.
arXiv Detail & Related papers (2022-03-23T03:43:12Z) - TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches.
We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
arXiv Detail & Related papers (2021-06-14T10:33:47Z) - Motion-blurred Video Interpolation and Extrapolation [72.3254384191509]
We present a novel framework for deblurring, interpolating and extrapolating sharp frames from a motion-blurred video in an end-to-end manner.
To ensure temporal coherence across predicted frames and address potential temporal ambiguity, we propose a simple, yet effective flow-based rule.
arXiv Detail & Related papers (2021-03-04T12:18:25Z) - Single Image Optical Flow Estimation with an Event Camera [38.92408855196647]
Event cameras are bio-inspired sensors that report intensity changes in microsecond resolution.
We propose a single image (potentially blurred) and events based optical flow estimation approach.
arXiv Detail & Related papers (2020-04-01T11:28:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.