Joint Video Multi-Frame Interpolation and Deblurring under Unknown Exposure Time

• URL: http://arxiv.org/abs/2303.15043v1
• Date: Mon, 27 Mar 2023 09:43:42 GMT
• Title: Joint Video Multi-Frame Interpolation and Deblurring under Unknown Exposure Time
• Authors: Wei Shang, Dongwei Ren, Yi Yang, Hongzhi Zhang, Kede Ma, Wangmeng Zuo
• Abstract summary: In this work, we aim ambitiously for a more realistic and challenging task - joint video multi-frame and deblurring under unknown exposure time. We first adopt a variant of supervised contrastive learning to construct an exposure-aware representation from input blurred frames. We then build our video reconstruction network upon the exposure and motion representation by progressive exposure-adaptive convolution and motion refinement.
• Score: 101.91824315554682
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural videos captured by consumer cameras often suffer from low framerate and motion blur due to the combination of dynamic scene complexity, lens and sensor imperfection, and less than ideal exposure setting. As a result, computational methods that jointly perform video frame interpolation and deblurring begin to emerge with the unrealistic assumption that the exposure time is known and fixed. In this work, we aim ambitiously for a more realistic and challenging task - joint video multi-frame interpolation and deblurring under unknown exposure time. Toward this goal, we first adopt a variant of supervised contrastive learning to construct an exposure-aware representation from input blurred frames. We then train two U-Nets for intra-motion and inter-motion analysis, respectively, adapting to the learned exposure representation via gain tuning. We finally build our video reconstruction network upon the exposure and motion representation by progressive exposure-adaptive convolution and motion refinement. Extensive experiments on both simulated and real-world datasets show that our optimized method achieves notable performance gains over the state-of-the-art on the joint video x8 interpolation and deblurring task. Moreover, on the seemingly implausible x16 interpolation task, our method outperforms existing methods by more than 1.5 dB in terms of PSNR.

Related papers

• CMTA: Cross-Modal Temporal Alignment for Event-guided Video Deblurring [44.30048301161034]
  Video deblurring aims to enhance the quality of restored results in motion-red videos by gathering information from adjacent video frames. We propose two modules: 1) Intra-frame feature enhancement operates within the exposure time of a single blurred frame, and 2) Inter-frame temporal feature alignment gathers valuable long-range temporal information to target frames. We demonstrate that our proposed methods outperform state-of-the-art frame-based and event-based motion deblurring methods through extensive experiments conducted on both synthetic and real-world deblurring datasets.
  arXiv  Detail & Related papers  (2024-08-27T10:09:17Z)
• Event-Based Frame Interpolation with Ad-hoc Deblurring [68.97825675372354]
  We propose a general method for event-based frame that performs deblurring ad-hoc on input videos. Our network consistently outperforms state-of-the-art methods on frame, single image deblurring and the joint task of deblurring. Our code and dataset will be made publicly available.
  arXiv  Detail & Related papers  (2023-01-12T18:19:00Z)
• Time Lens++: Event-based Frame Interpolation with Parametric Non-linear Flow and Multi-scale Fusion [47.57998625129672]
  We introduce multi-scale feature-level fusion and computing one-shot non-linear inter-frame motion from events and images. We show that our method improves the reconstruction quality by up to 0.2 dB in terms of PSNR and up to 15% in LPIPS score.
  arXiv  Detail & Related papers  (2022-03-31T17:14:58Z)
• Unifying Motion Deblurring and Frame Interpolation with Events [11.173687810873433]
  Slow shutter speed and long exposure time of frame-based cameras often cause visual blur and loss of inter-frame information, degenerating the overall quality of captured videos. We present a unified framework of event-based motion deblurring and frame enhancement for blurry video enhancement, where the extremely low latency of events is leveraged to alleviate motion blur and facilitate intermediate frame prediction. By exploring the mutual constraints among blurry frames, latent images, and event streams, we further propose a self-supervised learning framework to enable network training with real-world blurry videos and events.
  arXiv  Detail & Related papers  (2022-03-23T03:43:12Z)
• Video Frame Interpolation without Temporal Priors [91.04877640089053]
  Video frame aims to synthesize non-exist intermediate frames in a video sequence. The temporal priors of videos, i.e. frames per second (FPS) and frame exposure time, may vary from different camera sensors. We devise a novel optical flow refinement strategy for better synthesizing results.
  arXiv  Detail & Related papers  (2021-12-02T12:13:56Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)
• Motion-blurred Video Interpolation and Extrapolation [72.3254384191509]
  We present a novel framework for deblurring, interpolating and extrapolating sharp frames from a motion-blurred video in an end-to-end manner. To ensure temporal coherence across predicted frames and address potential temporal ambiguity, we propose a simple, yet effective flow-based rule.
  arXiv  Detail & Related papers  (2021-03-04T12:18:25Z)
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
  FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised. We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
  arXiv  Detail & Related papers  (2020-12-15T18:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.