ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and
Interpolation
- URL: http://arxiv.org/abs/2009.01005v1
- Date: Mon, 31 Aug 2020 21:11:53 GMT
- Title: ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and
Interpolation
- Authors: Akash Gupta, Abhishek Aich, Amit K. Roy-Chowdhury
- Abstract summary: We introduce a novel architecture, Adaptive Latent Attention Network (ALANET), which synthesizes sharp high frame-rate videos.
We employ combination of self-attention and cross-attention module between consecutive frames in the latent space to generate optimized representation for each frame.
Our method performs favorably against various state-of-the-art approaches, even though we tackle a much more difficult problem.
- Score: 38.52446103418748
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Existing works address the problem of generating high frame-rate sharp videos
by separately learning the frame deblurring and frame interpolation modules.
Most of these approaches have a strong prior assumption that all the input
frames are blurry whereas in a real-world setting, the quality of frames
varies. Moreover, such approaches are trained to perform either of the two
tasks - deblurring or interpolation - in isolation, while many practical
situations call for both. Different from these works, we address a more
realistic problem of high frame-rate sharp video synthesis with no prior
assumption that input is always blurry. We introduce a novel architecture,
Adaptive Latent Attention Network (ALANET), which synthesizes sharp high
frame-rate videos with no prior knowledge of input frames being blurry or not,
thereby performing the task of both deblurring and interpolation. We
hypothesize that information from the latent representation of the consecutive
frames can be utilized to generate optimized representations for both frame
deblurring and frame interpolation. Specifically, we employ combination of
self-attention and cross-attention module between consecutive frames in the
latent space to generate optimized representation for each frame. The optimized
representation learnt using these attention modules help the model to generate
and interpolate sharp frames. Extensive experiments on standard datasets
demonstrate that our method performs favorably against various state-of-the-art
approaches, even though we tackle a much more difficult problem.
Related papers
- ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler [53.98558445900626]
Current image-to-video diffusion models, while powerful in generating videos from a single frame, need adaptation for two-frame conditioned generation.
We introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning.
Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames.
arXiv Detail & Related papers (2024-10-08T03:01:54Z) - Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation [93.18163456287164]
This paper proposes a novel text-guided video-to-video translation framework to adapt image models to videos.
Our framework achieves global style and local texture temporal consistency at a low cost.
arXiv Detail & Related papers (2023-06-13T17:52:23Z) - TTVFI: Learning Trajectory-Aware Transformer for Video Frame
Interpolation [50.49396123016185]
Video frame (VFI) aims to synthesize an intermediate frame between two consecutive frames.
We propose a novel Trajectory-aware Transformer for Video Frame Interpolation (TTVFI)
Our method outperforms other state-of-the-art methods in four widely-used VFI benchmarks.
arXiv Detail & Related papers (2022-07-19T03:37:49Z) - TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches.
We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
arXiv Detail & Related papers (2021-06-14T10:33:47Z) - ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring [92.40655035360729]
Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions.
We propose a novel implicit method to learn spatial correspondence among blurry frames in the feature space.
Our proposed method is evaluated on the widely-adopted DVD dataset, along with a newly collected High-Frame-Rate (1000 fps) dataset for Video Deblurring.
arXiv Detail & Related papers (2021-03-07T04:33:13Z) - Deep Sketch-guided Cartoon Video Inbetweening [24.00033622396297]
We propose a framework to produce cartoon videos by fetching the color information from two inputs while following the animated motion guided by a user sketch.
By explicitly considering the correspondence between frames and the sketch, we can achieve higher quality results than other image synthesis methods.
arXiv Detail & Related papers (2020-08-10T14:22:04Z) - All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced
Motion Modeling [52.425236515695914]
State-of-the-art methods are iterative solutions interpolating one frame at the time.
This work introduces a true multi-frame interpolator.
It utilizes a pyramidal style network in the temporal domain to complete the multi-frame task in one-shot.
arXiv Detail & Related papers (2020-07-23T02:34:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.