Video Frame Interpolation via Generalized Deformable Convolution
- URL: http://arxiv.org/abs/2008.10680v3
- Date: Thu, 18 Mar 2021 16:09:35 GMT
- Title: Video Frame Interpolation via Generalized Deformable Convolution
- Authors: Zhihao Shi, Xiaohong Liu, Kangdi Shi, Linhui Dai, Jun Chen
- Abstract summary: Video frame aims at synthesizing intermediate frames from nearby source frames while maintaining spatial and temporal consistencies.
Existing deeplearning-based video frame methods can be divided into two categories: flow-based methods and kernel-based methods.
A novel mechanism named generalized deformable convolution is proposed, which can effectively learn motion in a data-driven manner and freely select sampling points in space-time.
- Score: 18.357839820102683
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Video frame interpolation aims at synthesizing intermediate frames from
nearby source frames while maintaining spatial and temporal consistencies. The
existing deep-learning-based video frame interpolation methods can be roughly
divided into two categories: flow-based methods and kernel-based methods. The
performance of flow-based methods is often jeopardized by the inaccuracy of
flow map estimation due to oversimplified motion models, while that of
kernel-based methods tends to be constrained by the rigidity of kernel shape.
To address these performance-limiting issues, a novel mechanism named
generalized deformable convolution is proposed, which can effectively learn
motion information in a data-driven manner and freely select sampling points in
space-time. We further develop a new video frame interpolation method based on
this mechanism. Our extensive experiments demonstrate that the new method
performs favorably against the state-of-the-art, especially when dealing with
complex motions.
Related papers
- Motion-Aware Video Frame Interpolation [49.49668436390514]
We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames.
It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
arXiv Detail & Related papers (2024-02-05T11:00:14Z) - Meta-Interpolation: Time-Arbitrary Frame Interpolation via Dual
Meta-Learning [65.85319901760478]
We consider processing different time-steps with adaptively generated convolutional kernels in a unified way with the help of meta-learning.
We develop a dual meta-learned frame framework to synthesize intermediate frames with the guidance of context information and optical flow.
arXiv Detail & Related papers (2022-07-27T17:36:23Z) - Video Frame Interpolation Based on Deformable Kernel Region [18.55904569126297]
We propose a deformable convolution for video, which can break the fixed grid restrictions on the kernel region.
Experiments are conducted on four datasets to demonstrate the superior performance of the proposed model.
arXiv Detail & Related papers (2022-04-25T02:03:04Z) - Video Frame Interpolation Transformer [86.20646863821908]
We propose a Transformer-based video framework that allows content-aware aggregation weights and considers long-range dependencies with the self-attention operations.
To avoid the high computational cost of global self-attention, we introduce the concept of local attention into video.
In addition, we develop a multi-scale frame scheme to fully realize the potential of Transformers.
arXiv Detail & Related papers (2021-11-27T05:35:10Z) - TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches.
We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
arXiv Detail & Related papers (2021-06-14T10:33:47Z) - Video Frame Interpolation via Structure-Motion based Iterative Fusion [19.499969588931414]
We propose a structure-motion based iterative fusion method for video frame Interpolation.
Inspired by the observation that audiences have different visual preferences on foreground and background objects, we for the first time propose to use saliency masks in the evaluation processes of the task of video frame Interpolation.
arXiv Detail & Related papers (2021-05-11T22:11:17Z) - FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised.
We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
arXiv Detail & Related papers (2020-12-15T18:59:30Z) - Multiple Video Frame Interpolation via Enhanced Deformable Separable
Convolution [67.83074893311218]
Kernel-based methods predict pixels with a single convolution process that convolves source frames with spatially adaptive local kernels.
We propose enhanced deformable separable convolution (EDSC) to estimate not only adaptive kernels, but also offsets, masks and biases.
We show that our method performs favorably against the state-of-the-art methods across a broad range of datasets.
arXiv Detail & Related papers (2020-06-15T01:10:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.