All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced
Motion Modeling
- URL: http://arxiv.org/abs/2007.11762v2
- Date: Sat, 9 Jan 2021 03:50:58 GMT
- Title: All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced
Motion Modeling
- Authors: Zhixiang Chi, Rasoul Mohammadi Nasiri, Zheng Liu, Juwei Lu, Jin Tang,
Konstantinos N Plataniotis
- Abstract summary: State-of-the-art methods are iterative solutions interpolating one frame at the time.
This work introduces a true multi-frame interpolator.
It utilizes a pyramidal style network in the temporal domain to complete the multi-frame task in one-shot.
- Score: 52.425236515695914
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent advances in high refresh rate displays as well as the increased
interest in high rate of slow motion and frame up-conversion fuel the demand
for efficient and cost-effective multi-frame video interpolation solutions. To
that regard, inserting multiple frames between consecutive video frames are of
paramount importance for the consumer electronics industry. State-of-the-art
methods are iterative solutions interpolating one frame at the time. They
introduce temporal inconsistencies and clearly noticeable visual artifacts.
Departing from the state-of-the-art, this work introduces a true multi-frame
interpolator. It utilizes a pyramidal style network in the temporal domain to
complete the multi-frame interpolation task in one-shot. A novel flow
estimation procedure using a relaxed loss function, and an advanced,
cubic-based, motion model is also used to further boost interpolation accuracy
when complex motion segments are encountered. Results on the Adobe240 dataset
show that the proposed method generates visually pleasing, temporally
consistent frames, outperforms the current best off-the-shelf method by 1.57db
in PSNR with 8 times smaller model and 7.7 times faster. The proposed method
can be easily extended to interpolate a large number of new frames while
remaining efficient because of the one-shot mechanism.
Related papers
- Motion-Aware Video Frame Interpolation [49.49668436390514]
We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames.
It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
arXiv Detail & Related papers (2024-02-05T11:00:14Z) - Enhanced Bi-directional Motion Estimation for Video Frame Interpolation [0.05541644538483946]
We present a novel yet effective algorithm for motion-based video frame estimation.
Our method achieves excellent performance on a broad range of video frame benchmarks.
arXiv Detail & Related papers (2022-06-17T06:08:43Z) - Many-to-many Splatting for Efficient Video Frame Interpolation [80.10804399840927]
Motion-based video frame relies on optical flow to warp pixels from inputs to desired instant.
Many-to-Many (M2M) splatting framework to interpolate frames efficiently.
M2M has minuscule computational overhead when interpolating arbitrary number of in-between frames.
arXiv Detail & Related papers (2022-04-07T15:29:42Z) - Long-term Video Frame Interpolation via Feature Propagation [95.18170372022703]
Video frame (VFI) works generally predict intermediate frame(s) by first estimating the motion between inputs and then warping the inputs to the target time with the estimated motion.
This approach is not optimal when the temporal distance between the input sequence increases.
We propose a propagation network (PNet) by extending the classic feature-level forecasting with a novel motion-to-feature approach.
arXiv Detail & Related papers (2022-03-29T10:47:06Z) - FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised.
We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
arXiv Detail & Related papers (2020-12-15T18:59:30Z) - ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and
Interpolation [38.52446103418748]
We introduce a novel architecture, Adaptive Latent Attention Network (ALANET), which synthesizes sharp high frame-rate videos.
We employ combination of self-attention and cross-attention module between consecutive frames in the latent space to generate optimized representation for each frame.
Our method performs favorably against various state-of-the-art approaches, even though we tackle a much more difficult problem.
arXiv Detail & Related papers (2020-08-31T21:11:53Z) - Blurry Video Frame Interpolation [57.77512131536132]
We propose a blurry video frame method to reduce blur motion and up-convert frame rate simultaneously.
Specifically, we develop a pyramid module to cyclically synthesize clear intermediate frames.
Our method performs favorably against state-of-the-art methods.
arXiv Detail & Related papers (2020-02-27T17:00:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.