IDO-VFI: Identifying Dynamics via Optical Flow Guidance for Video Frame
Interpolation with Events
- URL: http://arxiv.org/abs/2305.10198v2
- Date: Thu, 18 May 2023 07:06:39 GMT
- Title: IDO-VFI: Identifying Dynamics via Optical Flow Guidance for Video Frame
Interpolation with Events
- Authors: Chenyang Shi, Hanxiao Liu, Jing Jin, Wenzhuo Li, Yuzhen Li, Boyi Wei,
Yibo Zhang
- Abstract summary: Event cameras are ideal for capturing inter-frame dynamics with their extremely high temporal resolution.
We propose an event-and-frame-based video frame method named IDO-VFI that assigns varying amounts of computation for different sub-regions.
Our proposed method maintains high-quality performance while reducing computation time and computational effort by 10% and 17% respectively on Vimeo90K datasets.
- Score: 14.098949778274733
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Video frame interpolation aims to generate high-quality intermediate frames
from boundary frames and increase frame rate. While existing linear, symmetric
and nonlinear models are used to bridge the gap from the lack of inter-frame
motion, they cannot reconstruct real motions. Event cameras, however, are ideal
for capturing inter-frame dynamics with their extremely high temporal
resolution. In this paper, we propose an event-and-frame-based video frame
interpolation method named IDO-VFI that assigns varying amounts of computation
for different sub-regions via optical flow guidance. The proposed method first
estimates the optical flow based on frames and events, and then decides whether
to further calculate the residual optical flow in those sub-regions via a
Gumbel gating module according to the optical flow amplitude. Intermediate
frames are eventually generated through a concise Transformer-based fusion
network. Our proposed method maintains high-quality performance while reducing
computation time and computational effort by 10% and 17% respectively on
Vimeo90K datasets, compared with a unified process on the whole region.
Moreover, our method outperforms state-of-the-art frame-only and
frames-plus-events methods on multiple video frame interpolation benchmarks.
Codes and models are available at https://github.com/shicy17/IDO-VFI.
Related papers
- Disentangled Motion Modeling for Video Frame Interpolation [40.83962594702387]
Video frame (VFI) aims to synthesize intermediate frames in between existing frames to enhance visual smoothness and quality.
We introduce disentangled Motion Modeling (MoMo), a diffusion-based approach for VFI that enhances visual quality by focusing on intermediate motion modeling.
arXiv Detail & Related papers (2024-06-25T03:50:20Z) - Motion-Aware Video Frame Interpolation [49.49668436390514]
We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames.
It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
arXiv Detail & Related papers (2024-02-05T11:00:14Z) - Dynamic Frame Interpolation in Wavelet Domain [57.25341639095404]
Video frame is an important low-level computation vision task, which can increase frame rate for more fluent visual experience.
Existing methods have achieved great success by employing advanced motion models and synthesis networks.
WaveletVFI can reduce computation up to 40% while maintaining similar accuracy, making it perform more efficiently against other state-of-the-arts.
arXiv Detail & Related papers (2023-09-07T06:41:15Z) - Meta-Interpolation: Time-Arbitrary Frame Interpolation via Dual
Meta-Learning [65.85319901760478]
We consider processing different time-steps with adaptively generated convolutional kernels in a unified way with the help of meta-learning.
We develop a dual meta-learned frame framework to synthesize intermediate frames with the guidance of context information and optical flow.
arXiv Detail & Related papers (2022-07-27T17:36:23Z) - TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches.
We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
arXiv Detail & Related papers (2021-06-14T10:33:47Z) - RAI-Net: Range-Adaptive LiDAR Point Cloud Frame Interpolation Network [5.225160072036824]
LiDAR point cloud frame, which synthesizes the intermediate frame between the captured frames, has emerged as an important issue for many applications.
We propose a novel LiDAR point cloud optical frame method, which exploits range images (RIs) as an intermediate representation with CNNs to conduct the frame process.
Our method consistently achieves superior frame results with better perceptual quality to that of using state-of-the-art video frame methods.
arXiv Detail & Related papers (2021-06-01T13:59:08Z) - FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised.
We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
arXiv Detail & Related papers (2020-12-15T18:59:30Z) - All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced
Motion Modeling [52.425236515695914]
State-of-the-art methods are iterative solutions interpolating one frame at the time.
This work introduces a true multi-frame interpolator.
It utilizes a pyramidal style network in the temporal domain to complete the multi-frame task in one-shot.
arXiv Detail & Related papers (2020-07-23T02:34:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.