Real-Time Intermediate Flow Estimation for Video Frame Interpolation

• URL: http://arxiv.org/abs/2011.06294v12
• Date: Fri, 12 Nov 2021 13:58:23 GMT
• Title: Real-Time Intermediate Flow Estimation for Video Frame Interpolation
• Authors: Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, Shuchang Zhou
• Abstract summary: RIFE is a Real-time Intermediate Flow Estimation for VFI. A privileged distillation scheme is designed for stable IFNet training. RIFE achieves state-of-the-art performance on several public benchmarks.
• Score: 50.12253023531497
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-time video frame interpolation (VFI) is very useful in video processing, media players, and display devices. We propose RIFE, a Real-time Intermediate Flow Estimation algorithm for VFI. To realize a high-quality flow-based VFI method, RIFE uses a neural network named IFNet that can estimate the intermediate flows end-to-end with much faster speed. A privileged distillation scheme is designed for stable IFNet training and improve the overall performance. RIFE does not rely on pre-trained optical flow models and can support arbitrary-timestep frame interpolation with the temporal encoding input. Experiments demonstrate that RIFE achieves state-of-the-art performance on several public benchmarks. Compared with the popular SuperSlomo and DAIN methods, RIFE is 4--27 times faster and produces better results. Furthermore, RIFE can be extended to wider applications thanks to temporal encoding. The code is available at https://github.com/megvii-research/ECCV2022-RIFE.

Related papers

• Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers [95.68243351895107]
  We propose a holistic, video-centric paradigm named textbfLocal textbfDiffusion textbfForcing for textbfVideo textbfFrame textbfInterpolation (LDF-VFI)<n>Our framework is built upon an auto-regressive diffusion transformer that models the entire video sequence to ensure long-range temporal coherence.<n>LDF-VFI achieves state-of-the-art performance on challenging long-sequence benchmarks, demonstrating superior per
  arXiv  Detail & Related papers  (2026-01-21T12:58:52Z)
• SG-RIFE: Semantic-Guided Real-Time Intermediate Flow Estimation with Diffusion-Competitive Perceptual Quality [0.0]
  Real-time Video Frame Interpolation (VFI) has long been dominated by flow-based methods like RIFE.<n>Recent diffusion-based approaches achieve state-of-the-art perceptual quality but suffer from prohibitive latency, rendering them impractical for real-time applications.<n>We propose Semantic-Guided RIFE (SG-RIFE), which augments a pre-trained RIFE backbone with semantic priors from a frozen DINOv3 Vision Transformer.
  arXiv  Detail & Related papers  (2025-12-20T06:50:55Z)
• MemFlow: Optical Flow Estimation and Prediction with Memory [54.22820729477756]
  We present MemFlow, a real-time method for optical flow estimation and prediction with memory. Our method enables memory read-out and update modules for aggregating historical motion information in real-time. Our approach seamlessly extends to the future prediction of optical flow based on past observations.
  arXiv  Detail & Related papers  (2024-04-07T04:56:58Z)
• StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences [31.210626775505407]
  Occlusions between consecutive frames have long posed a significant challenge in optical flow estimation. We present a Streamlined In-batch Multi-frame (SIM) pipeline tailored to video input, attaining a similar level of time efficiency to two-frame networks. StreamFlow not only excels in terms of performance on challenging KITTI and Sintel datasets, with particular improvement in occluded areas.
  arXiv  Detail & Related papers  (2023-11-28T07:53:51Z)
• Frame Flexible Network [52.623337134518835]
  Existing video recognition algorithms always conduct different training pipelines for inputs with different frame numbers. If we evaluate the model using other frames which are not used in training, we observe the performance will drop significantly. We propose a general framework, named Frame Flexible Network (FFN), which enables the model to be evaluated at different frames to adjust its computation.
  arXiv  Detail & Related papers  (2023-03-26T20:51:35Z)
• IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation [44.04110765492441]
  We devise an efficient encoder-decoder based network, termed IFRNet, for fast intermediate frame synthesizing. Experiments on various benchmarks demonstrate the excellent performance and fast inference speed of proposed approaches.
  arXiv  Detail & Related papers  (2022-05-29T10:18:18Z)
• Spatio-Temporal Multi-Flow Network for Video Frame Interpolation [3.6053802212032995]
  Video frame (VFI) is a very active research topic, with applications spanning computer vision, post production and video encoding. We present a novel deep learning based VFI method, ST-MFNet, based on a Spatio-Temporal Multi-Flow architecture.
  arXiv  Detail & Related papers  (2021-11-30T15:18:46Z)
• DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting [50.17500790309477]
  DeMFI-Net is a joint deblurring and multi-frame framework. It converts blurry videos of lower-frame-rate to sharp videos at higher-frame-rate. It achieves state-of-the-art (SOTA) performances for diverse datasets.
  arXiv  Detail & Related papers  (2021-11-19T00:00:15Z)
• Optical-Flow-Reuse-Based Bidirectional Recurrent Network for Space-Time Video Super-Resolution [52.899234731501075]
  Space-time video super-resolution (ST-VSR) simultaneously increases the spatial resolution and frame rate for a given video. Existing methods typically suffer from difficulties in how to efficiently leverage information from a large range of neighboring frames. We propose a coarse-to-fine bidirectional recurrent neural network instead of using ConvLSTM to leverage knowledge between adjacent frames.
  arXiv  Detail & Related papers  (2021-10-13T15:21:30Z)
• FastRIFE: Optimization of Real-Time Intermediate Flow Estimation for Video Frame Interpolation [0.0]
  This paper proposes the FastRIFE algorithm, which is some speed improvement of the RIFE (Real-Time Intermediate Flow Estimation) model. All source codes are available at https://gitlab.com/malwinq/interpolation-of-images-for-slow-motion-videos.
  arXiv  Detail & Related papers  (2021-05-27T22:31:40Z)
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
  FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised. We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
  arXiv  Detail & Related papers  (2020-12-15T18:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.