VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation

• URL: http://arxiv.org/abs/2303.08340v3
• Date: Sun, 20 Aug 2023 15:14:34 GMT
• Title: VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation
• Authors: Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, Hongsheng Li
• Abstract summary: VideoFlow is a novel optical flow estimation framework for videos. We first propose a TRi-frame Optical Flow (TROF) module that estimates bi-directional optical flows for the center frame in a three-frame manner. With the iterative flow estimation refinement, the information fused in individual TROFs can be propagated into the whole sequence via MOP.
• Score: 61.660040308290796
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce VideoFlow, a novel optical flow estimation framework for videos. In contrast to previous methods that learn to estimate optical flow from two frames, VideoFlow concurrently estimates bi-directional optical flows for multiple frames that are available in videos by sufficiently exploiting temporal cues. We first propose a TRi-frame Optical Flow (TROF) module that estimates bi-directional optical flows for the center frame in a three-frame manner. The information of the frame triplet is iteratively fused onto the center frame. To extend TROF for handling more frames, we further propose a MOtion Propagation (MOP) module that bridges multiple TROFs and propagates motion features between adjacent TROFs. With the iterative flow estimation refinement, the information fused in individual TROFs can be propagated into the whole sequence via MOP. By effectively exploiting video information, VideoFlow presents extraordinary performance, ranking 1st on all public benchmarks. On the Sintel benchmark, VideoFlow achieves 1.649 and 0.991 average end-point-error (AEPE) on the final and clean passes, a 15.1% and 7.6% error reduction from the best-published results (1.943 and 1.073 from FlowFormer++). On the KITTI-2015 benchmark, VideoFlow achieves an F1-all error of 3.65%, a 19.2% error reduction from the best-published result (4.52% from FlowFormer++). Code is released at \url{https://github.com/XiaoyuShi97/VideoFlow}.

Related papers

• MemFlow: Optical Flow Estimation and Prediction with Memory [54.22820729477756]
  We present MemFlow, a real-time method for optical flow estimation and prediction with memory. Our method enables memory read-out and update modules for aggregating historical motion information in real-time. Our approach seamlessly extends to the future prediction of optical flow based on past observations.
  arXiv  Detail & Related papers  (2024-04-07T04:56:58Z)
• FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis [66.2611385251157]
  Diffusion models have transformed the image-to-image (I2I) synthesis and are now permeating into videos. This paper proposes a consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video.
  arXiv  Detail & Related papers  (2023-12-29T16:57:12Z)
• MVFlow: Deep Optical Flow Estimation of Compressed Videos with Motion Vector Prior [16.633665275166706]
  We propose an optical flow model, MVFlow, which uses motion vectors to improve the speed and accuracy of optical flow estimation for compressed videos. The experimental results demonstrate the superiority of our proposed MVFlow by 1.09 compared to existing models or save time to achieve similar accuracy to existing models.
  arXiv  Detail & Related papers  (2023-08-03T07:16:18Z)
• Towards An End-to-End Framework for Flow-Guided Video Inpainting [68.71844500391023]
  We propose an End-to-End framework for Flow-Guided Video Inpainting (E$2$FGVI) The proposed method outperforms state-of-the-art methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-04-06T08:24:47Z)
• PanoFlow: Learning Optical Flow for Panoramic Images [8.009873804948919]
  We put forward a novel network framework, PanoFlow, to learn optical flow for panoramic images. We propose a Flow Distortion Augmentation (FDA) method and a Cyclic Flow Estimation (CFE) method. Our proposed approach reduces the End-Point-Error (EPE) on the established Flow360 dataset by 26%.
  arXiv  Detail & Related papers  (2022-02-27T16:03:38Z)
• GMFlow: Learning Optical Flow via Global Matching [124.57850500778277]
  We propose a GMFlow framework for learning optical flow estimation. It consists of three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for global feature matching, and a self-attention layer for flow propagation. Our new framework outperforms 32-iteration RAFT's performance on the challenging Sintel benchmark.
  arXiv  Detail & Related papers  (2021-11-26T18:59:56Z)
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
  FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised. We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
  arXiv  Detail & Related papers  (2020-12-15T18:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.