VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation

• URL: http://arxiv.org/abs/2303.08340v3
• Date: Sun, 20 Aug 2023 15:14:34 GMT
• Title: VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation
• Abstract summary: VideoFlow is a novel optical flow estimation framework for videos. We first propose a TRi-frame Optical Flow (TROF) module that estimates bi-directional optical flows for the center frame in a three-frame manner. With the iterative flow estimation refinement, the information fused in individual TROFs can be propagated into the whole sequence via MOP.
• Score: 61.660040308290796
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce VideoFlow, a novel optical flow estimation framework for videos. In contrast to previous methods that learn to estimate optical flow from two frames, VideoFlow concurrently estimates bi-directional optical flows for multiple frames that are available in videos by sufficiently exploiting temporal cues. We first propose a TRi-frame Optical Flow (TROF) module that estimates bi-directional optical flows for the center frame in a three-frame manner. The information of the frame triplet is iteratively fused onto the center frame. To extend TROF for handling more frames, we further propose a MOtion Propagation (MOP) module that bridges multiple TROFs and propagates motion features between adjacent TROFs. With the iterative flow estimation refinement, the information fused in individual TROFs can be propagated into the whole sequence via MOP. By effectively exploiting video information, VideoFlow presents extraordinary performance, ranking 1st on all public benchmarks. On the Sintel benchmark, VideoFlow achieves 1.649 and 0.991 average end-point-error (AEPE) on the final and clean passes, a 15.1% and 7.6% error reduction from the best-published results (1.943 and 1.073 from FlowFormer++). On the KITTI-2015 benchmark, VideoFlow achieves an F1-all error of 3.65%, a 19.2% error reduction from the best-published result (4.52% from FlowFormer++). Code is released at \url{https://github.com/XiaoyuShi97/VideoFlow}.

Related papers

• Physics-Grounded Fluid Video Generation with a Simulation Dataset and Dual-Stream Optical-Flow Supervision [78.04385989413382]
  Video diffusion models generate visually compelling content but routinely violate elementary physics when the subject involves fluids.<n>We build a physics-simulation fluid dataset combining 1,638 MPM-simulated pouring/sloshing videos with 2,320 keyword-filtered real pouring videos.<n>We introduce a dual-stream image-to-video architecture built on a pretrained diffusion-filtereder video generator.
  arXiv  Detail & Related papers  (2026-07-28T06:13:18Z)
• MeanFlowNFT: Bringing Forward-Process RL to Average-Velocity Generators [48.85363546552086]
  MeanFlowNFT is inspired by the MeanFlow identity, which bridges average and instantaneous velocities.<n>Experiments on image and video generation show that MeanFlowNFT consistently improves baselines.<n>It can even surpass multi-step RL-tuned diffusion while using only a few sampling steps.
  arXiv  Detail & Related papers  (2026-07-16T17:59:02Z)
• FlowC2S: Flowing from Current to Succeeding Frames for Fast and Memory-Efficient Video Continuation [4.429835955129622]
  This paper introduces a novel methodology for generating fast and memory-efficient video continuations.<n>Our method, dubbed FlowC2S, fine-tunes a pre-trained text-to-video flow model to learn a vector field between the current and succeeding video chunks.
  arXiv  Detail & Related papers  (2026-04-19T21:38:21Z)
• TLB-VFI: Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation [4.261090951843438]
  Video Frame Interpolation (VFI) aims to predict the intermediate frame $I_n$ based on two consecutive neighboring frames.<n>Recent approaches apply diffusion models (both image-based and video-based) in this task and achieve strong performance.<n>We propose Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation (TLB-VFI), an efficient video-based diffusion model.
  arXiv  Detail & Related papers  (2025-07-07T13:25:32Z)
• Improving Video Generation with Human Feedback [81.48120703718774]
  Video generation has achieved significant advances, but issues like unsmooth motion and misalignment between videos and prompts persist. We develop a systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. We introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy.
  arXiv  Detail & Related papers  (2025-01-23T18:55:41Z)
• MemFlow: Optical Flow Estimation and Prediction with Memory [54.22820729477756]
  We present MemFlow, a real-time method for optical flow estimation and prediction with memory. Our method enables memory read-out and update modules for aggregating historical motion information in real-time. Our approach seamlessly extends to the future prediction of optical flow based on past observations.
  arXiv  Detail & Related papers  (2024-04-07T04:56:58Z)
• FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis [66.2611385251157]
  Diffusion models have transformed the image-to-image (I2I) synthesis and are now permeating into videos. This paper proposes a consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video.
  arXiv  Detail & Related papers  (2023-12-29T16:57:12Z)
• MVFlow: Deep Optical Flow Estimation of Compressed Videos with Motion Vector Prior [16.633665275166706]
  We propose an optical flow model, MVFlow, which uses motion vectors to improve the speed and accuracy of optical flow estimation for compressed videos. The experimental results demonstrate the superiority of our proposed MVFlow by 1.09 compared to existing models or save time to achieve similar accuracy to existing models.
  arXiv  Detail & Related papers  (2023-08-03T07:16:18Z)
• Towards An End-to-End Framework for Flow-Guided Video Inpainting [68.71844500391023]
  We propose an End-to-End framework for Flow-Guided Video Inpainting (E$2$FGVI) The proposed method outperforms state-of-the-art methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2022-04-06T08:24:47Z)
• PanoFlow: Learning Optical Flow for Panoramic Images [8.009873804948919]
  We put forward a novel network framework, PanoFlow, to learn optical flow for panoramic images. We propose a Flow Distortion Augmentation (FDA) method and a Cyclic Flow Estimation (CFE) method. Our proposed approach reduces the End-Point-Error (EPE) on the established Flow360 dataset by 26%.
  arXiv  Detail & Related papers  (2022-02-27T16:03:38Z)
• GMFlow: Learning Optical Flow via Global Matching [124.57850500778277]
  We propose a GMFlow framework for learning optical flow estimation. It consists of three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for global feature matching, and a self-attention layer for flow propagation. Our new framework outperforms 32-iteration RAFT's performance on the challenging Sintel benchmark.
  arXiv  Detail & Related papers  (2021-11-26T18:59:56Z)
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
  FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised. We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
  arXiv  Detail & Related papers  (2020-12-15T18:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.