OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation

• URL: http://arxiv.org/abs/2403.18092v1
• Date: Tue, 26 Mar 2024 20:23:48 GMT
• Title: OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation
• Authors: Jisoo Jeong, Hong Cai, Risheek Garrepalli, Jamie Menjay Lin, Munawar Hayat, Fatih Porikli,
• Abstract summary: We propose OCAI, a method that supports robust frame ambiguities by generating intermediate video frames alongside optical flows in between. Our evaluations demonstrate superior quality and enhanced optical flow accuracy on established benchmarks such as Sintel and KITTI.
• Score: 55.676358801492114
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The scarcity of ground-truth labels poses one major challenge in developing optical flow estimation models that are both generalizable and robust. While current methods rely on data augmentation, they have yet to fully exploit the rich information available in labeled video sequences. We propose OCAI, a method that supports robust frame interpolation by generating intermediate video frames alongside optical flows in between. Utilizing a forward warping approach, OCAI employs occlusion awareness to resolve ambiguities in pixel values and fills in missing values by leveraging the forward-backward consistency of optical flows. Additionally, we introduce a teacher-student style semi-supervised learning method on top of the interpolated frames. Using a pair of unlabeled frames and the teacher model's predicted optical flow, we generate interpolated frames and flows to train a student model. The teacher's weights are maintained using Exponential Moving Averaging of the student. Our evaluations demonstrate perceptually superior interpolation quality and enhanced optical flow accuracy on established benchmarks such as Sintel and KITTI.

Related papers

• Motion-Aware Video Frame Interpolation [49.49668436390514]
  We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
  arXiv  Detail & Related papers  (2024-02-05T11:00:14Z)
• Meta-Interpolation: Time-Arbitrary Frame Interpolation via Dual Meta-Learning [65.85319901760478]
  We consider processing different time-steps with adaptively generated convolutional kernels in a unified way with the help of meta-learning. We develop a dual meta-learned frame framework to synthesize intermediate frames with the guidance of context information and optical flow.
  arXiv  Detail & Related papers  (2022-07-27T17:36:23Z)
• Imposing Consistency for Optical Flow Estimation [73.53204596544472]
  Imposing consistency through proxy tasks has been shown to enhance data-driven learning. This paper introduces novel and effective consistency strategies for optical flow estimation.
  arXiv  Detail & Related papers  (2022-04-14T22:58:30Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)
• RAI-Net: Range-Adaptive LiDAR Point Cloud Frame Interpolation Network [5.225160072036824]
  LiDAR point cloud frame, which synthesizes the intermediate frame between the captured frames, has emerged as an important issue for many applications. We propose a novel LiDAR point cloud optical frame method, which exploits range images (RIs) as an intermediate representation with CNNs to conduct the frame process. Our method consistently achieves superior frame results with better perceptual quality to that of using state-of-the-art video frame methods.
  arXiv  Detail & Related papers  (2021-06-01T13:59:08Z)
• PDWN: Pyramid Deformable Warping Network for Video Interpolation [11.62213584807003]
  We propose a light but effective model, called Pyramid Deformable Warping Network (PDWN) PDWN uses a pyramid structure to generate DConv offsets of the unknown middle frame with respect to the known frames through coarse-to-fine successive refinements. Our method achieves better or on-par accuracy compared to state-of-the-art models on multiple datasets.
  arXiv  Detail & Related papers  (2021-04-04T02:08:57Z)
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
  FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised. We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
  arXiv  Detail & Related papers  (2020-12-15T18:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.