UFD-PRiME: Unsupervised Joint Learning of Optical Flow and Stereo Depth through Pixel-Level Rigid Motion Estimation

• URL: http://arxiv.org/abs/2310.04712v1
• Date: Sat, 7 Oct 2023 07:08:25 GMT
• Title: UFD-PRiME: Unsupervised Joint Learning of Optical Flow and Stereo Depth through Pixel-Level Rigid Motion Estimation
• Authors: Shuai Yuan, Carlo Tomasi
• Abstract summary: Both optical flow and stereo disparities are image matches and can therefore benefit from joint training. We design a first network that estimates flow and disparity jointly and is trained without supervision. A second network, trained with optical flow from the first as pseudo-labels, takes disparities from the first network, estimates 3D rigid motion at every pixel, and reconstructs optical flow again.
• Score: 4.445751695675388
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Both optical flow and stereo disparities are image matches and can therefore benefit from joint training. Depth and 3D motion provide geometric rather than photometric information and can further improve optical flow. Accordingly, we design a first network that estimates flow and disparity jointly and is trained without supervision. A second network, trained with optical flow from the first as pseudo-labels, takes disparities from the first network, estimates 3D rigid motion at every pixel, and reconstructs optical flow again. A final stage fuses the outputs from the two networks. In contrast with previous methods that only consider camera motion, our method also estimates the rigid motions of dynamic objects, which are of key interest in applications. This leads to better optical flow with visibly more detailed occlusions and object boundaries as a result. Our unsupervised pipeline achieves 7.36% optical flow error on the KITTI-2015 benchmark and outperforms the previous state-of-the-art 9.38% by a wide margin. It also achieves slightly better or comparable stereo depth results. Code will be made available.

Related papers

• Skin the sheep not only once: Reusing Various Depth Datasets to Drive the Learning of Optical Flow [25.23550076996421]
  We propose to leverage the geometric connection between optical flow estimation and stereo matching. We turn the monocular depth datasets into stereo ones via virtual disparity. We also introduce virtual camera motion into stereo data to produce additional flows along the vertical direction.
  arXiv  Detail & Related papers  (2023-10-03T06:56:07Z)
• Deep Dynamic Scene Deblurring from Optical Flow [53.625999196063574]
  Deblurring can provide visually more pleasant pictures and make photography more convenient. It is difficult to model the non-uniform blur mathematically. We develop a convolutional neural network (CNN) to restore the sharp images from the deblurred features.
  arXiv  Detail & Related papers  (2023-01-18T06:37:21Z)
• Weakly-Supervised Optical Flow Estimation for Time-of-Flight [11.496094830445054]
  We propose a training algorithm, which allows to supervise Optical Flow networks directly on the reconstructed depth. We demonstrate that this approach enables the training of OF networks to align raw iToF measurements and compensate motion artifacts in the iToF depth images.
  arXiv  Detail & Related papers  (2022-10-11T09:47:23Z)
• USegScene: Unsupervised Learning of Depth, Optical Flow and Ego-Motion with Semantic Guidance and Coupled Networks [31.600708674008384]
  USegScene is a framework for semantically guided unsupervised learning of depth, optical flow and ego-motion estimation for stereo camera images. We present results on the popular KITTI dataset and show that our approach outperforms other methods by a large margin.
  arXiv  Detail & Related papers  (2022-07-15T13:25:47Z)
• Sensor-Guided Optical Flow [53.295332513139925]
  This paper proposes a framework to guide an optical flow network with external cues to achieve superior accuracy on known or unseen domains. We show how these can be obtained by combining depth measurements from active sensors with geometry and hand-crafted optical flow algorithms.
  arXiv  Detail & Related papers  (2021-09-30T17:59:57Z)
• Dense Optical Flow from Event Cameras [55.79329250951028]
  We propose to incorporate feature correlation and sequential processing into dense optical flow estimation from event cameras. Our proposed approach computes dense optical flow and reduces the end-point error by 23% on MVSEC.
  arXiv  Detail & Related papers  (2021-08-24T07:39:08Z)
• Self-Supervised Approach for Facial Movement Based Optical Flow [8.19666118455293]
  We generate optical flow ground truth for face images using facial key-points. We train the FlowNetS architecture to test its performance on the generated dataset. The optical flow obtained using this work has promising applications in facial expression analysis.
  arXiv  Detail & Related papers  (2021-05-04T02:38:11Z)
• High-Resolution Optical Flow from 1D Attention and Correlation [89.61824964952949]
  We propose a new method for high-resolution optical flow estimation with significantly less computation. We first perform a 1D attention operation in the vertical direction of the target image, and then a simple 1D correlation in the horizontal direction of the attended image. Experiments on Sintel, KITTI and real-world 4K resolution images demonstrated the effectiveness and superiority of our proposed method.
  arXiv  Detail & Related papers  (2021-04-28T17:56:34Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)
• Self-Supervised Linear Motion Deblurring [112.75317069916579]
  Deep convolutional neural networks are state-of-the-art for image deblurring. We present a differentiable reblur model for self-supervised motion deblurring. Our experiments demonstrate that self-supervised single image deblurring is really feasible.
  arXiv  Detail & Related papers  (2020-02-10T20:15:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.