Skin the sheep not only once: Reusing Various Depth Datasets to Drive the Learning of Optical Flow

• URL: http://arxiv.org/abs/2310.01833v1
• Date: Tue, 3 Oct 2023 06:56:07 GMT
• Title: Skin the sheep not only once: Reusing Various Depth Datasets to Drive the Learning of Optical Flow
• Authors: Sheng-Chi Huang, Wei-Chen Chiu
• Abstract summary: We propose to leverage the geometric connection between optical flow estimation and stereo matching. We turn the monocular depth datasets into stereo ones via virtual disparity. We also introduce virtual camera motion into stereo data to produce additional flows along the vertical direction.
• Score: 25.23550076996421
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Optical flow estimation is crucial for various applications in vision and robotics. As the difficulty of collecting ground truth optical flow in real-world scenarios, most of the existing methods of learning optical flow still adopt synthetic dataset for supervised training or utilize photometric consistency across temporally adjacent video frames to drive the unsupervised learning, where the former typically has issues of generalizability while the latter usually performs worse than the supervised ones. To tackle such challenges, we propose to leverage the geometric connection between optical flow estimation and stereo matching (based on the similarity upon finding pixel correspondences across images) to unify various real-world depth estimation datasets for generating supervised training data upon optical flow. Specifically, we turn the monocular depth datasets into stereo ones via synthesizing virtual disparity, thus leading to the flows along the horizontal direction; moreover, we introduce virtual camera motion into stereo data to produce additional flows along the vertical direction. Furthermore, we propose applying geometric augmentations on one image of an optical flow pair, encouraging the optical flow estimator to learn from more challenging cases. Lastly, as the optical flow maps under different geometric augmentations actually exhibit distinct characteristics, an auxiliary classifier which trains to identify the type of augmentation from the appearance of the flow map is utilized to further enhance the learning of the optical flow estimator. Our proposed method is general and is not tied to any particular flow estimator, where extensive experiments based on various datasets and optical flow estimation models verify its efficacy and superiority.

Related papers

• OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation [55.676358801492114]
  We propose OCAI, a method that supports robust frame ambiguities by generating intermediate video frames alongside optical flows in between. Our evaluations demonstrate superior quality and enhanced optical flow accuracy on established benchmarks such as Sintel and KITTI.
  arXiv  Detail & Related papers  (2024-03-26T20:23:48Z)
• Joint Self-supervised Depth and Optical Flow Estimation towards Dynamic Objects [3.794605440322862]
  In this work, we construct a joint inter-frame-supervised depth and optical flow estimation framework. For motion segmentation, we adaptively segment the preliminary estimated optical flow map with large areas of connectivity. Our proposed joint depth and optical flow estimation outperforms existing depth estimators on the KITTI Depth dataset.
  arXiv  Detail & Related papers  (2023-09-07T04:00:52Z)
• Improving Lens Flare Removal with General Purpose Pipeline and Multiple Light Sources Recovery [69.71080926778413]
  flare artifacts can affect image visual quality and downstream computer vision tasks. Current methods do not consider automatic exposure and tone mapping in image signal processing pipeline. We propose a solution to improve the performance of lens flare removal by revisiting the ISP and design a more reliable light sources recovery strategy.
  arXiv  Detail & Related papers  (2023-08-31T04:58:17Z)
• Optical Flow for Autonomous Driving: Applications, Challenges and Improvements [0.9023847175654602]
  We propose and evaluate training strategies to improve a learning-based optical flow algorithm. While trained with synthetic data, the model demonstrates strong capabilities to generalize to real world fisheye data. We propose a novel, generic semi-supervised framework that significantly boosts performances of existing methods in low light.
  arXiv  Detail & Related papers  (2023-01-11T12:01:42Z)
• Deep 360$^\circ$ Optical Flow Estimation Based on Multi-Projection Fusion [10.603670927163002]
  This paper focuses on the 360$circ$ optical flow estimation using deep neural networks to support increasingly popular VR applications. We propose a novel multi-projection fusion framework that fuses the optical flow predicted by the models trained using different projection methods. We also build the first large-scale panoramic optical flow dataset to support the training of neural networks and the evaluation of panoramic optical flow estimation methods.
  arXiv  Detail & Related papers  (2022-07-27T16:48:32Z)
• Dense Optical Flow from Event Cameras [55.79329250951028]
  We propose to incorporate feature correlation and sequential processing into dense optical flow estimation from event cameras. Our proposed approach computes dense optical flow and reduces the end-point error by 23% on MVSEC.
  arXiv  Detail & Related papers  (2021-08-24T07:39:08Z)
• PCA Event-Based Otical Flow for Visual Odometry [0.0]
  We present a Principal Component Analysis approach to the problem of event-based optical flow estimation. We show that the best variant of our proposed method, dedicated to the real-time context of visual odometry, is about two times faster compared to state-of-the-art implementations.
  arXiv  Detail & Related papers  (2021-05-08T18:30:44Z)
• Learning optical flow from still images [53.295332513139925]
  We introduce a framework to generate accurate ground-truth optical flow annotations quickly and in large amounts from any readily available single real picture. We virtually move the camera in the reconstructed environment with known motion vectors and rotation angles. When trained with our data, state-of-the-art optical flow networks achieve superior generalization to unseen real data.
  arXiv  Detail & Related papers  (2021-04-08T17:59:58Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)
• Joint Unsupervised Learning of Optical Flow and Egomotion with Bi-Level Optimization [59.9673626329892]
  We exploit the global relationship between optical flow and camera motion using epipolar geometry. We use implicit differentiation to enable back-propagation through the lower-level geometric optimization layer independent of its implementation.
  arXiv  Detail & Related papers  (2020-02-26T22:28:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.