Dense Monocular Motion Segmentation Using Optical Flow and Pseudo Depth Map: A Zero-Shot Approach

• URL: http://arxiv.org/abs/2406.18837v1
• Date: Thu, 27 Jun 2024 02:11:33 GMT
• Title: Dense Monocular Motion Segmentation Using Optical Flow and Pseudo Depth Map: A Zero-Shot Approach
• Authors: Yuxiang Huang, Yuhao Chen, John Zelek,
• Abstract summary: We propose an innovative hybrid approach to perform dense motion segmentation without requiring any training. Our method initiates by automatically generating object proposals for each frame using foundation models. The integration of depth maps derived from state-of-the-art monocular depth estimation models significantly enhances the motion cues provided by optical flow.
• Score: 6.805017878728801
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Motion segmentation from a single moving camera presents a significant challenge in the field of computer vision. This challenge is compounded by the unknown camera movements and the lack of depth information of the scene. While deep learning has shown impressive capabilities in addressing these issues, supervised models require extensive training on massive annotated datasets, and unsupervised models also require training on large volumes of unannotated data, presenting significant barriers for both. In contrast, traditional methods based on optical flow do not require training data, however, they often fail to capture object-level information, leading to over-segmentation or under-segmentation. In addition, they also struggle in complex scenes with substantial depth variations and non-rigid motion, due to the overreliance of optical flow. To overcome these challenges, we propose an innovative hybrid approach that leverages the advantages of both deep learning methods and traditional optical flow based methods to perform dense motion segmentation without requiring any training. Our method initiates by automatically generating object proposals for each frame using foundation models. These proposals are then clustered into distinct motion groups using both optical flow and relative depth maps as motion cues. The integration of depth maps derived from state-of-the-art monocular depth estimation models significantly enhances the motion cues provided by optical flow, particularly in handling motion parallax issues. Our method is evaluated on the DAVIS-Moving and YTVOS-Moving datasets, and the results demonstrate that our method outperforms the best unsupervised method and closely matches with the state-of-theart supervised methods.

Related papers

• Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation [34.529280562470746]
  We introduce a novel self-supervised loss combining the Contrast Maximization framework with a non-linear motion prior in the form of pixel-level trajectories. Their effectiveness is demonstrated in two scenarios: In dense continuous-time motion estimation, our method improves the zero-shot performance of a synthetically trained model by 29%.
  arXiv  Detail & Related papers  (2024-07-15T15:18:28Z)
• Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenes [40.46121828229776]
  Dynamo-Depth is an approach that disambiguates dynamical motion by jointly learning monocular depth, 3D independent flow field, and motion segmentation from unlabeled monocular videos. Our proposed method achieves state-of-the-art performance on monocular depth estimation on Open and nuScenes with significant improvement in the depth of moving objects.
  arXiv  Detail & Related papers  (2023-10-29T03:24:16Z)
• Skin the sheep not only once: Reusing Various Depth Datasets to Drive the Learning of Optical Flow [25.23550076996421]
  We propose to leverage the geometric connection between optical flow estimation and stereo matching. We turn the monocular depth datasets into stereo ones via virtual disparity. We also introduce virtual camera motion into stereo data to produce additional flows along the vertical direction.
  arXiv  Detail & Related papers  (2023-10-03T06:56:07Z)
• ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild [57.37891682117178]
  We present a robust dense indirect structure-from-motion method for videos that is based on dense correspondence from pairwise optical flow. A novel neural network architecture is proposed for processing irregular point trajectory data. Experiments on MPI Sintel dataset show that our system produces significantly more accurate camera trajectories.
  arXiv  Detail & Related papers  (2022-07-19T09:19:45Z)
• Towards Scale-Aware, Robust, and Generalizable Unsupervised Monocular Depth Estimation by Integrating IMU Motion Dynamics [74.1720528573331]
  Unsupervised monocular depth and ego-motion estimation has drawn extensive research attention in recent years. We propose DynaDepth, a novel scale-aware framework that integrates information from vision and IMU motion dynamics. We validate the effectiveness of DynaDepth by conducting extensive experiments and simulations on the KITTI and Make3D datasets.
  arXiv  Detail & Related papers  (2022-07-11T07:50:22Z)
• Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation [76.58256020932312]
  Estimating the motion of the camera together with the 3D structure of the scene from a monocular vision system is a complex task. We present a self-supervised learning framework for 3D object motion field estimation from monocular videos.
  arXiv  Detail & Related papers  (2021-10-13T16:45:01Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)
• Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency [114.02182755620784]
  We present an end-to-end joint training framework that explicitly models 6-DoF motion of multiple dynamic objects, ego-motion and depth in a monocular camera setup without supervision. Our framework is shown to outperform the state-of-the-art depth and motion estimation methods.
  arXiv  Detail & Related papers  (2021-02-04T14:26:42Z)
• Joint Unsupervised Learning of Optical Flow and Egomotion with Bi-Level Optimization [59.9673626329892]
  We exploit the global relationship between optical flow and camera motion using epipolar geometry. We use implicit differentiation to enable back-propagation through the lower-level geometric optimization layer independent of its implementation.
  arXiv  Detail & Related papers  (2020-02-26T22:28:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.