Self-Supervised Learning of Non-Rigid Residual Flow and Ego-Motion

• URL: http://arxiv.org/abs/2009.10467v2
• Date: Mon, 19 Oct 2020 15:21:21 GMT
• Title: Self-Supervised Learning of Non-Rigid Residual Flow and Ego-Motion
• Authors: Ivan Tishchenko, Sandro Lombardi, Martin R. Oswald, Marc Pollefeys
• Abstract summary: We present an alternative method for end-to-end scene flow learning by joint estimation of non-rigid residual flow and ego-motion flow for dynamic 3D scenes. We extend the supervised framework with self-supervisory signals based on the temporal consistency property of a point cloud sequence.
• Score: 63.18340058854517
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most of the current scene flow methods choose to model scene flow as a per point translation vector without differentiating between static and dynamic components of 3D motion. In this work we present an alternative method for end-to-end scene flow learning by joint estimation of non-rigid residual flow and ego-motion flow for dynamic 3D scenes. We propose to learn the relative rigid transformation from a pair of point clouds followed by an iterative refinement. We then learn the non-rigid flow from transformed inputs with the deducted rigid part of the flow. Furthermore, we extend the supervised framework with self-supervisory signals based on the temporal consistency property of a point cloud sequence. Our solution allows both training in a supervised mode complemented by self-supervisory loss terms as well as training in a fully self-supervised mode. We demonstrate that decomposition of scene flow into non-rigid flow and ego-motion flow along with an introduction of the self-supervisory signals allowed us to outperform the current state-of-the-art supervised methods.

Related papers

• Let Occ Flow: Self-Supervised 3D Occupancy Flow Prediction [14.866463843514156]
  Let Occ Flow is the first self-supervised work for joint 3D occupancy and occupancy flow prediction using only camera inputs. Our approach incorporates a novel attention-based temporal fusion module to capture dynamic object dependencies. Our method extends differentiable rendering to 3D volumetric flow fields.
  arXiv  Detail & Related papers  (2024-07-10T12:20:11Z)
• SeFlow: A Self-Supervised Scene Flow Method in Autonomous Driving [18.88208422580103]
  Scene flow estimation predicts the 3D motion at each point in successive LiDAR scans. Current state-of-the-art methods require annotated data to train scene flow networks. We propose SeFlow, a self-supervised method that integrates efficient dynamic classification into a learning-based scene flow pipeline.
  arXiv  Detail & Related papers  (2024-07-01T18:22:54Z)
• Self-Supervised 3D Scene Flow Estimation and Motion Prediction using Local Rigidity Prior [100.98123802027847]
  We investigate self-supervised 3D scene flow estimation and class-agnostic motion prediction on point clouds. We generate pseudo scene flow labels for self-supervised learning through piecewise rigid motion estimation. Our method achieves new state-of-the-art performance in self-supervised scene flow learning.
  arXiv  Detail & Related papers  (2023-10-17T14:06:55Z)
• SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow [25.577386156273256]
  Scene flow estimation is a long-standing problem in computer vision, where the goal is to find the 3D motion of a scene from its consecutive observations. We introduce SCOOP, a new method for scene flow estimation that can be learned on a small amount of data without employing ground-truth flow supervision.
  arXiv  Detail & Related papers  (2022-11-25T10:52:02Z)
• Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels [71.11151016581806]
  We propose a novel scene flow method that captures 3D motions from point clouds without relying on ground-truth scene flow annotations. Our method not only outperforms state-of-the-art self-supervised approaches, but also outperforms some supervised approaches that use accurate ground-truth flows.
  arXiv  Detail & Related papers  (2022-03-23T18:20:03Z)
• SCTN: Sparse Convolution-Transformer Network for Scene Flow Estimation [71.2856098776959]
  Estimating 3D motions for point clouds is challenging, since a point cloud is unordered and its density is significantly non-uniform. We propose a novel architecture named Sparse Convolution-Transformer Network (SCTN) that equips the sparse convolution with the transformer. We show that the learned relation-based contextual information is rich and helpful for matching corresponding points, benefiting scene flow estimation.
  arXiv  Detail & Related papers  (2021-05-10T15:16:14Z)
• Weakly Supervised Learning of Rigid 3D Scene Flow [81.37165332656612]
  We propose a data-driven scene flow estimation algorithm exploiting the observation that many 3D scenes can be explained by a collection of agents moving as rigid bodies. We showcase the effectiveness and generalization capacity of our method on four different autonomous driving datasets.
  arXiv  Detail & Related papers  (2021-02-17T18:58:02Z)
• FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation [87.74617110803189]
  Estimating the 3D motion of points in a scene, known as scene flow, is a core problem in computer vision. We present a recurrent architecture that learns a single step of an unrolled iterative alignment procedure for refining scene flow predictions.
  arXiv  Detail & Related papers  (2020-11-19T23:23:48Z)
• Do not trust the neighbors! Adversarial Metric Learning for Self-Supervised Scene Flow Estimation [0.0]
  Scene flow is the task of estimating 3D motion vectors to individual points of a dynamic 3D scene. We propose a 3D scene flow benchmark and a novel self-supervised setup for training flow models. We find that our setup is able to keep motion coherence and preserve local geometries, which many self-supervised baselines fail to grasp.
  arXiv  Detail & Related papers  (2020-11-01T17:41:32Z)
• Adversarial Self-Supervised Scene Flow Estimation [15.278302535191866]
  This work proposes a metric learning approach for self-supervised scene flow estimation. We outline a benchmark for self-supervised scene flow estimation: the Scene Flow Sandbox.
  arXiv  Detail & Related papers  (2020-11-01T16:37:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.