Multi-Body Neural Scene Flow

• URL: http://arxiv.org/abs/2310.10301v2
• Date: Tue, 6 Feb 2024 05:34:12 GMT
• Title: Multi-Body Neural Scene Flow
• Authors: Kavisha Vidanapathirana, Shin-Fang Chng, Xueqian Li, Simon Lucey
• Abstract summary: We show that multi-body rigidity can be achieved without the cumbersome and brittle strategy of constraining the $SE(3)$ parameters of each rigid body. This is achieved by regularizing the scene flow optimization to encourage isometry in flow predictions for rigid bodies. We conduct extensive experiments on real-world datasets and demonstrate that our approach outperforms the state-of-the-art in 3D scene flow and long-term point-wise 4D trajectory prediction.
• Score: 37.31530794244607
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The test-time optimization of scene flow - using a coordinate network as a neural prior - has gained popularity due to its simplicity, lack of dataset bias, and state-of-the-art performance. We observe, however, that although coordinate networks capture general motions by implicitly regularizing the scene flow predictions to be spatially smooth, the neural prior by itself is unable to identify the underlying multi-body rigid motions present in real-world data. To address this, we show that multi-body rigidity can be achieved without the cumbersome and brittle strategy of constraining the $SE(3)$ parameters of each rigid body as done in previous works. This is achieved by regularizing the scene flow optimization to encourage isometry in flow predictions for rigid bodies. This strategy enables multi-body rigidity in scene flow while maintaining a continuous flow field, hence allowing dense long-term scene flow integration across a sequence of point clouds. We conduct extensive experiments on real-world datasets and demonstrate that our approach outperforms the state-of-the-art in 3D scene flow and long-term point-wise 4D trajectory prediction. The code is available at: https://github.com/kavisha725/MBNSF.

Related papers

• MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion [118.74385965694694]
  We present Motion DUSt3R (MonST3R), a novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes. By simply estimating a pointmap for each timestep, we can effectively adapt DUST3R's representation, previously only used for static scenes, to dynamic scenes. We show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics.
  arXiv  Detail & Related papers  (2024-10-04T18:00:07Z)
• Neural Eulerian Scene Flow Fields [59.57980592109722]
  EulerFlow works out-of-the-box without tuning across multiple domains. It exhibits emergent 3D point tracking behavior by solving its estimated ODE over long-time horizons. It outperforms all prior art on the Argoverse 2 2024 Scene Flow Challenge.
  arXiv  Detail & Related papers  (2024-10-02T20:56:45Z)
• STARFlow: Spatial Temporal Feature Re-embedding with Attentive Learning for Real-world Scene Flow [5.476991379461233]
  We propose global attentive flow embedding to match all-to-all point pairs in both Euclidean space. We leverage novel domain adaptive losses to bridge the gap of motion inference from synthetic to real-world. Our approach achieves state-of-the-art performance across various datasets, with particularly outstanding results on real-world LiDAR-scanned datasets.
  arXiv  Detail & Related papers  (2024-03-11T04:56:10Z)
• SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow [25.577386156273256]
  Scene flow estimation is a long-standing problem in computer vision, where the goal is to find the 3D motion of a scene from its consecutive observations. We introduce SCOOP, a new method for scene flow estimation that can be learned on a small amount of data without employing ground-truth flow supervision.
  arXiv  Detail & Related papers  (2022-11-25T10:52:02Z)
• Self-SuperFlow: Self-supervised Scene Flow Prediction in Stereo Sequences [12.650574326251023]
  In this paper, we explore the extension of a self-supervised loss for scene flow prediction. Regarding the KITTI scene flow benchmark, our method outperforms the corresponding supervised pre-training of the same network.
  arXiv  Detail & Related papers  (2022-06-30T13:55:17Z)
• Neural Scene Flow Prior [30.878829330230797]
  Before the deep learning revolution, many perception algorithms were based on runtime optimization in conjunction with a strong prior/regularization penalty. This paper revisits the scene flow problem that relies predominantly on runtime optimization and strong regularization. A central innovation here is the inclusion of a neural scene flow prior, which uses the architecture of neural networks as a new type of implicit regularizer.
  arXiv  Detail & Related papers  (2021-11-01T20:44:12Z)
• SCTN: Sparse Convolution-Transformer Network for Scene Flow Estimation [71.2856098776959]
  Estimating 3D motions for point clouds is challenging, since a point cloud is unordered and its density is significantly non-uniform. We propose a novel architecture named Sparse Convolution-Transformer Network (SCTN) that equips the sparse convolution with the transformer. We show that the learned relation-based contextual information is rich and helpful for matching corresponding points, benefiting scene flow estimation.
  arXiv  Detail & Related papers  (2021-05-10T15:16:14Z)
• Weakly Supervised Learning of Rigid 3D Scene Flow [81.37165332656612]
  We propose a data-driven scene flow estimation algorithm exploiting the observation that many 3D scenes can be explained by a collection of agents moving as rigid bodies. We showcase the effectiveness and generalization capacity of our method on four different autonomous driving datasets.
  arXiv  Detail & Related papers  (2021-02-17T18:58:02Z)
• Occlusion Guided Scene Flow Estimation on 3D Point Clouds [4.518012967046983]
  3D scene flow estimation is a vital tool in perceiving our environment given depth or range sensors. Here we propose a new scene flow architecture called OGSF-Net which tightly couples the learning for both flow and occlusions between frames. Their coupled symbiosis results in a more accurate prediction of flow in space.
  arXiv  Detail & Related papers  (2020-11-30T15:22:03Z)
• FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation [87.74617110803189]
  Estimating the 3D motion of points in a scene, known as scene flow, is a core problem in computer vision. We present a recurrent architecture that learns a single step of an unrolled iterative alignment procedure for refining scene flow predictions.
  arXiv  Detail & Related papers  (2020-11-19T23:23:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.