Neural Scene Flow Prior

• URL: http://arxiv.org/abs/2111.01253v1
• Date: Mon, 1 Nov 2021 20:44:12 GMT
• Title: Neural Scene Flow Prior
• Authors: Xueqian Li, Jhony Kaesemodel Pontes, Simon Lucey
• Abstract summary: Before the deep learning revolution, many perception algorithms were based on runtime optimization in conjunction with a strong prior/regularization penalty. This paper revisits the scene flow problem that relies predominantly on runtime optimization and strong regularization. A central innovation here is the inclusion of a neural scene flow prior, which uses the architecture of neural networks as a new type of implicit regularizer.
• Score: 30.878829330230797
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Before the deep learning revolution, many perception algorithms were based on runtime optimization in conjunction with a strong prior/regularization penalty. A prime example of this in computer vision is optical and scene flow. Supervised learning has largely displaced the need for explicit regularization. Instead, they rely on large amounts of labeled data to capture prior statistics, which are not always readily available for many problems. Although optimization is employed to learn the neural network, the weights of this network are frozen at runtime. As a result, these learning solutions are domain-specific and do not generalize well to other statistically different scenarios. This paper revisits the scene flow problem that relies predominantly on runtime optimization and strong regularization. A central innovation here is the inclusion of a neural scene flow prior, which uses the architecture of neural networks as a new type of implicit regularizer. Unlike learning-based scene flow methods, optimization occurs at runtime, and our approach needs no offline datasets -- making it ideal for deployment in new environments such as autonomous driving. We show that an architecture based exclusively on multilayer perceptrons (MLPs) can be used as a scene flow prior. Our method attains competitive -- if not better -- results on scene flow benchmarks. Also, our neural prior's implicit and continuous scene flow representation allows us to estimate dense long-term correspondences across a sequence of point clouds. The dense motion information is represented by scene flow fields where points can be propagated through time by integrating motion vectors. We demonstrate such a capability by accumulating a sequence of lidar point clouds.

Related papers

• Understanding Optimization in Deep Learning with Central Flows [53.66160508990508]
  We show that an RMS's implicit behavior can be explicitly captured by a "central flow:" a differential equation. We show that these flows can empirically predict long-term optimization trajectories of generic neural networks.
  arXiv  Detail & Related papers  (2024-10-31T17:58:13Z)
• Neural Eulerian Scene Flow Fields [59.57980592109722]
  EulerFlow works out-of-the-box without tuning across multiple domains. It exhibits emergent 3D point tracking behavior by solving its estimated ODE over long-time horizons. It outperforms all prior art on the Argoverse 2 2024 Scene Flow Challenge.
  arXiv  Detail & Related papers  (2024-10-02T20:56:45Z)
• MemFlow: Optical Flow Estimation and Prediction with Memory [54.22820729477756]
  We present MemFlow, a real-time method for optical flow estimation and prediction with memory. Our method enables memory read-out and update modules for aggregating historical motion information in real-time. Our approach seamlessly extends to the future prediction of optical flow based on past observations.
  arXiv  Detail & Related papers  (2024-04-07T04:56:58Z)
• OptFlow: Fast Optimization-based Scene Flow Estimation without Supervision [6.173968909465726]
  We present OptFlow, a fast optimization-based scene flow estimation method. It achieves state-of-the-art performance for scene flow estimation on popular autonomous driving benchmarks.
  arXiv  Detail & Related papers  (2024-01-04T21:47:56Z)
• Multi-Body Neural Scene Flow [37.31530794244607]
  We show that multi-body rigidity can be achieved without the cumbersome and brittle strategy of constraining the $SE(3)$ parameters of each rigid body. This is achieved by regularizing the scene flow optimization to encourage isometry in flow predictions for rigid bodies. We conduct extensive experiments on real-world datasets and demonstrate that our approach outperforms the state-of-the-art in 3D scene flow and long-term point-wise 4D trajectory prediction.
  arXiv  Detail & Related papers  (2023-10-16T11:37:53Z)
• Fast Neural Scene Flow [36.29234109363439]
  A coordinate neural network estimates scene flow at runtime, without any training. In this paper, we demonstrate that scene flow is different -- with the dominant computational bottleneck stemming from the loss function itself. Our fast neural scene flow (FNSF) approach reports for the first time real-time performance comparable to learning methods.
  arXiv  Detail & Related papers  (2023-04-18T16:37:18Z)
• SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow [25.577386156273256]
  Scene flow estimation is a long-standing problem in computer vision, where the goal is to find the 3D motion of a scene from its consecutive observations. We introduce SCOOP, a new method for scene flow estimation that can be learned on a small amount of data without employing ground-truth flow supervision.
  arXiv  Detail & Related papers  (2022-11-25T10:52:02Z)
• Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Critic [137.04558017227583]
  Actor-critic (AC) algorithms, empowered by neural networks, have had significant empirical success in recent years. We take a mean-field perspective on the evolution and convergence of feature-based neural AC. We prove that neural AC finds the globally optimal policy at a sublinear rate.
  arXiv  Detail & Related papers  (2021-12-27T06:09:50Z)
• FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation [87.74617110803189]
  Estimating the 3D motion of points in a scene, known as scene flow, is a core problem in computer vision. We present a recurrent architecture that learns a single step of an unrolled iterative alignment procedure for refining scene flow predictions.
  arXiv  Detail & Related papers  (2020-11-19T23:23:48Z)
• Scene Flow from Point Clouds with or without Learning [47.03163552693887]
  Scene flow is the three-dimensional (3D) motion field of a scene. Current learning-based approaches seek to estimate the scene flow directly from point clouds. We present a simple and interpretable objective function to recover the scene flow from point clouds.
  arXiv  Detail & Related papers  (2020-10-31T17:24:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.