Degrees of Freedom Matter: Inferring Dynamics from Point Trajectories

• URL: http://arxiv.org/abs/2406.03625v1
• Date: Wed, 5 Jun 2024 21:02:10 GMT
• Title: Degrees of Freedom Matter: Inferring Dynamics from Point Trajectories
• Authors: Yan Zhang, Sergey Prokudin, Marko Mihajlovic, Qianli Ma, Siyu Tang,
• Abstract summary: We aim to learn an implicit motion field parameterized by a neural network to predict the movement of novel points within same domain. We exploit intrinsic regularization provided by SIREN, and modify the input layer to produce atemporally smooth motion field. Our experiments assess the model's performance in predicting unseen point trajectories and its application in temporal mesh alignment with deformation.
• Score: 28.701879490459675
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding the dynamics of generic 3D scenes is fundamentally challenging in computer vision, essential in enhancing applications related to scene reconstruction, motion tracking, and avatar creation. In this work, we address the task as the problem of inferring dense, long-range motion of 3D points. By observing a set of point trajectories, we aim to learn an implicit motion field parameterized by a neural network to predict the movement of novel points within the same domain, without relying on any data-driven or scene-specific priors. To achieve this, our approach builds upon the recently introduced dynamic point field model that learns smooth deformation fields between the canonical frame and individual observation frames. However, temporal consistency between consecutive frames is neglected, and the number of required parameters increases linearly with the sequence length due to per-frame modeling. To address these shortcomings, we exploit the intrinsic regularization provided by SIREN, and modify the input layer to produce a spatiotemporally smooth motion field. Additionally, we analyze the motion field Jacobian matrix, and discover that the motion degrees of freedom (DOFs) in an infinitesimal area around a point and the network hidden variables have different behaviors to affect the model's representational power. This enables us to improve the model representation capability while retaining the model compactness. Furthermore, to reduce the risk of overfitting, we introduce a regularization term based on the assumption of piece-wise motion smoothness. Our experiments assess the model's performance in predicting unseen point trajectories and its application in temporal mesh alignment with guidance. The results demonstrate its superiority and effectiveness. The code and data for the project are publicly available: \url{https://yz-cnsdqz.github.io/eigenmotion/DOMA/}

Related papers

• MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion [118.74385965694694]
  We present Motion DUSt3R (MonST3R), a novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes. By simply estimating a pointmap for each timestep, we can effectively adapt DUST3R's representation, previously only used for static scenes, to dynamic scenes. We show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics.
  arXiv  Detail & Related papers  (2024-10-04T18:00:07Z)
• Future Does Matter: Boosting 3D Object Detection with Temporal Motion Estimation in Point Cloud Sequences [25.74000325019015]
  We introduce a novel LiDAR 3D object detection framework, namely LiSTM, to facilitate spatial-temporal feature learning with cross-frame motion forecasting information. We have conducted experiments on the aggregation and nuScenes datasets to demonstrate that the proposed framework achieves superior 3D detection performance.
  arXiv  Detail & Related papers  (2024-09-06T16:29:04Z)
• Shuffled Autoregression For Motion Interpolation [53.61556200049156]
  This work aims to provide a deep-learning solution for the motion task. We propose a novel framework, referred to as emphShuffled AutoRegression, which expands the autoregression to generate in arbitrary (shuffled) order. We also propose an approach to constructing a particular kind of dependency graph, with three stages assembled into an end-to-end spatial-temporal motion Transformer.
  arXiv  Detail & Related papers  (2023-06-10T07:14:59Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• PREF: Predictability Regularized Neural Motion Fields [68.60019434498703]
  Knowing 3D motions in a dynamic scene is essential to many vision applications. We leverage a neural motion field for estimating the motion of all points in a multiview setting. We propose to regularize the estimated motion to be predictable.
  arXiv  Detail & Related papers  (2022-09-21T22:32:37Z)
• Neural Motion Fields: Encoding Grasp Trajectories as Implicit Value Functions [65.84090965167535]
  We present Neural Motion Fields, a novel object representation which encodes both object point clouds and the relative task trajectories as an implicit value function parameterized by a neural network. This object-centric representation models a continuous distribution over the SE(3) space and allows us to perform grasping reactively by leveraging sampling-based MPC to optimize this value function.
  arXiv  Detail & Related papers  (2022-06-29T18:47:05Z)
• NeMF: Neural Motion Fields for Kinematic Animation [6.570955948572252]
  We express the vast motion space as a continuous function over time, hence the name Neural Motion Fields (NeMF) We use a neural network to learn this function for miscellaneous sets of motions. We train our model with diverse human motion dataset and quadruped dataset to prove its versatility.
  arXiv  Detail & Related papers  (2022-06-04T05:53:27Z)
• Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding [61.57847727651068]
  Temporal sentence grounding aims to localize a target segment in an untrimmed video semantically according to a given sentence query. Most previous works focus on learning frame-level features of each whole frame in the entire video, and directly match them with the textual information. We propose a novel Motion- and Appearance-guided 3D Semantic Reasoning Network (MA3SRN), which incorporates optical-flow-guided motion-aware, detection-based appearance-aware, and 3D-aware object-level features.
  arXiv  Detail & Related papers  (2022-03-06T13:57:09Z)
• Any Motion Detector: Learning Class-agnostic Scene Dynamics from a Sequence of LiDAR Point Clouds [4.640835690336654]
  We propose a novel real-time approach of temporal context aggregation for motion detection and motion parameters estimation. We introduce an ego-motion compensation layer to achieve real-time inference with performance comparable to a naive odometric transform of the original point cloud sequence.
  arXiv  Detail & Related papers  (2020-04-24T10:40:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.