CanFields: Consolidating 4D Dynamic Shapes from Raw Scans

• URL: http://arxiv.org/abs/2406.18582v2
• Date: Wed, 27 Nov 2024 18:14:05 GMT
• Title: CanFields: Consolidating 4D Dynamic Shapes from Raw Scans
• Authors: Miaowei Wang, Changjian Li, Amir Vaxman,
• Abstract summary: We introduce Canonical Consolidation Fields (CanFields), a new method for reconstructing a time series of independently captured 3D scans into a single, coherent deforming shape.<n>CanFields effectively learns geometry and deformation in an unsupervised way by incorporating two geometric priors.<n>We validate the robustness and accuracy of CanFields on diverse raw scans, demonstrating its superior performance even with missing regions, sparse frames, and noise.
• Score: 12.221737707194261
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Canonical Consolidation Fields (CanFields), a new method for reconstructing a time series of independently captured 3D scans into a single, coherent deforming shape. This 4D representation enables continuous refinement across both space and time. Unlike prior methods that often over-smooth the geometry or produce topological and geometric artifacts, CanFields effectively learns geometry and deformation in an unsupervised way by incorporating two geometric priors. First, we introduce a dynamic consolidator module that adjusts the input and assigns confidence scores, balancing the learning of the canonical shape and its deformations. Second, we use low-frequency velocity fields to guide deformation while preserving fine details in canonical shapes through high-frequency bias. We validate the robustness and accuracy of CanFields on diverse raw scans, demonstrating its superior performance even with missing regions, sparse frames, and noise. Code is available in the supplementary materials and will be released publicly upon acceptance.

Related papers

• 4DPV: 4D Pet from Videos by Coarse-to-Fine Non-Rigid Radiance Fields [16.278222277579655]
  We present a coarse-to-fine neural model to recover the camera pose and the 4D reconstruction of an unknown object from multiple RGB sequences in the wild. Our approach does not consider any pre-built 3D template nor 3D training data as well as controlled conditions. We thoroughly validate the method on challenging scenarios with complex and real-world deformations.
  arXiv  Detail & Related papers  (2024-11-15T15:31:58Z)
• Deformation-Guided Unsupervised Non-Rigid Shape Matching [7.327850781641328]
  We present an unsupervised data-driven approach for non-rigid shape matching. Our approach is particularly robust when matching digitized shapes using 3D scanners.
  arXiv  Detail & Related papers  (2023-11-27T09:55:55Z)
• Non-Rigid Shape Registration via Deep Functional Maps Prior [1.9249120068573227]
  We propose a learning-based framework for non-rigid shape registration without correspondence supervision. We deform source mesh towards the target point cloud, guided by correspondences induced by high-dimensional embeddings. Our pipeline achieves state-of-the-art results on several benchmarks of non-rigid point cloud matching.
  arXiv  Detail & Related papers  (2023-11-08T06:52:57Z)
• Explorable Mesh Deformation Subspaces from Unstructured Generative Models [53.23510438769862]
  Deep generative models of 3D shapes often feature continuous latent spaces that can be used to explore potential variations. We present a method to explore variations among a given set of landmark shapes by constructing a mapping from an easily-navigable 2D exploration space to a subspace of a pre-trained generative model.
  arXiv  Detail & Related papers  (2023-10-11T18:53:57Z)
• PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation [59.70430570779819]
  We introduce a data-driven shape completion approach that focuses on completing geometric details of missing regions of 3D shapes. Our key insight is to copy and deform patches from the partial input to complete missing regions. We leverage repeating patterns by retrieving patches from the partial input, and learn global structural priors by using a neural network to guide the retrieval and deformation steps.
  arXiv  Detail & Related papers  (2022-07-24T18:59:09Z)
• IDEA-Net: Dynamic 3D Point Cloud Interpolation via Deep Embedding Alignment [58.8330387551499]
  We formulate the problem as estimation of point-wise trajectories (i.e., smooth curves) We propose IDEA-Net, an end-to-end deep learning framework, which disentangles the problem under the assistance of the explicitly learned temporal consistency. We demonstrate the effectiveness of our method on various point cloud sequences and observe large improvement over state-of-the-art methods both quantitatively and visually.
  arXiv  Detail & Related papers  (2022-03-22T10:14:08Z)
• Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos [63.16888987770885]
  This paper addresses the challenge of reconstructing an animatable human model from a multi-view video. We introduce a pose-driven deformation field based on the linear blend skinning algorithm. We show that our approach significantly outperforms recent human modeling methods.
  arXiv  Detail & Related papers  (2022-03-15T17:56:59Z)
• Implicit field supervision for robust non-rigid shape matching [29.7672368261038]
  Establishing a correspondence between two non-rigidly deforming shapes is one of the most fundamental problems in visual computing. We introduce an approach based on auto-decoder framework, that learns a continuous shape-wise deformation field over a fixed template. Our method is remarkably robust in the presence of strong artefacts and can be generalised to arbitrary shape categories.
  arXiv  Detail & Related papers  (2022-03-15T07:22:52Z)
• EditVAE: Unsupervised Part-Aware Controllable 3D Point Cloud Shape Generation [19.817166425038753]
  This paper tackles the problem of parts-aware point cloud generation. A simple modification of the Variational Auto-Encoder yields a joint model of the point cloud itself. In addition to the flexibility afforded by our disentangled representation, the inductive bias introduced by our joint modelling approach yields the state-of-the-art experimental results on the ShapeNet dataset.
  arXiv  Detail & Related papers  (2021-10-13T12:38:01Z)
• Animatable Neural Radiance Fields for Human Body Modeling [54.41477114385557]
  This paper addresses the challenge of reconstructing an animatable human model from a multi-view video. We introduce neural blend weight fields to produce the deformation fields. Experiments show that our approach significantly outperforms recent human methods.
  arXiv  Detail & Related papers  (2021-05-06T17:58:13Z)
• SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes [117.76767853430243]
  We introduce SNARF, which combines the advantages of linear blend skinning for polygonal meshes with neural implicit surfaces. We propose a forward skinning model that finds all canonical correspondences of any deformed point using iterative root finding. Compared to state-of-the-art neural implicit representations, our approach generalizes better to unseen poses while preserving accuracy.
  arXiv  Detail & Related papers  (2021-04-08T17:54:59Z)
• Weakly-supervised 3D Shape Completion in the Wild [91.04095516680438]
  We address the problem of learning 3D complete shape from unaligned and real-world partial point clouds. We propose a weakly-supervised method to estimate both 3D canonical shape and 6-DoF pose for alignment, given multiple partial observations. Experiments on both synthetic and real data show that it is feasible and promising to learn 3D shape completion through large-scale data without shape and pose supervision.
  arXiv  Detail & Related papers  (2020-08-20T17:53:42Z)
• Learning non-rigid surface reconstruction from spatio-temporal image patches [0.0]
  We present a method to reconstruct a dense-temporal depth map of a deformable object from a video sequence. The estimation of depth is performed locally on non-temporal patches of the video, and the full depth video of entire shape is recovered by combining them together. We tested our method on both synthetic and Kinect data and experimentally observed that the reconstruction error is significantly lower than the one obtained using other approaches like conventional non-rigid structure.
  arXiv  Detail & Related papers  (2020-06-18T20:25:15Z)
• Dense Non-Rigid Structure from Motion: A Manifold Viewpoint [162.88686222340962]
  Non-Rigid Structure-from-Motion (NRSfM) problem aims to recover 3D geometry of a deforming object from its 2D feature correspondences across multiple frames. We show that our approach significantly improves accuracy, scalability, and robustness against noise.
  arXiv  Detail & Related papers  (2020-06-15T09:15:54Z)
• Neural Subdivision [58.97214948753937]
  This paper introduces Neural Subdivision, a novel framework for data-driven coarseto-fine geometry modeling. We optimize for the same set of network weights across all local mesh patches, thus providing an architecture that is not constrained to a specific input mesh, fixed genus, or category. We demonstrate that even when trained on a single high-resolution mesh our method generates reasonable subdivisions for novel shapes.
  arXiv  Detail & Related papers  (2020-05-04T20:03:21Z)
• Shape-Oriented Convolution Neural Network for Point Cloud Analysis [59.405388577930616]
  Point cloud is a principal data structure adopted for 3D geometric information encoding. Shape-oriented message passing scheme dubbed ShapeConv is proposed to focus on the representation learning of the underlying shape formed by each local neighboring point.
  arXiv  Detail & Related papers  (2020-04-20T16:11:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.