Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capture

• URL: http://arxiv.org/abs/2011.12866v2
• Date: Thu, 14 Oct 2021 13:41:15 GMT
• Title: Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capture
• Authors: Yue Li, Marc Habermann, Bernhard Thomaszewski, Stelian Coros, Thabo Beeler, Christian Theobalt
• Abstract summary: We show how integrating physics into the training process improves the learned cloth deformations and allows modeling clothing as a separate piece of geometry. Our approach leads to a significant improvement over current state-of-the-art methods and is thus a clear step towards realistic monocular capture of the entire deforming surface of a human clothed.
• Score: 84.73946704272113
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent monocular human performance capture approaches have shown compelling dense tracking results of the full body from a single RGB camera. However, existing methods either do not estimate clothing at all or model cloth deformation with simple geometric priors instead of taking into account the underlying physical principles. This leads to noticeable artifacts in their reconstructions, e.g. baked-in wrinkles, implausible deformations that seemingly defy gravity, and intersections between cloth and body. To address these problems, we propose a person-specific, learning-based method that integrates a simulation layer into the training process to provide for the first time physics supervision in the context of weakly supervised deep monocular human performance capture. We show how integrating physics into the training process improves the learned cloth deformations, allows modeling clothing as a separate piece of geometry, and largely reduces cloth-body intersections. Relying only on weak 2D multi-view supervision during training, our approach leads to a significant improvement over current state-of-the-art methods and is thus a clear step towards realistic monocular capture of the entire deforming surface of a clothed human.

Related papers

• PhyRecon: Physically Plausible Neural Scene Reconstruction [81.73129450090684]
  We introduce PHYRECON, the first approach to leverage both differentiable rendering and differentiable physics simulation to learn implicit surface representations. Central to this design is an efficient transformation between SDF-based implicit representations and explicit surface points. Our results also exhibit superior physical stability in physical simulators, with at least a 40% improvement across all datasets.
  arXiv  Detail & Related papers  (2024-04-25T15:06:58Z)
• Within the Dynamic Context: Inertia-aware 3D Human Modeling with Pose Sequence [47.16903508897047]
  In this study, we elucidate that variations in human appearance depend not only on the current frame's pose condition but also on past pose states. We introduce Dyco, a novel method utilizing the delta pose sequence representation for non-rigid deformations. In addition, our inertia-aware 3D human method can unprecedentedly simulate appearance changes caused by inertia at different velocities.
  arXiv  Detail & Related papers  (2024-03-28T06:05:14Z)
• CaPhy: Capturing Physical Properties for Animatable Human Avatars [44.95805736197971]
  CaPhy is a novel method for reconstructing animatable human avatars with realistic dynamic properties for clothing. We aim for capturing the geometric and physical properties of the clothing from real observations. We combine unsupervised training with physics-based losses and 3D-supervised training using scanned data to reconstruct a dynamic model of clothing.
  arXiv  Detail & Related papers  (2023-08-11T04:01:13Z)
• Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video [31.96672354594643]
  We focus on the task of estimating a physically plausible articulated human motion from monocular video. Existing approaches that do not consider physics often produce temporally inconsistent output with motion artifacts. We show that our approach achieves competitive results with respect to existing physics-based methods on the Human3.6M benchmark.
  arXiv  Detail & Related papers  (2022-05-24T18:02:49Z)
• SNUG: Self-Supervised Neural Dynamic Garments [14.83072352654608]
  We present a self-supervised method to learn dynamic 3D deformations of garments worn by parametric human bodies. This allows us to learn models for interactive garments, including dynamic deformations and fine wrinkles, with two orders of magnitude speed up in training time.
  arXiv  Detail & Related papers  (2022-04-05T13:50:21Z)
• LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies [78.17425779503047]
  We propose a novel neural implicit representation for the human body. It is fully differentiable and optimizable with disentangled shape and pose latent spaces. Our model can be trained and fine-tuned directly on non-watertight raw data with well-designed losses.
  arXiv  Detail & Related papers  (2021-11-30T04:10:57Z)
• MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images [60.56518548286836]
  To generate realistic cloth deformations from novel input poses, watertight meshes or dense full-body scans are usually needed as inputs. We propose an approach that can quickly generate realistic clothed human avatars, represented as controllable neural SDFs, given only monocular depth images.
  arXiv  Detail & Related papers  (2021-06-22T17:30:12Z)
• SCALE: Modeling Clothed Humans with a Surface Codec of Articulated Local Elements [62.652588951757764]
  Learning to model and reconstruct humans in clothing is challenging due to articulation, non-rigid deformation, and varying clothing types and topologies. Recent work uses neural networks to parameterize local surface elements. We present three key innovations: First, we deform surface elements based on a human body model. Second, we address the limitations of existing neural surface elements by regressing local geometry from local features.
  arXiv  Detail & Related papers  (2021-04-15T17:59:39Z)
• Style and Pose Control for Image Synthesis of Humans from a Single Monocular View [78.6284090004218]
  StylePoseGAN is a non-controllable generator to accept conditioning of pose and appearance separately. Our network can be trained in a fully supervised way with human images to disentangle pose, appearance and body parts. StylePoseGAN achieves state-of-the-art image generation fidelity on common perceptual metrics.
  arXiv  Detail & Related papers  (2021-02-22T18:50:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.