HINT: Learning Complete Human Neural Representations from Limited Viewpoints

• URL: http://arxiv.org/abs/2405.19712v1
• Date: Thu, 30 May 2024 05:43:09 GMT
• Title: HINT: Learning Complete Human Neural Representations from Limited Viewpoints
• Authors: Alessandro Sanvito, Andrea Ramazzina, Stefanie Walz, Mario Bijelic, Felix Heide,
• Abstract summary: We propose a NeRF-based algorithm able to learn a detailed and complete human model from limited viewing angles. As a result, our method can reconstruct complete humans even from a few viewing angles, increasing performance by more than 15% PSNR.
• Score: 69.76947323932107
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: No augmented application is possible without animated humanoid avatars. At the same time, generating human replicas from real-world monocular hand-held or robotic sensor setups is challenging due to the limited availability of views. Previous work showed the feasibility of virtual avatars but required the presence of 360 degree views of the targeted subject. To address this issue, we propose HINT, a NeRF-based algorithm able to learn a detailed and complete human model from limited viewing angles. We achieve this by introducing a symmetry prior, regularization constraints, and training cues from large human datasets. In particular, we introduce a sagittal plane symmetry prior to the appearance of the human, directly supervise the density function of the human model using explicit 3D body modeling, and leverage a co-learned human digitization network as additional supervision for the unseen angles. As a result, our method can reconstruct complete humans even from a few viewing angles, increasing performance by more than 15% PSNR compared to previous state-of-the-art algorithms.

Related papers

• WonderHuman: Hallucinating Unseen Parts in Dynamic 3D Human Reconstruction [51.22641018932625]
  We present WonderHuman to reconstruct dynamic human avatars from a monocular video for high-fidelity novel view synthesis. Our method achieves SOTA performance in producing photorealistic renderings from the given monocular video.
  arXiv  Detail & Related papers  (2025-02-03T04:43:41Z)
• IDOL: Instant Photorealistic 3D Human Creation from a Single Image [47.51400420928373]
  This work rethinks the task from the perspectives of dataset, model, and representation. We introduce a large-scale HUman-centric GEnerated dataset, HuGe100K, consisting of 100K diverse, photorealistic sets of human images. We develop a scalable feed-forward transformer model to predict a 3D human Gaussian representation in a uniform space from a given human image.
  arXiv  Detail & Related papers  (2024-12-19T15:43:05Z)
• Deformable 3D Gaussian Splatting for Animatable Human Avatars [50.61374254699761]
  We propose a fully explicit approach to construct a digital avatar from as little as a single monocular sequence. ParDy-Human constitutes an explicit model for realistic dynamic human avatars which requires significantly fewer training views and images. Our avatars learning is free of additional annotations such as Splat masks and can be trained with variable backgrounds while inferring full-resolution images efficiently even on consumer hardware.
  arXiv  Detail & Related papers  (2023-12-22T20:56:46Z)
• GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar [48.21353924040671]
  We propose to learn person-specific animatable avatars from images without assuming to have access to precise facial expression tracking. We learn a mapping from 3DMM facial expression parameters to the latent space of the generative model. With this scheme, we decouple 3D appearance reconstruction and animation control to achieve high fidelity in image synthesis.
  arXiv  Detail & Related papers  (2023-11-22T19:13:00Z)
• Cross-view and Cross-pose Completion for 3D Human Understanding [22.787947086152315]
  We propose a pre-training approach based on self-supervised learning that works on human-centric data using only images. We pre-train a model for body-centric tasks and one for hand-centric tasks. With a generic transformer architecture, these models outperform existing self-supervised pre-training methods on a wide set of human-centric downstream tasks.
  arXiv  Detail & Related papers  (2023-11-15T16:51:18Z)
• AvatarGen: A 3D Generative Model for Animatable Human Avatars [108.11137221845352]
  AvatarGen is an unsupervised generation of 3D-aware clothed humans with various appearances and controllable geometries. Our method can generate animatable 3D human avatars with high-quality appearance and geometry modeling. It is competent for many applications, e.g., single-view reconstruction, re-animation, and text-guided synthesis/editing.
  arXiv  Detail & Related papers  (2022-11-26T15:15:45Z)
• AvatarGen: a 3D Generative Model for Animatable Human Avatars [108.11137221845352]
  AvatarGen is the first method that enables not only non-rigid human generation with diverse appearance but also full control over poses and viewpoints. To model non-rigid dynamics, it introduces a deformation network to learn pose-dependent deformations in the canonical space. Our method can generate animatable human avatars with high-quality appearance and geometry modeling, significantly outperforming previous 3D GANs.
  arXiv  Detail & Related papers  (2022-08-01T01:27:02Z)
• LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies [78.17425779503047]
  We propose a novel neural implicit representation for the human body. It is fully differentiable and optimizable with disentangled shape and pose latent spaces. Our model can be trained and fine-tuned directly on non-watertight raw data with well-designed losses.
  arXiv  Detail & Related papers  (2021-11-30T04:10:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.