HFNeRF: Learning Human Biomechanic Features with Neural Radiance Fields

• URL: http://arxiv.org/abs/2404.06152v1
• Date: Tue, 9 Apr 2024 09:23:04 GMT
• Title: HFNeRF: Learning Human Biomechanic Features with Neural Radiance Fields
• Authors: Arnab Dey, Di Yang, Antitza Dantcheva, Jean Martinet,
• Abstract summary: We introduce HFNeRF: a novel generalizable human feature NeRF aimed at generating human biomechanic features. We leverage 2D pre-trained foundation models toward learning human features in 3D using neural rendering. We evaluate HFNeRF in the skeleton estimation task by predicting heatmaps as features.
• Score: 11.961164199224351
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In recent advancements in novel view synthesis, generalizable Neural Radiance Fields (NeRF) based methods applied to human subjects have shown remarkable results in generating novel views from few images. However, this generalization ability cannot capture the underlying structural features of the skeleton shared across all instances. Building upon this, we introduce HFNeRF: a novel generalizable human feature NeRF aimed at generating human biomechanic features using a pre-trained image encoder. While previous human NeRF methods have shown promising results in the generation of photorealistic virtual avatars, such methods lack underlying human structure or biomechanic features such as skeleton or joint information that are crucial for downstream applications including Augmented Reality (AR)/Virtual Reality (VR). HFNeRF leverages 2D pre-trained foundation models toward learning human features in 3D using neural rendering, and then volume rendering towards generating 2D feature maps. We evaluate HFNeRF in the skeleton estimation task by predicting heatmaps as features. The proposed method is fully differentiable, allowing to successfully learn color, geometry, and human skeleton in a simultaneous manner. This paper presents preliminary results of HFNeRF, illustrating its potential in generating realistic virtual avatars with biomechanic features using NeRF.

Related papers

• GHNeRF: Learning Generalizable Human Features with Efficient Neural Radiance Fields [12.958200963257381]
  We introduce GHNeRF, designed to address limitations by learning 2D/3D joint locations of human subjects with NeRF representation. GHNeRF uses a pre-trained 2D encoder streamlined to extract essential human features from 2D images, which are then incorporated into the NeRF framework. Our results show that GHNeRF can achieve state-of-the-art results in near real-time.
  arXiv  Detail & Related papers  (2024-04-09T12:11:25Z)
• UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling [71.87807614875497]
  We propose UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures. We collect and process a new dataset of human motion, which includes multi-view images, scanned models, parametric model registration, and corresponding texture maps. Experimental results demonstrate that our method achieves state-of-the-art synthesis of novel view and novel pose.
  arXiv  Detail & Related papers  (2024-03-18T09:03:56Z)
• InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation [61.62346472443454]
  InceptionHuman is a prompt-to-NeRF framework that allows easy control via a combination of prompts in different modalities to generate photorealistic 3D humans. InceptionHuman achieves consistent 3D human generation within a progressively refined NeRF space.
  arXiv  Detail & Related papers  (2023-11-27T15:49:41Z)
• BeyondPixels: A Comprehensive Review of the Evolution of Neural Radiance Fields [1.1531932979578041]
  NeRF, short for Neural Radiance Fields, is a recent innovation that uses AI algorithms to create 3D objects from 2D images. This survey reviews recent advances in NeRF and categorizes them according to their architectural designs.
  arXiv  Detail & Related papers  (2023-06-05T16:10:21Z)
• GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images [79.39247661907397]
  We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
  arXiv  Detail & Related papers  (2023-03-24T03:32:02Z)
• SHERF: Generalizable Human NeRF from a Single Image [59.10589479808622]
  SHERF is the first generalizable Human NeRF model for recovering animatable 3D humans from a single input image. We propose a bank of 3D-aware hierarchical features, including global, point-level, and pixel-aligned features, to facilitate informative encoding.
  arXiv  Detail & Related papers  (2023-03-22T17:59:12Z)
• HDHumans: A Hybrid Approach for High-fidelity Digital Humans [107.19426606778808]
  HDHumans is the first method for HD human character synthesis that jointly produces an accurate and temporally coherent 3D deforming surface. Our method is carefully designed to achieve a synergy between classical surface deformation and neural radiance fields (NeRF)
  arXiv  Detail & Related papers  (2022-10-21T14:42:11Z)
• DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars [92.37436369781692]
  We present DRaCoN, a framework for learning full-body volumetric avatars. It exploits the advantages of both the 2D and 3D neural rendering techniques. Experiments on the challenging ZJU-MoCap and Human3.6M datasets indicate that DRaCoN outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T17:59:15Z)
• NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras [17.18904717379273]
  4D reconstruction and rendering of human activities is critical for immersive VR/AR experience. Recent advances still fail to recover fine geometry and texture results with the level of detail present in the input images from sparse multi-view RGB cameras. We propose a real-time neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of human activities in arbitrary novel views.
  arXiv  Detail & Related papers  (2021-03-13T12:03:38Z)
• Learning Compositional Radiance Fields of Dynamic Human Heads [13.272666180264485]
  We propose a novel compositional 3D representation that combines the best of previous methods to produce both higher-resolution and faster results. Differentiable volume rendering is employed to compute photo-realistic novel views of the human head and upper body. Our approach achieves state-of-the-art results for synthesizing novel views of dynamic human heads and the upper body.
  arXiv  Detail & Related papers  (2020-12-17T22:19:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.