HFGaussian: Learning Generalizable Gaussian Human with Integrated Human Features

• URL: http://arxiv.org/abs/2411.03086v1
• Date: Tue, 05 Nov 2024 13:31:04 GMT
• Title: HFGaussian: Learning Generalizable Gaussian Human with Integrated Human Features
• Authors: Arnab Dey, Cheng-You Lu, Andrew I. Comport, Srinath Sridhar, Chin-Teng Lin, Jean Martinet,
• Abstract summary: We present a novel approach called HFGaussian that can estimate novel views and human features, such as the 3D skeleton, 3D key points, and dense pose, from sparse input images in real time at 25 FPS. We thoroughly evaluate our HFGaussian method against the latest state-of-the-art techniques in human Gaussian splatting and pose estimation, demonstrating its real-time, state-of-the-art performance.
• Score: 23.321087432786605
• License:
• Abstract: Recent advancements in radiance field rendering show promising results in 3D scene representation, where Gaussian splatting-based techniques emerge as state-of-the-art due to their quality and efficiency. Gaussian splatting is widely used for various applications, including 3D human representation. However, previous 3D Gaussian splatting methods either use parametric body models as additional information or fail to provide any underlying structure, like human biomechanical features, which are essential for different applications. In this paper, we present a novel approach called HFGaussian that can estimate novel views and human features, such as the 3D skeleton, 3D key points, and dense pose, from sparse input images in real time at 25 FPS. The proposed method leverages generalizable Gaussian splatting technique to represent the human subject and its associated features, enabling efficient and generalizable reconstruction. By incorporating a pose regression network and the feature splatting technique with Gaussian splatting, HFGaussian demonstrates improved capabilities over existing 3D human methods, showcasing the potential of 3D human representations with integrated biomechanics. We thoroughly evaluate our HFGaussian method against the latest state-of-the-art techniques in human Gaussian splatting and pose estimation, demonstrating its real-time, state-of-the-art performance.

Related papers

• GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views [67.34073368933814]
  We propose a generalizable Gaussian Splatting approach for high-resolution image rendering under a sparse-view camera setting. We train our Gaussian parameter regression module on human-only data or human-scene data, jointly with a depth estimation module to lift 2D parameter maps to 3D space. Experiments on several datasets demonstrate that our method outperforms state-of-the-art methods while achieving an exceeding rendering speed.
  arXiv  Detail & Related papers  (2024-11-18T08:18:44Z)
• EVA-Gaussian: 3D Gaussian-based Real-time Human Novel View Synthesis under Diverse Camera Settings [11.248908608011941]
  EVA-Gaussian is a real-time pipeline for 3D human novel view synthesis across diverse camera settings. We introduce an Efficient cross-View Attention (EVA) module to accurately estimate the position of each 3D Gaussian from the source images. We incorporate a powerful anchor loss function for both 3D Gaussian attributes and human face landmarks.
  arXiv  Detail & Related papers  (2024-10-02T11:23:08Z)
• Generalizable Human Gaussians for Sparse View Synthesis [48.47812125126829]
  This paper introduces a new method to learn generalizable human Gaussians that allows photorealistic and accurate view-rendering of a new human subject from a limited set of sparse views. A pivotal innovation of our approach involves reformulating the learning of 3D Gaussian parameters into a regression process defined on the 2D UV space of a human template. Our method outperforms recent methods on both within-dataset generalization as well as cross-dataset generalization settings.
  arXiv  Detail & Related papers  (2024-07-17T17:56:30Z)
• Effective Rank Analysis and Regularization for Enhanced 3D Gaussian Splatting [33.01987451251659]
  3D Gaussian Splatting (3DGS) has emerged as a promising technique capable of real-time rendering with high-quality 3D reconstruction. Despite its potential, 3DGS encounters challenges, including needle-like artifacts, suboptimal geometries, and inaccurate normals. We introduce effective rank as a regularization, which constrains the structure of the Gaussians.
  arXiv  Detail & Related papers  (2024-06-17T15:51:59Z)
• Generalizable Human Gaussians from Single-View Image [52.100234836129786]
  We introduce a single-view generalizable Human Gaussian Model (HGM) Our approach uses a ControlNet to refine rendered back-view images from coarse predicted human Gaussians. To mitigate the potential generation of unrealistic human poses and shapes, we incorporate human priors from the SMPL-X model as a dual branch.
  arXiv  Detail & Related papers  (2024-06-10T06:38:11Z)
• Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenes [50.92217884840301]
  Gaussian Opacity Fields (GOF) is a novel approach for efficient, high-quality, and adaptive surface reconstruction in scenes. GOF is derived from ray-tracing-based volume rendering of 3D Gaussians. GOF surpasses existing 3DGS-based methods in surface reconstruction and novel view synthesis.
  arXiv  Detail & Related papers  (2024-04-16T17:57:19Z)
• Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing [53.05069432989608]
  We present a novel framework for generating 3D human heads with remarkable flexibility. Our method facilitates the creation of diverse and realistic 3D human heads with fine-grained editing over facial features and expressions.
  arXiv  Detail & Related papers  (2023-12-05T19:05:58Z)
• GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis [70.24111297192057]
  We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner. The proposed method enables 2K-resolution rendering under a sparse-view camera setting.
  arXiv  Detail & Related papers  (2023-12-04T18:59:55Z)
• Animatable 3D Gaussians for High-fidelity Synthesis of Human Motions [37.50707388577952]
  We present a novel animatable 3D Gaussian model for rendering high-fidelity free-view human motions in real time. Compared to existing NeRF-based methods, the model owns better capability in high-frequency details without the jittering problem across video frames.
  arXiv  Detail & Related papers  (2023-11-22T14:00:23Z)
• SplatArmor: Articulated Gaussian splatting for animatable humans from monocular RGB videos [15.74530749823217]
  We propose SplatArmor, a novel approach for recovering detailed and animatable human models by armoring' a parameterized body model with 3D Gaussians. Our approach represents the human as a set of 3D Gaussians within a canonical space, whose articulation is defined by extending the skinning of the underlying SMPL geometry. We show compelling results on the ZJU MoCap and People Snapshot datasets, which underscore the effectiveness of our method for controllable human synthesis.
  arXiv  Detail & Related papers  (2023-11-17T18:47:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.