HuGeDiff: 3D Human Generation via Diffusion with Gaussian Splatting

• URL: http://arxiv.org/abs/2506.04351v1
• Date: Wed, 04 Jun 2025 18:11:23 GMT
• Title: HuGeDiff: 3D Human Generation via Diffusion with Gaussian Splatting
• Authors: Maksym Ivashechkin, Oscar Mendez, Richard Bowden,
• Abstract summary: Current methods struggle with fine detail, accurate rendering of hands and faces, human realism, and controlability over appearance.<n>We present a weakly supervised pipeline that tries to address these challenges.<n>We demonstrate orders-of-magnitude speed-ups in 3D human generation compared to the state-of-the-art approaches.
• Score: 33.9893684177763
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: 3D human generation is an important problem with a wide range of applications in computer vision and graphics. Despite recent progress in generative AI such as diffusion models or rendering methods like Neural Radiance Fields or Gaussian Splatting, controlling the generation of accurate 3D humans from text prompts remains an open challenge. Current methods struggle with fine detail, accurate rendering of hands and faces, human realism, and controlability over appearance. The lack of diversity, realism, and annotation in human image data also remains a challenge, hindering the development of a foundational 3D human model. We present a weakly supervised pipeline that tries to address these challenges. In the first step, we generate a photorealistic human image dataset with controllable attributes such as appearance, race, gender, etc using a state-of-the-art image diffusion model. Next, we propose an efficient mapping approach from image features to 3D point clouds using a transformer-based architecture. Finally, we close the loop by training a point-cloud diffusion model that is conditioned on the same text prompts used to generate the original samples. We demonstrate orders-of-magnitude speed-ups in 3D human generation compared to the state-of-the-art approaches, along with significantly improved text-prompt alignment, realism, and rendering quality. We will make the code and dataset available.

Related papers

• 3D Scene Generation: A Survey [41.202497008985425]
  3D scene generation seeks to synthesize spatially structured, semantically meaningful, and photorealistic environments for applications such as immersive media, robotics, autonomous driving, and embodied AI.<n>This review organizes recent advances in 3D scene generation and highlights promising directions at the intersection of generative AI, 3D vision, and embodied intelligence.
  arXiv  Detail & Related papers  (2025-05-08T17:59:54Z)
• SIGMAN:Scaling 3D Human Gaussian Generation with Millions of Assets [72.26350984924129]
  We propose a latent space generation paradigm for 3D human digitization.<n>We transform the ill-posed low-to-high-dimensional mapping problem into a learnable distribution shift.<n>We employ the multi-view optimization approach combined with synthetic data to construct the HGS-1M dataset.
  arXiv  Detail & Related papers  (2025-04-09T15:38:18Z)
• UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling [71.87807614875497]
  We propose UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures. We collect and process a new dataset of human motion, which includes multi-view images, scanned models, parametric model registration, and corresponding texture maps. Experimental results demonstrate that our method achieves state-of-the-art synthesis of novel view and novel pose.
  arXiv  Detail & Related papers  (2024-03-18T09:03:56Z)
• 3D Human Reconstruction in the Wild with Synthetic Data Using Generative Models [52.96248836582542]
  We propose an effective approach based on recent diffusion models, termed HumanWild, which can effortlessly generate human images and corresponding 3D mesh annotations. By exclusively employing generative models, we generate large-scale in-the-wild human images and high-quality annotations, eliminating the need for real-world data collection.
  arXiv  Detail & Related papers  (2024-03-17T06:31:16Z)
• MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D Human Generation [45.88714821939144]
  We present an alternative scheme named MVHuman to generate human radiance fields from text guidance. Our core is a multi-view sampling strategy to tailor the denoising processes of the pre-trained network for generating consistent multi-view images.
  arXiv  Detail & Related papers  (2023-12-15T11:56:26Z)
• HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion [53.1558345421646]
  We propose HumanRef, a 3D human generation framework from a single-view input. To ensure the generated 3D model is photorealistic and consistent with the input image, HumanRef introduces a novel method called reference-guided score distillation sampling. Experimental results demonstrate that HumanRef outperforms state-of-the-art methods in generating 3D clothed humans.
  arXiv  Detail & Related papers  (2023-11-28T17:06:28Z)
• InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation [61.62346472443454]
  InceptionHuman is a prompt-to-NeRF framework that allows easy control via a combination of prompts in different modalities to generate photorealistic 3D humans. InceptionHuman achieves consistent 3D human generation within a progressively refined NeRF space.
  arXiv  Detail & Related papers  (2023-11-27T15:49:41Z)
• HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation [41.82589219009301]
  We propose HumanNorm, a novel approach for high-quality and realistic 3D human generation. The main idea is to enhance the model's 2D perception of 3D geometry by learning a normal-adapted diffusion model and a normal-aligned diffusion model. HumanNorm outperforms existing text-to-3D methods in both geometry and texture quality.
  arXiv  Detail & Related papers  (2023-10-02T17:59:17Z)
• DRaCoN -- Differentiable Rasterization Conditioned Neural Radiance Fields for Articulated Avatars [92.37436369781692]
  We present DRaCoN, a framework for learning full-body volumetric avatars. It exploits the advantages of both the 2D and 3D neural rendering techniques. Experiments on the challenging ZJU-MoCap and Human3.6M datasets indicate that DRaCoN outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T17:59:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.