MVD-HuGaS: Human Gaussians from a Single Image via 3D Human Multi-view Diffusion Prior

• URL: http://arxiv.org/abs/2503.08218v1
• Date: Tue, 11 Mar 2025 09:37:15 GMT
• Title: MVD-HuGaS: Human Gaussians from a Single Image via 3D Human Multi-view Diffusion Prior
• Authors: Kaiqiang Xiong, Ying Feng, Qi Zhang, Jianbo Jiao, Yang Zhao, Zhihao Liang, Huachen Gao, Ronggang Wang,
• Abstract summary: We present emphMVD-HuGaS, enabling free-view 3D human rendering from a single image via a multi-view human diffusion model.<n>Experiments on Thuman2.0 and 2K2K datasets show that the proposed MVD-HuGaS achieves state-of-the-art performance on single-view 3D human rendering.
• Score: 35.704591162502375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D human reconstruction from a single image is a challenging problem and has been exclusively studied in the literature. Recently, some methods have resorted to diffusion models for guidance, optimizing a 3D representation via Score Distillation Sampling(SDS) or generating one back-view image for facilitating reconstruction. However, these methods tend to produce unsatisfactory artifacts (\textit{e.g.} flattened human structure or over-smoothing results caused by inconsistent priors from multiple views) and struggle with real-world generalization in the wild. In this work, we present \emph{MVD-HuGaS}, enabling free-view 3D human rendering from a single image via a multi-view human diffusion model. We first generate multi-view images from the single reference image with an enhanced multi-view diffusion model, which is well fine-tuned on high-quality 3D human datasets to incorporate 3D geometry priors and human structure priors. To infer accurate camera poses from the sparse generated multi-view images for reconstruction, an alignment module is introduced to facilitate joint optimization of 3D Gaussians and camera poses. Furthermore, we propose a depth-based Facial Distortion Mitigation module to refine the generated facial regions, thereby improving the overall fidelity of the reconstruction.Finally, leveraging the refined multi-view images, along with their accurate camera poses, MVD-HuGaS optimizes the 3D Gaussians of the target human for high-fidelity free-view renderings. Extensive experiments on Thuman2.0 and 2K2K datasets show that the proposed MVD-HuGaS achieves state-of-the-art performance on single-view 3D human rendering.

Related papers

• SIGMAN:Scaling 3D Human Gaussian Generation with Millions of Assets [72.26350984924129]
  We propose a latent space generation paradigm for 3D human digitization. We transform the ill-posed low-to-high-dimensional mapping problem into a learnable distribution shift. We employ the multi-view optimization approach combined with synthetic data to construct the HGS-1M dataset.
  arXiv  Detail & Related papers  (2025-04-09T15:38:18Z)
• HumanDreamer-X: Photorealistic Single-image Human Avatars Reconstruction via Gaussian Restoration [29.03216532351979]
  We introduce textbfHumanDreamer-X, a novel framework that integrates multi-view human generation and reconstruction into a unified pipeline. In this framework, 3D Gaussian Splatting serves as an explicit 3D representation to provide initial geometry and appearance priority. We also propose an attention modulation strategy that effectively enhances geometric details identity consistency across multi-view.
  arXiv  Detail & Related papers  (2025-04-04T15:35:14Z)
• CDI3D: Cross-guided Dense-view Interpolation for 3D Reconstruction [25.468907201804093]
  Large Reconstruction Models (LRMs) have shown great promise in leveraging multi-view images generated by 2D diffusion models to extract 3D content.<n>However, 2D diffusion models often struggle to produce dense images with strong multi-view consistency.<n>We present CDI3D, a feed-forward framework designed for efficient, high-quality image-to-3D generation with view.
  arXiv  Detail & Related papers  (2025-03-11T03:08:43Z)
• HuGDiffusion: Generalizable Single-Image Human Rendering via 3D Gaussian Diffusion [50.02316409061741]
  HuGDiffusion is a learning pipeline to achieve novel view synthesis (NVS) of human characters from single-view input images. We aim to generate the set of 3DGS attributes via a diffusion-based framework conditioned on human priors extracted from a single image. Our HuGDiffusion shows significant performance improvements over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-25T01:00:33Z)
• DSplats: 3D Generation by Denoising Splats-Based Multiview Diffusion Models [67.50989119438508]
  We introduce DSplats, a novel method that directly denoises multiview images using Gaussian-based Reconstructors to produce realistic 3D assets.<n>Our experiments demonstrate that DSplats not only produces high-quality, spatially consistent outputs, but also sets a new standard in single-image to 3D reconstruction.
  arXiv  Detail & Related papers  (2024-12-11T07:32:17Z)
• MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware Diffusion and Iterative Refinement [23.707586182294932]
  Existing works in single-image human reconstruction suffer from weak generalizability due to insufficient training data or 3D inconsistencies for a lack of comprehensive multi-view knowledge. We introduce MagicMan, a human-specific multi-view diffusion model designed to generate high-quality novel view images from a single reference image.
  arXiv  Detail & Related papers  (2024-08-26T12:10:52Z)
• MVGamba: Unify 3D Content Generation as State Space Sequence Modeling [150.80564081817786]
  We introduce MVGamba, a general and lightweight Gaussian reconstruction model featuring a multi-view Gaussian reconstructor. With off-the-detail multi-view diffusion models integrated, MVGamba unifies 3D generation tasks from a single image, sparse images, or text prompts. Experiments demonstrate that MVGamba outperforms state-of-the-art baselines in all 3D content generation scenarios with approximately only $0.1times$ of the model size.
  arXiv  Detail & Related papers  (2024-06-10T15:26:48Z)
• 3D Human Reconstruction in the Wild with Synthetic Data Using Generative Models [52.96248836582542]
  We propose an effective approach based on recent diffusion models, termed HumanWild, which can effortlessly generate human images and corresponding 3D mesh annotations. By exclusively employing generative models, we generate large-scale in-the-wild human images and high-quality annotations, eliminating the need for real-world data collection.
  arXiv  Detail & Related papers  (2024-03-17T06:31:16Z)
• Template-Free Single-View 3D Human Digitalization with Diffusion-Guided LRM [29.13412037370585]
  We present Human-LRM, a diffusion-guided feed-forward model that predicts the implicit field of a human from a single image. Our method is able to capture human without any template prior, e.g., SMPL, and effectively enhance occluded parts with rich and realistic details.
  arXiv  Detail & Related papers  (2024-01-22T18:08:22Z)
• MVHuman: Tailoring 2D Diffusion with Multi-view Sampling For Realistic 3D Human Generation [45.88714821939144]
  We present an alternative scheme named MVHuman to generate human radiance fields from text guidance. Our core is a multi-view sampling strategy to tailor the denoising processes of the pre-trained network for generating consistent multi-view images.
  arXiv  Detail & Related papers  (2023-12-15T11:56:26Z)
• Wonder3D: Single Image to 3D using Cross-Domain Diffusion [105.16622018766236]
  Wonder3D is a novel method for efficiently generating high-fidelity textured meshes from single-view images. To holistically improve the quality, consistency, and efficiency of image-to-3D tasks, we propose a cross-domain diffusion model.
  arXiv  Detail & Related papers  (2023-10-23T15:02:23Z)
• NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors [24.05480789681139]
  We propose NeRDi, a single-view NeRF synthesis framework with general image priors from 2D diffusion models. We leverage off-the-shelf vision-language models and introduce a two-section language guidance as conditioning inputs to the diffusion model. We also demonstrate our generalizability in zero-shot NeRF synthesis for in-the-wild images.
  arXiv  Detail & Related papers  (2022-12-06T19:00:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.