MVGaussian: High-Fidelity text-to-3D Content Generation with Multi-View Guidance and Surface Densification

• URL: http://arxiv.org/abs/2409.06620v1
• Date: Tue, 10 Sep 2024 16:16:34 GMT
• Title: MVGaussian: High-Fidelity text-to-3D Content Generation with Multi-View Guidance and Surface Densification
• Authors: Phu Pham, Aradhya N. Mathur, Ojaswa Sharma, Aniket Bera,
• Abstract summary: This paper introduces a unified framework for text-to-3D content generation. Our approach utilizes multi-view guidance to iteratively form the structure of the 3D model. We also introduce a novel densification algorithm that aligns gaussians close to the surface.
• Score: 13.872254142378772
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The field of text-to-3D content generation has made significant progress in generating realistic 3D objects, with existing methodologies like Score Distillation Sampling (SDS) offering promising guidance. However, these methods often encounter the "Janus" problem-multi-face ambiguities due to imprecise guidance. Additionally, while recent advancements in 3D gaussian splitting have shown its efficacy in representing 3D volumes, optimization of this representation remains largely unexplored. This paper introduces a unified framework for text-to-3D content generation that addresses these critical gaps. Our approach utilizes multi-view guidance to iteratively form the structure of the 3D model, progressively enhancing detail and accuracy. We also introduce a novel densification algorithm that aligns gaussians close to the surface, optimizing the structural integrity and fidelity of the generated models. Extensive experiments validate our approach, demonstrating that it produces high-quality visual outputs with minimal time cost. Notably, our method achieves high-quality results within half an hour of training, offering a substantial efficiency gain over most existing methods, which require hours of training time to achieve comparable results.

Related papers

• GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation [75.39457097832113]
  This paper introduces a novel 3D generation framework, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs a Variational Autoencoder with multi-view posed RGB-D(epth)-N(ormal) renderings as input, using a unique latent space design that preserves 3D shape information. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single/multi-view image inputs.
  arXiv  Detail & Related papers  (2024-11-12T18:59:32Z)
• Semantic Score Distillation Sampling for Compositional Text-to-3D Generation [28.88237230872795]
  Generating high-quality 3D assets from textual descriptions remains a pivotal challenge in computer graphics and vision research. We introduce a novel SDS approach, designed to improve the expressiveness and accuracy of compositional text-to-3D generation. Our approach integrates new semantic embeddings that maintain consistency across different rendering views. By leveraging explicit semantic guidance, our method unlocks the compositional capabilities of existing pre-trained diffusion models.
  arXiv  Detail & Related papers  (2024-10-11T17:26:00Z)
• DreamMapping: High-Fidelity Text-to-3D Generation via Variational Distribution Mapping [20.7584503748821]
  Score Distillation Sampling (SDS) has emerged as a prevalent technique for text-to-3D generation, enabling 3D content creation by distilling view-dependent information from text-to-2D guidance. We conduct a thorough analysis of SDS and refine its formulation, finding that the core design is to model the distribution of rendered images. We introduce a novel strategy called Variational Distribution Mapping (VDM), which expedites the distribution modeling process by regarding the rendered images as instances of degradation from diffusion-based generation.
  arXiv  Detail & Related papers  (2024-09-08T14:04:48Z)
• VividDreamer: Towards High-Fidelity and Efficient Text-to-3D Generation [69.68568248073747]
  We propose Pose-dependent Consistency Distillation Sampling (PCDS), a novel yet efficient objective for diffusion-based 3D generation tasks. PCDS builds the pose-dependent consistency function within diffusion trajectories, allowing to approximate true gradients through minimal sampling steps. For efficient generation, we propose a coarse-to-fine optimization strategy, which first utilizes 1-step PCDS to create the basic structure of 3D objects, and then gradually increases PCDS steps to generate fine-grained details.
  arXiv  Detail & Related papers  (2024-06-21T08:21:52Z)
• Part-aware Shape Generation with Latent 3D Diffusion of Neural Voxel Fields [50.12118098874321]
  We introduce a latent 3D diffusion process for neural voxel fields, enabling generation at significantly higher resolutions. A part-aware shape decoder is introduced to integrate the part codes into the neural voxel fields, guiding the accurate part decomposition. The results demonstrate the superior generative capabilities of our proposed method in part-aware shape generation, outperforming existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-05-02T04:31:17Z)
• LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching [33.696757740830506]
  Recent advancements in text-to-3D generation have shown promise. Many methods base themselves on Score Distillation Sampling (SDS) We propose Interval Score Matching (ISM) to counteract over-smoothing.
  arXiv  Detail & Related papers  (2023-11-19T09:59:09Z)
• DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation [55.661467968178066]
  We propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks.
  arXiv  Detail & Related papers  (2023-09-28T17:55:05Z)
• Volumetric Semantically Consistent 3D Panoptic Mapping [77.13446499924977]
  We introduce an online 2D-to-3D semantic instance mapping algorithm aimed at generating semantic 3D maps suitable for autonomous agents in unstructured environments. It introduces novel ways of integrating semantic prediction confidence during mapping, producing semantic and instance-consistent 3D regions. The proposed method achieves accuracy superior to the state of the art on public large-scale datasets, improving on a number of widely used metrics.
  arXiv  Detail & Related papers  (2023-09-26T08:03:10Z)
• IT3D: Improved Text-to-3D Generation with Explicit View Synthesis [71.68595192524843]
  This study presents a novel strategy that leverages explicitly synthesized multi-view images to address these issues. Our approach involves the utilization of image-to-image pipelines, empowered by LDMs, to generate posed high-quality images. For the incorporated discriminator, the synthesized multi-view images are considered real data, while the renderings of the optimized 3D models function as fake data.
  arXiv  Detail & Related papers  (2023-08-22T14:39:17Z)
• HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance [19.252300247300145]
  This work proposes holistic sampling and smoothing approaches to achieve high-quality text-to-3D generation. We compute denoising scores in the text-to-image diffusion model's latent and image spaces. To generate high-quality renderings in a single-stage optimization, we propose regularization for the variance of z-coordinates along NeRF rays.
  arXiv  Detail & Related papers  (2023-05-30T05:56:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.