DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation

• URL: http://arxiv.org/abs/2412.15200v1
• Date: Thu, 19 Dec 2024 18:58:46 GMT
• Title: DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation
• Authors: Wang Zhao, Yan-Pei Cao, Jiale Xu, Yuejiang Dong, Ying Shan,
• Abstract summary: Inverse Procedural Content Generation aims to automatically find the best parameters under the input condition. We present DI-PCG, a novel and efficient method for Inverse PCG from general image conditions. With only 7.6M network parameters and 30 GPU hours to train, it demonstrates superior performance in recovering parameters accurately.
• Score: 39.08891847512135
• License:
• Abstract: Procedural Content Generation (PCG) is powerful in creating high-quality 3D contents, yet controlling it to produce desired shapes is difficult and often requires extensive parameter tuning. Inverse Procedural Content Generation aims to automatically find the best parameters under the input condition. However, existing sampling-based and neural network-based methods still suffer from numerous sample iterations or limited controllability. In this work, we present DI-PCG, a novel and efficient method for Inverse PCG from general image conditions. At its core is a lightweight diffusion transformer model, where PCG parameters are directly treated as the denoising target and the observed images as conditions to control parameter generation. DI-PCG is efficient and effective. With only 7.6M network parameters and 30 GPU hours to train, it demonstrates superior performance in recovering parameters accurately, and generalizing well to in-the-wild images. Quantitative and qualitative experiment results validate the effectiveness of DI-PCG in inverse PCG and image-to-3D generation tasks. DI-PCG offers a promising approach for efficient inverse PCG and represents a valuable exploration step towards a 3D generation path that models how to construct a 3D asset using parametric models.

Related papers

• Data-Driven, Parameterized Reduced-order Models for Predicting Distortion in Metal 3D Printing [39.58317527488534]
  This study introduces data-driven parameterized reduced-order models (ROMs) to predict distortion in Laser Powder Bed Fusion (LPBF) We propose a ROM framework that combines proper Orthogonal Decomposition (POD) with Gaussian Process Regression (GPR) and compare its performance against a deep-learning based parameterized graph convolutional autoencoder (GCA) The POD-GPR model demonstrates high accuracy, predicting distortions within $pm0.001mm$, and delivers a computational speed-up of approximately 1800x.
  arXiv  Detail & Related papers  (2024-12-05T19:47:25Z)
• Conditional LoRA Parameter Generation [18.34892473337235]
  We propose COND P-DIFF, a novel approach that demonstrates the feasibility of controllable high-performance parameter generation. Experimental results in both computer vision and natural language processing domains consistently demonstrate that COND P-DIFF can generate high-performance parameters conditioned on the given task. Our work paves the way for further exploration of condition-driven parameter generation, offering a promising direction for task-specific adaptation of neural networks.
  arXiv  Detail & Related papers  (2024-08-02T17:43:34Z)
• VividDreamer: Towards High-Fidelity and Efficient Text-to-3D Generation [69.68568248073747]
  We propose Pose-dependent Consistency Distillation Sampling (PCDS), a novel yet efficient objective for diffusion-based 3D generation tasks. PCDS builds the pose-dependent consistency function within diffusion trajectories, allowing to approximate true gradients through minimal sampling steps. For efficient generation, we propose a coarse-to-fine optimization strategy, which first utilizes 1-step PCDS to create the basic structure of 3D objects, and then gradually increases PCDS steps to generate fine-grained details.
  arXiv  Detail & Related papers  (2024-06-21T08:21:52Z)
• LP-3DGS: Learning to Prune 3D Gaussian Splatting [71.97762528812187]
  We propose learning-to-prune 3DGS, where a trainable binary mask is applied to the importance score that can find optimal pruning ratio automatically. Experiments have shown that LP-3DGS consistently produces a good balance that is both efficient and high quality.
  arXiv  Detail & Related papers  (2024-05-29T05:58:34Z)
• Revising Densification in Gaussian Splatting [23.037676471903215]
  We introduce a pixel-error driven formulation for density control in 3DGS, leveraging an auxiliary, per-pixel error function as the criterion for densification. Our approach leads to consistent quality improvements across a variety of benchmark scenes, without sacrificing the method's efficiency.
  arXiv  Detail & Related papers  (2024-04-09T08:20:37Z)
• GVP: Generative Volumetric Primitives [76.95231302205235]
  We present Generative Volumetric Primitives (GVP), the first pure 3D generative model that can sample and render 512-resolution images in real-time. GVP jointly models a number of primitives and their spatial information, both of which can be efficiently generated via a 2D convolutional network. Experiments on several datasets demonstrate superior efficiency and 3D consistency of GVP over the state-of-the-art.
  arXiv  Detail & Related papers  (2023-03-31T16:50:23Z)
• NeRF-GAN Distillation for Efficient 3D-Aware Generation with Convolutions [97.27105725738016]
  integration of Neural Radiance Fields (NeRFs) and generative models, such as Generative Adversarial Networks (GANs) has transformed 3D-aware generation from single-view images. We propose a simple and effective method, based on re-using the well-disentangled latent space of a pre-trained NeRF-GAN in a pose-conditioned convolutional network to directly generate 3D-consistent images corresponding to the underlying 3D representations.
  arXiv  Detail & Related papers  (2023-03-22T18:59:48Z)
• Training and Tuning Generative Neural Radiance Fields for Attribute-Conditional 3D-Aware Face Generation [66.21121745446345]
  We propose a conditional GNeRF model that integrates specific attribute labels as input, thus amplifying the controllability and disentanglement capabilities of 3D-aware generative models. Our approach builds upon a pre-trained 3D-aware face model, and we introduce a Training as Init and fidelity for Tuning (TRIOT) method to train a conditional normalized flow module. Our experiments substantiate the efficacy of our model, showcasing its ability to generate high-quality edits with enhanced view consistency.
  arXiv  Detail & Related papers  (2022-08-26T10:05:39Z)
• Point Cloud Generation with Continuous Conditioning [2.9238500578557303]
  We propose a novel generative adversarial network (GAN) setup that generates 3D point cloud shapes conditioned on a continuous parameter. In an exemplary application, we use this to guide the generative process to create a 3D object with a custom-fit shape.
  arXiv  Detail & Related papers  (2022-02-17T09:05:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.