SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity

• URL: http://arxiv.org/abs/2401.00604v2
• Date: Fri, 29 Mar 2024 18:33:30 GMT
• Title: SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity
• Authors: Peihao Wang, Zhiwen Fan, Dejia Xu, Dilin Wang, Sreyas Mohan, Forrest Iandola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, Vikas Chandra,
• Abstract summary: We show that gradient estimation in score distillation is inherent to high variance. We propose a more general solution to reduce variance for score distillation, termed Stein Score Distillation (SSD) We demonstrate that SteinDreamer achieves faster convergence than existing methods due to more stable gradient updates.
• Score: 70.32101198891465
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Score distillation has emerged as one of the most prevalent approaches for text-to-3D asset synthesis. Essentially, score distillation updates 3D parameters by lifting and back-propagating scores averaged over different views. In this paper, we reveal that the gradient estimation in score distillation is inherent to high variance. Through the lens of variance reduction, the effectiveness of SDS and VSD can be interpreted as applications of various control variates to the Monte Carlo estimator of the distilled score. Motivated by this rethinking and based on Stein's identity, we propose a more general solution to reduce variance for score distillation, termed Stein Score Distillation (SSD). SSD incorporates control variates constructed by Stein identity, allowing for arbitrary baseline functions. This enables us to include flexible guidance priors and network architectures to explicitly optimize for variance reduction. In our experiments, the overall pipeline, dubbed SteinDreamer, is implemented by instantiating the control variate with a monocular depth estimator. The results suggest that SSD can effectively reduce the distillation variance and consistently improve visual quality for both object- and scene-level generation. Moreover, we demonstrate that SteinDreamer achieves faster convergence than existing methods due to more stable gradient updates.

Related papers

• VividDreamer: Invariant Score Distillation For Hyper-Realistic Text-to-3D Generation [33.05759961083337]
  This paper presents Invariant Score Distillation (ISD), a novel method for high-fidelity text-to-3D generation. ISD aims to tackle the over-saturation and over-smoothing problems in Score Distillation Sampling (SDS)
  arXiv  Detail & Related papers  (2024-07-13T09:33:16Z)
• VividDreamer: Towards High-Fidelity and Efficient Text-to-3D Generation [69.68568248073747]
  We propose Pose-dependent Consistency Distillation Sampling (PCDS), a novel yet efficient objective for diffusion-based 3D generation tasks. PCDS builds the pose-dependent consistency function within diffusion trajectories, allowing to approximate true gradients through minimal sampling steps. For efficient generation, we propose a coarse-to-fine optimization strategy, which first utilizes 1-step PCDS to create the basic structure of 3D objects, and then gradually increases PCDS steps to generate fine-grained details.
  arXiv  Detail & Related papers  (2024-06-21T08:21:52Z)
• Taming Mode Collapse in Score Distillation for Text-to-3D Generation [70.32101198891465]
  "Janus" artifact is a problem in text-to-3D generation where the generated objects fake each view with multiple front faces. We propose a new update rule for 3D score distillation, dubbed Entropic Score Distillation ( ESD) Although embarrassingly straightforward, our experiments successfully demonstrate that ESD can be an effective treatment for Janus artifacts in score distillation.
  arXiv  Detail & Related papers  (2023-12-31T22:47:06Z)
• Adversarial Score Distillation: When score distillation meets GAN [3.2794321281011394]
  We decipher existing score distillation with the Wasserstein Generative Adversarial Network (WGAN) paradigm. With the WGAN paradigm, we find that existing score distillation either employs a fixed sub-optimal discriminator or conducts incomplete discriminator optimization. We propose the Adversarial Score Distillation (ASD), which maintains an optimizable discriminator and updates it using the complete optimization objective.
  arXiv  Detail & Related papers  (2023-12-01T17:20:47Z)
• Text-to-3D with Classifier Score Distillation [80.14832887529259]
  Classifier-free guidance is considered an auxiliary trick rather than the most essential. We name this method Score Distillation (CSD), which can be interpreted as using an implicit classification model for generation. We validate the effectiveness of CSD across a variety of text-to-3D tasks including shape generation, texture synthesis, and shape editing.
  arXiv  Detail & Related papers  (2023-10-30T10:25:40Z)
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation [48.59711140119368]
  We present variational score distillation (VSD) to explain and address issues in text-to-3D generation. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., $512times512$) and high-fidelity NeRF with rich structure and complex effects.
  arXiv  Detail & Related papers  (2023-05-25T16:19:18Z)
• StereoDistill: Pick the Cream from LiDAR for Distilling Stereo-based 3D Object Detection [93.10989714186788]
  We propose a cross-modal distillation method named StereoDistill to narrow the gap between the stereo and LiDAR-based approaches. Key designs of StereoDistill are: the X-component Guided Distillation(XGD) for regression and the Cross-anchor Logit Distillation(CLD) for classification.
  arXiv  Detail & Related papers  (2023-01-04T13:38:48Z)
• Sliced Kernelized Stein Discrepancy [17.159499204595527]
  Kernelized Stein discrepancy (KSD) is extensively used in goodness-of-fit tests and model learning. We propose the sliced Stein discrepancy and its scalable and kernelized variants, which employ kernel-based test functions defined on the optimal one-dimensional projections. For model learning, we show its advantages over existing Stein discrepancy baselines by training independent component analysis models with different discrepancies.
  arXiv  Detail & Related papers  (2020-06-30T04:58:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.