Adaptive Guidance: Training-free Acceleration of Conditional Diffusion
Models
- URL: http://arxiv.org/abs/2312.12487v1
- Date: Tue, 19 Dec 2023 17:08:48 GMT
- Title: Adaptive Guidance: Training-free Acceleration of Conditional Diffusion
Models
- Authors: Angela Castillo, Jonas Kohler, Juan C. P\'erez, Juan Pablo P\'erez,
Albert Pumarola, Bernard Ghanem, Pablo Arbel\'aez, Ali Thabet
- Abstract summary: "Adaptive Guidance" (AG) is an efficient variant of computation-Free Guidance (CFG)
AG preserves CFG's image quality while reducing by 25%.
" LinearAG" offers even cheaper inference at the cost of deviating from the baseline model.
- Score: 44.58960475893552
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: This paper presents a comprehensive study on the role of Classifier-Free
Guidance (CFG) in text-conditioned diffusion models from the perspective of
inference efficiency. In particular, we relax the default choice of applying
CFG in all diffusion steps and instead search for efficient guidance policies.
We formulate the discovery of such policies in the differentiable Neural
Architecture Search framework. Our findings suggest that the denoising steps
proposed by CFG become increasingly aligned with simple conditional steps,
which renders the extra neural network evaluation of CFG redundant, especially
in the second half of the denoising process. Building upon this insight, we
propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively
omits network evaluations when the denoising process displays convergence. Our
experiments demonstrate that AG preserves CFG's image quality while reducing
computation by 25%. Thus, AG constitutes a plug-and-play alternative to
Guidance Distillation, achieving 50% of the speed-ups of the latter while being
training-free and retaining the capacity to handle negative prompts. Finally,
we uncover further redundancies of CFG in the first half of the diffusion
process, showing that entire neural function evaluations can be replaced by
simple affine transformations of past score estimates. This method, termed
LinearAG, offers even cheaper inference at the cost of deviating from the
baseline model. Our findings provide insights into the efficiency of the
conditional denoising process that contribute to more practical and swift
deployment of text-conditioned diffusion models.
Related papers
- Purify Unlearnable Examples via Rate-Constrained Variational Autoencoders [101.42201747763178]
Unlearnable examples (UEs) seek to maximize testing error by making subtle modifications to training examples that are correctly labeled.
Our work provides a novel disentanglement mechanism to build an efficient pre-training purification method.
arXiv Detail & Related papers (2024-05-02T16:49:25Z) - PALM: Pushing Adaptive Learning Rate Mechanisms for Continual Test-Time Adaptation [6.181548939188321]
Continual test-time adaptation (CTTA) directly adjusts a pre-trained source discriminative model to changing domains using test data.
We identify layers through the quantification of model prediction uncertainty without relying on pseudo-labels.
We conduct extensive image classification experiments on CIFAR-10C, CIFAR-100C, and ImageNet-C and demonstrate the efficacy of our method.
arXiv Detail & Related papers (2024-03-15T19:35:10Z) - Adaptive Federated Learning Over the Air [108.62635460744109]
We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training.
Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
arXiv Detail & Related papers (2024-03-11T09:10:37Z) - Observation-Guided Diffusion Probabilistic Models [41.749374023639156]
We propose a novel diffusion-based image generation method called the observation-guided diffusion probabilistic model (OGDM)
Our approach reestablishes the training objective by integrating the guidance of the observation process with the Markov chain.
We demonstrate the effectiveness of our training algorithm using diverse inference techniques on strong diffusion model baselines.
arXiv Detail & Related papers (2023-10-06T06:29:06Z) - Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs)
GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations.
We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
arXiv Detail & Related papers (2023-04-22T15:32:59Z) - Second-order step-size tuning of SGD for non-convex optimization [6.021787236982659]
In view of a direct and simple improvement of vanilla SGD, this paper presents a fine-tuning of its step-sizes in the mini-batch case.
One obtains a new first-order gradient method (Step-Tuned SGD) which can be seen as a version of the classical Barzilai-Borwein method.
arXiv Detail & Related papers (2021-03-05T10:01:48Z) - An Uncertainty-Driven GCN Refinement Strategy for Organ Segmentation [53.425900196763756]
We propose a segmentation refinement method based on uncertainty analysis and graph convolutional networks.
We employ the uncertainty levels of the convolutional network in a particular input volume to formulate a semi-supervised graph learning problem.
We show that our method outperforms the state-of-the-art CRF refinement method by improving the dice score by 1% for the pancreas and 2% for spleen.
arXiv Detail & Related papers (2020-12-06T18:55:07Z) - Unsupervised learning of disentangled representations in deep restricted
kernel machines with orthogonality constraints [15.296955630621566]
Constr-DRKM is a deep kernel method for the unsupervised learning of disentangled data representations.
We quantitatively evaluate the proposed method's effectiveness in disentangled feature learning.
arXiv Detail & Related papers (2020-11-25T11:40:10Z) - Detached Error Feedback for Distributed SGD with Random Sparsification [98.98236187442258]
Communication bottleneck has been a critical problem in large-scale deep learning.
We propose a new distributed error feedback (DEF) algorithm, which shows better convergence than error feedback for non-efficient distributed problems.
We also propose DEFA to accelerate the generalization of DEF, which shows better bounds than DEF.
arXiv Detail & Related papers (2020-04-11T03:50:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.