Score Normalization for a Faster Diffusion Exponential Integrator Sampler

• URL: http://arxiv.org/abs/2311.00157v2
• Date: Fri, 10 Nov 2023 00:30:14 GMT
• Title: Score Normalization for a Faster Diffusion Exponential Integrator Sampler
• Authors: Guoxuan Xia, Duolikun Danier, Ayan Das, Stathi Fotiadis, Farhang Nabiei, Ushnish Sengupta, Alberto Bernacchia
• Abstract summary: Zhang et al. have proposed the Diffusion Exponential Integrator Sampler (DEIS) for fast generation of samples from Diffusion Models. Key to this approach is the score function re parameterisation, which reduces the integration error incurred from using a fixed score function estimate. We find that our score normalisation (DEIS-SN) consistently improves FID compared to vanilla DEIS.
• Score: 8.914068241467234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, Zhang et al. have proposed the Diffusion Exponential Integrator Sampler (DEIS) for fast generation of samples from Diffusion Models. It leverages the semi-linear nature of the probability flow ordinary differential equation (ODE) in order to greatly reduce integration error and improve generation quality at low numbers of function evaluations (NFEs). Key to this approach is the score function reparameterisation, which reduces the integration error incurred from using a fixed score function estimate over each integration step. The original authors use the default parameterisation used by models trained for noise prediction -- multiply the score by the standard deviation of the conditional forward noising distribution. We find that although the mean absolute value of this score parameterisation is close to constant for a large portion of the reverse sampling process, it changes rapidly at the end of sampling. As a simple fix, we propose to instead reparameterise the score (at inference) by dividing it by the average absolute value of previous score estimates at that time step collected from offline high NFE generations. We find that our score normalisation (DEIS-SN) consistently improves FID compared to vanilla DEIS, showing an improvement at 10 NFEs from 6.44 to 5.57 on CIFAR-10 and from 5.9 to 4.95 on LSUN-Church 64x64. Our code is available at https://github.com/mtkresearch/Diffusion-DEIS-SN

Related papers

• FSampler: Training Free Acceleration of Diffusion Sampling via Epsilon Extrapolation [0.0]
  FSampler is a training free, sampler execution layer that accelerates diffusion sampling by reducing the number of function evaluations (NFE)<n>FSampler maintains a short history of denoising signals from recent real model calls and extrapolates the next epsilon using finite difference predictors.<n> operating at the sampler level, FSampler integrates with Euler/DDIM, DPM++ 2M/2S, LMS/AB2, and RES family exponential multistep methods.
  arXiv  Detail & Related papers  (2025-11-12T10:21:25Z)
• Faster Diffusion Models via Higher-Order Approximation [28.824924809206255]
  We propose a principled, training-free sampling algorithm that requires only the order of d1+2/K varepsilon-1/K $$ score function evaluations.<n>Our theory is robust vis-a-vis inexact score estimation, degrading gracefully as the score estimation error increases.
  arXiv  Detail & Related papers  (2025-06-30T16:49:03Z)
• Fast Convergence for High-Order ODE Solvers in Diffusion Probabilistic Models [5.939858158928473]
  Diffusion probabilistic models generate samples by learning to reverse a noise-injection process that transforms data into noise.<n>Reformulating this reverse process as a deterministic probability flow ordinary differential equation (ODE) enables efficient sampling using high-order solvers.<n>Since the score function is typically approximated by a neural network, analyzing the interaction between its regularity, approximation error, and numerical integration error is key to understanding the overall sampling accuracy.
  arXiv  Detail & Related papers  (2025-06-16T03:09:25Z)
• Dimension-free Score Matching and Time Bootstrapping for Diffusion Models [19.62665684173391]
  Diffusion models generate samples by estimating the score function of the target distribution at various noise levels.<n>We introduce a martingale-based error decomposition and sharp variance bounds, enabling efficient learning from dependent data.<n>Building on these insights, we propose Bootstrapped Score Matching (BSM), a variance reduction technique that leverages previously learned scores to improve accuracy at higher noise levels.
  arXiv  Detail & Related papers  (2025-02-14T18:32:22Z)
• Distributional Diffusion Models with Scoring Rules [83.38210785728994]
  Diffusion models generate high-quality synthetic data. generating high-quality outputs requires many discretization steps. We propose to accomplish sample generation by learning the posterior em distribution of clean data samples.
  arXiv  Detail & Related papers  (2025-02-04T16:59:03Z)
• Robust Fine-tuning of Zero-shot Models via Variance Reduction [56.360865951192324]
  When fine-tuning zero-shot models, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD) We propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs.
  arXiv  Detail & Related papers  (2024-11-11T13:13:39Z)
• PFDiff: Training-Free Acceleration of Diffusion Models Combining Past and Future Scores [4.595421654683656]
  Diffusion Probabilistic Models (DPMs) have shown remarkable potential in image generation. Most existing solutions accelerate the sampling process by proposing fast ODE solvers. We propose PFDiff, a novel training-free and timestep-skipping strategy, which enables existing fast ODE solvers to operate with fewer NFE.
  arXiv  Detail & Related papers  (2024-08-16T16:12:44Z)
• Towards Fast Stochastic Sampling in Diffusion Generative Models [22.01769257075573]
  Diffusion models suffer from slow sample generation at inference time. We propose Splittings for fast sampling in pre-trained diffusion models in augmented spaces. We show that a naive application of splitting is sub-optimal for fast sampling.
  arXiv  Detail & Related papers  (2024-02-11T14:04:13Z)
• Closed-Form Diffusion Models [14.20871291924173]
  Score-based generative models (SGMs) sample from a target distribution by iteratively transforming noise using the score function of the target. For any finite training set, this score function can be evaluated in closed form, but the resulting SGM memorizes its training data and does not generate novel samples. We propose an efficient nearest-neighbor-based estimator of its score function.
  arXiv  Detail & Related papers  (2023-10-19T00:45:05Z)
• On Accelerating Diffusion-Based Sampling Process via Improved Integration Approximation [12.882586878998579]
  A popular approach to sample a diffusion-based generative model is to solve an ordinary differential equation (ODE) We consider accelerating several popular ODE-based sampling processes by optimizing certain coefficients via improved integration approximation (IIA) We show that considerably better FID scores can be achieved by using IIA-EDM, IIA-DDIM, and IIA-DPM-r than the original counterparts.
  arXiv  Detail & Related papers  (2023-04-22T06:06:28Z)
• Post-Processing Temporal Action Detection [134.26292288193298]
  Temporal Action Detection (TAD) methods typically take a pre-processing step in converting an input varying-length video into a fixed-length snippet representation sequence. This pre-processing step would temporally downsample the video, reducing the inference resolution and hampering the detection performance in the original temporal resolution. We introduce a novel model-agnostic post-processing method without model redesign and retraining.
  arXiv  Detail & Related papers  (2022-11-27T19:50:37Z)
• Combining Gradients and Probabilities for Heterogeneous Approximation of Neural Networks [2.5744053804694893]
  We discuss the validity of additive Gaussian noise as a surrogate model for behavioral simulation of approximate multipliers. The amount of noise injected into the accurate computations is learned during network training using backpropagation. Our experiments show that the combination of heterogeneous approximation and neural network retraining reduces the energy consumption for variants.
  arXiv  Detail & Related papers  (2022-08-15T15:17:34Z)
• Detecting Label Noise via Leave-One-Out Cross Validation [0.0]
  We present a simple algorithm for identifying and correcting real-valued noisy labels from a mixture of clean and corrupted samples. A heteroscedastic noise model is employed, in which additive Gaussian noise terms with independent variances are associated with each and all of the observed labels. We show that the presented method can pinpoint corrupted samples and lead to better regression models when trained on synthetic and real-world scientific data sets.
  arXiv  Detail & Related papers  (2021-03-21T10:02:50Z)
• Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach [38.45568741734893]
  Inference in our GP model is done by a single feed forward pass through the network, significantly faster than semi-amortized methods. We show that our approach attains higher test data likelihood than the state-of-the-arts on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-02-05T13:01:12Z)
• Score-Based Generative Modeling through Stochastic Differential Equations [114.39209003111723]
  We present a differential equation that transforms a complex data distribution to a known prior distribution by injecting noise. A corresponding reverse-time SDE transforms the prior distribution back into the data distribution by slowly removing the noise. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks. We demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.
  arXiv  Detail & Related papers  (2020-11-26T19:39:10Z)
• Gaussian MRF Covariance Modeling for Efficient Black-Box Adversarial Attacks [86.88061841975482]
  We study the problem of generating adversarial examples in a black-box setting, where we only have access to a zeroth order oracle. We use this setting to find fast one-step adversarial attacks, akin to a black-box version of the Fast Gradient Sign Method(FGSM) We show that the method uses fewer queries and achieves higher attack success rates than the current state of the art.
  arXiv  Detail & Related papers  (2020-10-08T18:36:51Z)
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models [80.22609163316459]
  We introduce an unbiased estimator of the log marginal likelihood and its gradients for latent variable models based on randomized truncation of infinite series. We show that models trained using our estimator give better test-set likelihoods than a standard importance-sampling based approach for the same average computational cost.
  arXiv  Detail & Related papers  (2020-04-01T11:49:30Z)
• Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples [67.11669996924671]
  We introduce a simple (one line of code) modification to the Generative Adversarial Network (GAN) training algorithm. When updating the generator parameters, we zero out the gradient contributions from the elements of the batch that the critic scores as least realistic' We show that this top-k update' procedure is a generally applicable improvement.
  arXiv  Detail & Related papers  (2020-02-14T19:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.