S4S: Solving for a Diffusion Model Solver

• URL: http://arxiv.org/abs/2502.17423v1
• Date: Mon, 24 Feb 2025 18:55:54 GMT
• Title: S4S: Solving for a Diffusion Model Solver
• Authors: Eric Frankel, Sitan Chen, Jerry Li, Pang Wei Koh, Lillian J. Ratliff, Sewoong Oh,
• Abstract summary: Diffusion models (DMs) create samples from a data distribution by starting from random noise and solving a reverse-time ordinary differential equation (ODE)<n>We propose a new method that learns a good solver for the DM, which we call Solving for the Solver (S4S)<n>In all settings, S4S uniformly improves the sample quality relative to traditional ODE solvers.
• Score: 52.99341671532249
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models (DMs) create samples from a data distribution by starting from random noise and iteratively solving a reverse-time ordinary differential equation (ODE). Because each step in the iterative solution requires an expensive neural function evaluation (NFE), there has been significant interest in approximately solving these diffusion ODEs with only a few NFEs without modifying the underlying model. However, in the few NFE regime, we observe that tracking the true ODE evolution is fundamentally impossible using traditional ODE solvers. In this work, we propose a new method that learns a good solver for the DM, which we call Solving for the Solver (S4S). S4S directly optimizes a solver to obtain good generation quality by learning to match the output of a strong teacher solver. We evaluate S4S on six different pre-trained DMs, including pixel-space and latent-space DMs for both conditional and unconditional sampling. In all settings, S4S uniformly improves the sample quality relative to traditional ODE solvers. Moreover, our method is lightweight, data-free, and can be plugged in black-box on top of any discretization schedule or architecture to improve performance. Building on top of this, we also propose S4S-Alt, which optimizes both the solver and the discretization schedule. By exploiting the full design space of DM solvers, with 5 NFEs, we achieve an FID of 3.73 on CIFAR10 and 13.26 on MS-COCO, representing a $1.5\times$ improvement over previous training-free ODE methods.

Related papers

• Fast ODE-based Sampling for Diffusion Models in Around 5 Steps [17.500594480727617]
  We propose Approximate MEan-Direction solver (AMED-r) that eliminates truncation errors by directly learning the mean direction for fast sampling. Our method can be easily used as a plugin to further improve existing ODE-based samplers.
  arXiv  Detail & Related papers  (2023-11-30T13:07:19Z)
• Gaussian Mixture Solvers for Diffusion Models [84.83349474361204]
  We introduce a novel class of SDE-based solvers called GMS for diffusion models. Our solver outperforms numerous SDE-based solvers in terms of sample quality in image generation and stroke-based synthesis.
  arXiv  Detail & Related papers  (2023-11-02T02:05:38Z)
• DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics [23.030972042695275]
  Diffusion models (DPMs) have exhibited excellent performance for high-fidelity image generation while suffering from inefficient sampling. Recent works accelerate the sampling procedure by proposing fast ODE solvers that leverage the specific ODE form of DPMs. We propose a novel formulation towards the optimal parameterization during sampling that minimizes the first-order discretization error.
  arXiv  Detail & Related papers  (2023-10-20T04:23:12Z)
• Distilling ODE Solvers of Diffusion Models into Smaller Steps [32.49916706943228]
  We introduce Distilled-ODE solvers, a straightforward distillation approach grounded in ODE solver formulations. Our method seamlessly integrates the strengths of both learning-free and learning-based sampling. Our method incurs negligible computational overhead compared to previous distillation techniques.
  arXiv  Detail & Related papers  (2023-09-28T13:12:18Z)
• Faster Training of Neural ODEs Using Gau{\ss}-Legendre Quadrature [68.9206193762751]
  We propose an alternative way to speed up the training of neural ODEs. We use Gauss-Legendre quadrature to solve integrals faster than ODE-based methods. We also extend the idea to training SDEs using the Wong-Zakai theorem, by training a corresponding ODE and transferring the parameters.
  arXiv  Detail & Related papers  (2023-08-21T11:31:15Z)
• Experimental study of Neural ODE training with adaptive solver for dynamical systems modeling [72.84259710412293]
  Some ODE solvers called adaptive can adapt their evaluation strategy depending on the complexity of the problem at hand. This paper describes a simple set of experiments to show why adaptive solvers cannot be seamlessly leveraged as a black-box for dynamical systems modelling.
  arXiv  Detail & Related papers  (2022-11-13T17:48:04Z)
• Learning to Optimize Permutation Flow Shop Scheduling via Graph-based Imitation Learning [70.65666982566655]
  Permutation flow shop scheduling (PFSS) is widely used in manufacturing systems. We propose to train the model via expert-driven imitation learning, which accelerates convergence more stably and accurately. Our model's network parameters are reduced to only 37% of theirs, and the solution gap of our model towards the expert solutions decreases from 6.8% to 1.3% on average.
  arXiv  Detail & Related papers  (2022-10-31T09:46:26Z)
• Message Passing Neural PDE Solvers [60.77761603258397]
  We build a neural message passing solver, replacing allally designed components in the graph with backprop-optimized neural function approximators. We show that neural message passing solvers representationally contain some classical methods, such as finite differences, finite volumes, and WENO schemes. We validate our method on various fluid-like flow problems, demonstrating fast, stable, and accurate performance across different domain topologies, equation parameters, discretizations, etc., in 1D and 2D.
  arXiv  Detail & Related papers  (2022-02-07T17:47:46Z)
• Meta-Solver for Neural Ordinary Differential Equations [77.8918415523446]
  We investigate how the variability in solvers' space can improve neural ODEs performance. We show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-03-15T17:26:34Z)
• ResNet After All? Neural ODEs and Their Numerical Solution [28.954378025052925]
  We show that trained Neural Ordinary Differential Equation models actually depend on the specific numerical method used during training. We propose a method that monitors the behavior of the ODE solver during training to adapt its step size.
  arXiv  Detail & Related papers  (2020-07-30T11:24:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.