Fast Samplers for Inverse Problems in Iterative Refinement Models

• URL: http://arxiv.org/abs/2405.17673v2
• Date: Fri, 01 Nov 2024 06:22:30 GMT
• Title: Fast Samplers for Inverse Problems in Iterative Refinement Models
• Authors: Kushagra Pandey, Ruihan Yang, Stephan Mandt,
• Abstract summary: We propose a plug-and-play framework for constructing efficient samplers for inverse problems. Our method can generate high-quality samples in as few as 5 conditional sampling steps and outperforms competing baselines requiring 20-1000 steps.
• Score: 19.099632445326826
• License:
• Abstract: Constructing fast samplers for unconditional diffusion and flow-matching models has received much attention recently; however, existing methods for solving inverse problems, such as super-resolution, inpainting, or deblurring, still require hundreds to thousands of iterative steps to obtain high-quality results. We propose a plug-and-play framework for constructing efficient samplers for inverse problems, requiring only pre-trained diffusion or flow-matching models. We present Conditional Conjugate Integrators, which leverage the specific form of the inverse problem to project the respective conditional diffusion/flow dynamics into a more amenable space for sampling. Our method complements popular posterior approximation methods for solving inverse problems using diffusion/flow models. We evaluate the proposed method's performance on various linear image restoration tasks across multiple datasets, employing diffusion and flow-matching models. Notably, on challenging inverse problems like 4x super-resolution on the ImageNet dataset, our method can generate high-quality samples in as few as 5 conditional sampling steps and outperforms competing baselines requiring 20-1000 steps. Our code will be publicly available at https://github.com/mandt-lab/c-pigdm

Related papers

• Gaussian is All You Need: A Unified Framework for Solving Inverse Problems via Diffusion Posterior Sampling [16.683393726483978]
  Diffusion models can generate a variety of high-quality images by modeling complex data distributions. Most of the existing diffusion-based methods integrate data consistency steps within the diffusion reverse sampling process. We show that the existing approximations are either insufficient or computationally inefficient.
  arXiv  Detail & Related papers  (2024-09-13T15:20:03Z)
• Solving Video Inverse Problems Using Image Diffusion Models [58.464465016269614]
  We introduce an innovative video inverse solver that leverages only image diffusion models. Our method treats the time dimension of a video as the batch dimension image diffusion models. We also introduce a batch-consistent sampling strategy that encourages consistency across batches.
  arXiv  Detail & Related papers  (2024-09-04T09:48:27Z)
• Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problems [12.482127049881026]
  We propose a novel approach to solve inverse problems with a diffusion prior from an amortized variational inference perspective. Our amortized inference learns a function that directly maps measurements to the implicit posterior distributions of corresponding clean data, enabling a single-step posterior sampling even for unseen measurements.
  arXiv  Detail & Related papers  (2024-07-23T02:14:18Z)
• Accelerating Parallel Sampling of Diffusion Models [25.347710690711562]
  We propose a novel approach that accelerates the sampling of diffusion models by parallelizing the autoregressive process. Applying these techniques, we introduce ParaTAA, a universal and training-free parallel sampling algorithm. Our experiments demonstrate that ParaTAA can decrease the inference steps required by common sequential sampling algorithms by a factor of 4$sim$14 times.
  arXiv  Detail & Related papers  (2024-02-15T14:27:58Z)
• Prompt-tuning latent diffusion models for inverse problems [72.13952857287794]
  We propose a new method for solving imaging inverse problems using text-to-image latent diffusion models as general priors. Our method, called P2L, outperforms both image- and latent-diffusion model-based inverse problem solvers on a variety of tasks, such as super-resolution, deblurring, and inpainting.
  arXiv  Detail & Related papers  (2023-10-02T11:31:48Z)
• A Variational Perspective on Solving Inverse Problems with Diffusion Models [101.831766524264]
  Inverse tasks can be formulated as inferring a posterior distribution over data. This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. We propose a variational approach that by design seeks to approximate the true posterior distribution.
  arXiv  Detail & Related papers  (2023-05-07T23:00:47Z)
• Reflected Diffusion Models [93.26107023470979]
  We present Reflected Diffusion Models, which reverse a reflected differential equation evolving on the support of the data. Our approach learns the score function through a generalized score matching loss and extends key components of standard diffusion models.
  arXiv  Detail & Related papers  (2023-04-10T17:54:38Z)
• Accelerating Guided Diffusion Sampling with Splitting Numerical Methods [8.689906452450938]
  Recent techniques can accelerate unguided sampling by applying high-order numerical methods to the sampling process. This paper explores the culprit of this problem and provides a solution based on operator splitting methods. Our proposed method can re-utilize the high-order methods for guided sampling and can generate images with the same quality as a 250-step DDIM baseline.
  arXiv  Detail & Related papers  (2023-01-27T06:48:29Z)
• Diffusion Posterior Sampling for General Noisy Inverse Problems [50.873313752797124]
  We extend diffusion solvers to handle noisy (non)linear inverse problems via approximation of the posterior sampling. Our method demonstrates that diffusion models can incorporate various measurement noise statistics.
  arXiv  Detail & Related papers  (2022-09-29T11:12:27Z)
• Improving Diffusion Models for Inverse Problems using Manifold Constraints [55.91148172752894]
  We show that current solvers throw the sample path off the data manifold, and hence the error accumulates. To address this, we propose an additional correction term inspired by the manifold constraint. We show that our method is superior to the previous methods both theoretically and empirically.
  arXiv  Detail & Related papers  (2022-06-02T09:06:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.