Zero-Shot Adaptation for Approximate Posterior Sampling of Diffusion Models in Inverse Problems

• URL: http://arxiv.org/abs/2407.11288v1
• Date: Tue, 16 Jul 2024 00:09:37 GMT
• Title: Zero-Shot Adaptation for Approximate Posterior Sampling of Diffusion Models in Inverse Problems
• Authors: Yaşar Utku Alçalar, Mehmet Akçakaya,
• Abstract summary: We propose zero-shot approximate posterior sampling (ZAPS) to solve inverse problems in imaging. ZAPS fixes the number of sampling steps, and uses zero-shot training with a physics-guided loss function to learn log-likelihood weights at each irregular timestep. Our results show ZAPS reduces inference time, provides robustness to irregular noise schedules, and improves reconstruction quality.
• Score: 2.8237889121096034
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Diffusion models have emerged as powerful generative techniques for solving inverse problems. Despite their success in a variety of inverse problems in imaging, these models require many steps to converge, leading to slow inference time. Recently, there has been a trend in diffusion models for employing sophisticated noise schedules that involve more frequent iterations of timesteps at lower noise levels, thereby improving image generation and convergence speed. However, application of these ideas for solving inverse problems with diffusion models remain challenging, as these noise schedules do not perform well when using empirical tuning for the forward model log-likelihood term weights. To tackle these challenges, we propose zero-shot approximate posterior sampling (ZAPS) that leverages connections to zero-shot physics-driven deep learning. ZAPS fixes the number of sampling steps, and uses zero-shot training with a physics-guided loss function to learn log-likelihood weights at each irregular timestep. We apply ZAPS to the recently proposed diffusion posterior sampling method as baseline, though ZAPS can also be used with other posterior sampling diffusion models. We further approximate the Hessian of the logarithm of the prior using a diagonalization approach with learnable diagonal entries for computational efficiency. These parameters are optimized over a fixed number of epochs with a given computational budget. Our results for various noisy inverse problems, including Gaussian and motion deblurring, inpainting, and super-resolution show that ZAPS reduces inference time, provides robustness to irregular noise schedules and improves reconstruction quality. Code is available at https://github.com/ualcalar17/ZAPS

Related papers

• Sequential Posterior Sampling with Diffusion Models [15.028061496012924]
  We propose a novel approach that models the transition dynamics to improve the efficiency of sequential diffusion posterior sampling in conditional image synthesis. We demonstrate the effectiveness of our approach on a real-world dataset of high frame rate cardiac ultrasound images. Our method opens up new possibilities for real-time applications of diffusion models in imaging and other domains requiring real-time inference.
  arXiv  Detail & Related papers  (2024-09-09T07:55:59Z)
• Solving Video Inverse Problems Using Image Diffusion Models [58.464465016269614]
  We introduce an innovative video inverse solver that leverages only image diffusion models. Our method treats the time dimension of a video as the batch dimension image diffusion models. We also introduce a batch-consistent sampling strategy that encourages consistency across batches.
  arXiv  Detail & Related papers  (2024-09-04T09:48:27Z)
• One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls [77.42510898755037]
  One More Step (OMS) is a compact network that incorporates an additional simple yet effective step during inference. OMS elevates image fidelity and harmonizes the dichotomy between training and inference, while preserving original model parameters. Once trained, various pre-trained diffusion models with the same latent domain can share the same OMS module.
  arXiv  Detail & Related papers  (2023-11-27T12:02:42Z)
• Deep Equilibrium Diffusion Restoration with Parallel Sampling [120.15039525209106]
  Diffusion model-based image restoration (IR) aims to use diffusion models to recover high-quality (HQ) images from degraded images, achieving promising performance. Most existing methods need long serial sampling chains to restore HQ images step-by-step, resulting in expensive sampling time and high computation costs. In this work, we aim to rethink the diffusion model-based IR models through a different perspective, i.e., a deep equilibrium (DEQ) fixed point system, called DeqIR.
  arXiv  Detail & Related papers  (2023-11-20T08:27:56Z)
• A Variational Perspective on Solving Inverse Problems with Diffusion Models [101.831766524264]
  Inverse tasks can be formulated as inferring a posterior distribution over data. This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. We propose a variational approach that by design seeks to approximate the true posterior distribution.
  arXiv  Detail & Related papers  (2023-05-07T23:00:47Z)
• Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems [14.809545109705256]
  This paper presents a fast and effective solution by proposing a simple closed-form approximation to the likelihood score. For both diffusion and flow-based models, extensive experiments are conducted on various noisy linear inverse problems. Our method demonstrates highly competitive or even better reconstruction performances while being significantly faster than all the baseline methods.
  arXiv  Detail & Related papers  (2022-11-20T01:09:49Z)
• Diffusion Posterior Sampling for General Noisy Inverse Problems [50.873313752797124]
  We extend diffusion solvers to handle noisy (non)linear inverse problems via approximation of the posterior sampling. Our method demonstrates that diffusion models can incorporate various measurement noise statistics.
  arXiv  Detail & Related papers  (2022-09-29T11:12:27Z)
• Denoising Diffusion Restoration Models [110.1244240726802]
  Denoising Diffusion Restoration Models (DDRM) is an efficient, unsupervised posterior sampling method. We demonstrate DDRM's versatility on several image datasets for super-resolution, deblurring, inpainting, and colorization.
  arXiv  Detail & Related papers  (2022-01-27T20:19:07Z)
• Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction [31.61199061999173]
  Diffusion models have a critical downside - they are inherently slow to sample from, needing few thousand steps of iteration to generate images from pure Gaussian noise. We show that starting from Gaussian noise is unnecessary. Instead, starting from a single forward diffusion with better initialization significantly reduces the number of sampling steps in the reverse conditional diffusion. New sampling strategy, dubbed ComeCloser-DiffuseFaster (CCDF), also reveals a new insight on how the existing feedforward neural network approaches for inverse problems can be synergistically combined with the diffusion models.
  arXiv  Detail & Related papers  (2021-12-09T04:28:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.