Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion
Models
- URL: http://arxiv.org/abs/2309.06642v2
- Date: Sun, 4 Feb 2024 05:04:10 GMT
- Title: Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion
Models
- Authors: Zalan Fabian, Berk Tinaz, Mahdi Soltanolkotabi
- Abstract summary: Inverse problems arise in a multitude of applications, where the goal is to recover a clean signal from noisy and possibly (non)linear observations.
We propose a novel method that estimates the severity of degradation in the latent space of an autoencoder.
We show that the estimated severity has strong correlation with the true corruption level and can give useful hints at the difficulty of reconstruction problems on a sample-by-sample basis.
- Score: 27.57600878525296
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Inverse problems arise in a multitude of applications, where the goal is to
recover a clean signal from noisy and possibly (non)linear observations. The
difficulty of a reconstruction problem depends on multiple factors, such as the
structure of the ground truth signal, the severity of the degradation and the
complex interactions between the above. This results in natural
sample-by-sample variation in the difficulty of a reconstruction task, which is
often overlooked by contemporary techniques. Our key observation is that most
existing inverse problem solvers lack the ability to adapt their compute power
to the difficulty of the reconstruction task, resulting in subpar performance
and wasteful resource allocation. We propose a novel method that we call
severity encoding, to estimate the degradation severity of noisy, degraded
signals in the latent space of an autoencoder. We show that the estimated
severity has strong correlation with the true corruption level and can give
useful hints at the difficulty of reconstruction problems on a sample-by-sample
basis. Furthermore, we propose a reconstruction method based on latent
diffusion models that leverages the predicted degradation severities to
fine-tune the reverse diffusion sampling trajectory and thus achieve
sample-adaptive inference times. Our framework acts as a wrapper that can be
combined with any latent diffusion-based baseline solver, imbuing it with
sample-adaptivity and acceleration. We perform numerical experiments on both
linear and nonlinear inverse problems and demonstrate that our technique
greatly improves the performance of the baseline solver and achieves up to
$10\times$ acceleration in mean sampling speed.
Related papers
- Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing [84.97865583302244]
We propose a new method called Decoupled Annealing Posterior Sampling (DAPS) that relies on a novel noise annealing process.
DAPS significantly improves sample quality and stability across multiple image restoration tasks.
For example, we achieve a PSNR of 30.72dB on the FFHQ 256 dataset for phase retrieval, which is an improvement of 9.12dB compared to existing methods.
arXiv Detail & Related papers (2024-07-01T17:59:23Z) - Stability and Generalizability in SDE Diffusion Models with Measure-Preserving Dynamics [11.919291977879801]
Inverse problems describe the process of estimating the causal factors from a set of measurements or data.
Diffusion models have shown promise as potent generative tools for solving inverse problems.
arXiv Detail & Related papers (2024-06-19T15:55:12Z) - Deep Equilibrium Diffusion Restoration with Parallel Sampling [120.15039525209106]
Diffusion model-based image restoration (IR) aims to use diffusion models to recover high-quality (HQ) images from degraded images, achieving promising performance.
Most existing methods need long serial sampling chains to restore HQ images step-by-step, resulting in expensive sampling time and high computation costs.
In this work, we aim to rethink the diffusion model-based IR models through a different perspective, i.e., a deep equilibrium (DEQ) fixed point system, called DeqIR.
arXiv Detail & Related papers (2023-11-20T08:27:56Z) - Consistent Signal Reconstruction from Streaming Multivariate Time Series [5.448070998907116]
We formalize for the first time the concept of consistent signal reconstruction from streaming time-series data.
Our method achieves a favorable error-rate decay with the sampling rate compared to a similar but non-consistent reconstruction.
arXiv Detail & Related papers (2023-08-23T22:50:52Z) - A Variational Perspective on Solving Inverse Problems with Diffusion
Models [101.831766524264]
Inverse tasks can be formulated as inferring a posterior distribution over data.
This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable.
We propose a variational approach that by design seeks to approximate the true posterior distribution.
arXiv Detail & Related papers (2023-05-07T23:00:47Z) - Reflected Diffusion Models [93.26107023470979]
We present Reflected Diffusion Models, which reverse a reflected differential equation evolving on the support of the data.
Our approach learns the score function through a generalized score matching loss and extends key components of standard diffusion models.
arXiv Detail & Related papers (2023-04-10T17:54:38Z) - DiracDiffusion: Denoising and Incremental Reconstruction with Assured
Data-Consistency [32.2120650813129]
Diffusion models have established new state of the art in a multitude of computer vision tasks, including image restoration.
We propose a novel framework for inverse problem solving, namely we assume that the observation comes from a degradation process that gradually degrades and noises the original clean image.
Our technique maintains consistency with the original measurement throughout the reverse process, and allows for great flexibility in trading off perceptual quality for improved distortion metrics and sampling speedup via early-stopping.
arXiv Detail & Related papers (2023-03-25T04:37:20Z) - Diffusion Posterior Sampling for General Noisy Inverse Problems [50.873313752797124]
We extend diffusion solvers to handle noisy (non)linear inverse problems via approximation of the posterior sampling.
Our method demonstrates that diffusion models can incorporate various measurement noise statistics.
arXiv Detail & Related papers (2022-09-29T11:12:27Z) - Towards performant and reliable undersampled MR reconstruction via
diffusion model sampling [67.73698021297022]
DiffuseRecon is a novel diffusion model-based MR reconstruction method.
It guides the generation process based on the observed signals.
It does not require additional training on specific acceleration factors.
arXiv Detail & Related papers (2022-03-08T02:25:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.