Provable Compressed Sensing with Generative Priors via Langevin Dynamics
- URL: http://arxiv.org/abs/2102.12643v1
- Date: Thu, 25 Feb 2021 02:35:14 GMT
- Title: Provable Compressed Sensing with Generative Priors via Langevin Dynamics
- Authors: Thanh V. Nguyen, Gauri Jagatap and Chinmay Hegde
- Abstract summary: We introduce the use of gradient Langevin dynamics (SGLD) for compressed sensing with a generative prior.
Under mild assumptions on the generative model, we prove the convergence of SGLD to the true signal.
- Score: 43.59745920150787
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep generative models have emerged as a powerful class of priors for signals
in various inverse problems such as compressed sensing, phase retrieval and
super-resolution. Here, we assume an unknown signal to lie in the range of some
pre-trained generative model. A popular approach for signal recovery is via
gradient descent in the low-dimensional latent space. While gradient descent
has achieved good empirical performance, its theoretical behavior is not well
understood. In this paper, we introduce the use of stochastic gradient Langevin
dynamics (SGLD) for compressed sensing with a generative prior. Under mild
assumptions on the generative model, we prove the convergence of SGLD to the
true signal. We also demonstrate competitive empirical performance to standard
gradient descent.
Related papers
- Role of Momentum in Smoothing Objective Function and Generalizability of Deep Neural Networks [0.6906005491572401]
We show that noise in gradient descent (SGD) with momentum smoothes the objective function, the degree of which is determined by the learning rate, the batch size, the momentum factor, and the upper bound of the norm.
We also provide experimental results supporting our assertion model generalizability depends on the noise level.
arXiv Detail & Related papers (2024-02-04T02:48:28Z) - Towards Training Without Depth Limits: Batch Normalization Without
Gradient Explosion [83.90492831583997]
We show that a batch-normalized network can keep the optimal signal propagation properties, but avoid exploding gradients in depth.
We use a Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded depth.
We also design an activation shaping scheme that empirically achieves the same properties for certain non-linear activations.
arXiv Detail & Related papers (2023-10-03T12:35:02Z) - Max-affine regression via first-order methods [7.12511675782289]
The max-affine model ubiquitously arises in applications in signal processing and statistics.
We present a non-asymptotic convergence analysis of gradient descent (GD) and mini-batch gradient descent (SGD) for max-affine regression.
arXiv Detail & Related papers (2023-08-15T23:46:44Z) - Score-Guided Intermediate Layer Optimization: Fast Langevin Mixing for
Inverse Problem [97.64313409741614]
We prove fast mixing and characterize the stationary distribution of the Langevin Algorithm for inverting random weighted DNN generators.
We propose to do posterior sampling in the latent space of a pre-trained generative model.
arXiv Detail & Related papers (2022-06-18T03:47:37Z) - On the Double Descent of Random Features Models Trained with SGD [78.0918823643911]
We study properties of random features (RF) regression in high dimensions optimized by gradient descent (SGD)
We derive precise non-asymptotic error bounds of RF regression under both constant and adaptive step-size SGD setting.
We observe the double descent phenomenon both theoretically and empirically.
arXiv Detail & Related papers (2021-10-13T17:47:39Z) - Direction Matters: On the Implicit Bias of Stochastic Gradient Descent
with Moderate Learning Rate [105.62979485062756]
This paper attempts to characterize the particular regularization effect of SGD in the moderate learning rate regime.
We show that SGD converges along the large eigenvalue directions of the data matrix, while GD goes after the small eigenvalue directions.
arXiv Detail & Related papers (2020-11-04T21:07:52Z) - A High Probability Analysis of Adaptive SGD with Momentum [22.9530287983179]
Gradient Descent (DSG) and its variants are the most used algorithms in machine learning applications.
We show for the first time the probability of the gradients to zero in smooth non setting for DelayedGrad with momentum.
arXiv Detail & Related papers (2020-07-28T15:06:22Z) - When and How Can Deep Generative Models be Inverted? [28.83334026125828]
Deep generative models (GANs and VAEs) have been developed quite extensively in recent years.
We define conditions that are applicable to any inversion algorithm (gradient descent, deep encoder, etc.) under which such generative models are invertible.
We show that our method outperforms gradient descent when inverting such generators, both for clean and corrupted signals.
arXiv Detail & Related papers (2020-06-28T09:37:52Z) - When Does Preconditioning Help or Hurt Generalization? [74.25170084614098]
We show how the textitimplicit bias of first and second order methods affects the comparison of generalization properties.
We discuss several approaches to manage the bias-variance tradeoff, and the potential benefit of interpolating between GD and NGD.
arXiv Detail & Related papers (2020-06-18T17:57:26Z) - Federated Stochastic Gradient Langevin Dynamics [12.180900849847252]
gradient MCMC methods, such as gradient Langevin dynamics (SGLD), employ fast but noisy gradient estimates to enable large-scale posterior sampling.
We propose conducive gradients, a simple mechanism that combines local likelihood approximations to correct gradient updates.
We demonstrate that our approach can handle delayed communication rounds, converging to the target posterior in cases where DSGLD fails.
arXiv Detail & Related papers (2020-04-23T15:25:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.