Upgrading VAE Training With Unlimited Data Plans Provided by Diffusion
Models
- URL: http://arxiv.org/abs/2310.19653v2
- Date: Fri, 24 Nov 2023 13:02:55 GMT
- Title: Upgrading VAE Training With Unlimited Data Plans Provided by Diffusion
Models
- Authors: Tim Z. Xiao, Johannes Zenn, Robert Bamler
- Abstract summary: We show that overfitting encoders in VAEs can be effectively mitigated by training on samples from a pre-trained diffusion model.
We analyze generalization performance, amortization gap, and robustness of VAEs trained with our proposed method on three different data sets.
- Score: 12.542073306638988
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Variational autoencoders (VAEs) are popular models for representation
learning but their encoders are susceptible to overfitting (Cremer et al.,
2018) because they are trained on a finite training set instead of the true
(continuous) data distribution $p_{\mathrm{data}}(\mathbf{x})$. Diffusion
models, on the other hand, avoid this issue by keeping the encoder fixed. This
makes their representations less interpretable, but it simplifies training,
enabling accurate and continuous approximations of
$p_{\mathrm{data}}(\mathbf{x})$. In this paper, we show that overfitting
encoders in VAEs can be effectively mitigated by training on samples from a
pre-trained diffusion model. These results are somewhat unexpected as recent
findings (Alemohammad et al., 2023; Shumailov et al., 2023) observe a decay in
generative performance when models are trained on data generated by another
generative model. We analyze generalization performance, amortization gap, and
robustness of VAEs trained with our proposed method on three different data
sets. We find improvements in all metrics compared to both normal training and
conventional data augmentation methods, and we show that a modest amount of
samples from the diffusion model suffices to obtain these gains.
Related papers
- Amortizing intractable inference in diffusion models for vision, language, and control [89.65631572949702]
This paper studies amortized sampling of the posterior over data, $mathbfxsim prm post(mathbfx)propto p(mathbfx)r(mathbfx)$, in a model that consists of a diffusion generative model prior $p(mathbfx)$ and a black-box constraint or function $r(mathbfx)$.
We prove the correctness of a data-free learning objective, relative trajectory balance, for training a diffusion model that samples from
arXiv Detail & Related papers (2024-05-31T16:18:46Z) - Rejection via Learning Density Ratios [50.91522897152437]
Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.
We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.
Our framework is tested empirically over clean and noisy datasets.
arXiv Detail & Related papers (2024-05-29T01:32:17Z) - Learning Defect Prediction from Unrealistic Data [57.53586547895278]
Pretrained models of code have become popular choices for code understanding and generation tasks.
Such models tend to be large and require commensurate volumes of training data.
It has become popular to train models with far larger but less realistic datasets, such as functions with artificially injected bugs.
Models trained on such data tend to only perform well on similar data, while underperforming on real world programs.
arXiv Detail & Related papers (2023-11-02T01:51:43Z) - Diff-Instruct: A Universal Approach for Transferring Knowledge From
Pre-trained Diffusion Models [77.83923746319498]
We propose a framework called Diff-Instruct to instruct the training of arbitrary generative models.
We show that Diff-Instruct results in state-of-the-art single-step diffusion-based models.
Experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models.
arXiv Detail & Related papers (2023-05-29T04:22:57Z) - Variational Diffusion Auto-encoder: Latent Space Extraction from
Pre-trained Diffusion Models [0.0]
Variational Auto-Encoders (VAEs) face challenges with the quality of generated images, often presenting noticeable blurriness.
This issue stems from the unrealistic assumption that approximates the conditional data distribution, $p(textbfx | textbfz)$, as an isotropic Gaussian.
We illustrate how one can extract a latent space from a pre-existing diffusion model by optimizing an encoder to maximize the marginal data log-likelihood.
arXiv Detail & Related papers (2023-04-24T14:44:47Z) - Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be
Consistent [97.64313409741614]
We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time.
We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
arXiv Detail & Related papers (2023-02-17T18:45:04Z) - Learning from aggregated data with a maximum entropy model [73.63512438583375]
We show how a new model, similar to a logistic regression, may be learned from aggregated data only by approximating the unobserved feature distribution with a maximum entropy hypothesis.
We present empirical evidence on several public datasets that the model learned this way can achieve performances comparable to those of a logistic model trained with the full unaggregated data.
arXiv Detail & Related papers (2022-10-05T09:17:27Z) - Forgetting Data from Pre-trained GANs [28.326418377665345]
We investigate how to post-edit a model after training so that it forgets certain kinds of samples.
We provide three different algorithms for GANs that differ on how the samples to be forgotten are described.
Our algorithms are capable of forgetting data while retaining high generation quality at a fraction of the cost of full re-training.
arXiv Detail & Related papers (2022-06-29T03:46:16Z) - Improved Denoising Diffusion Probabilistic Models [4.919647298882951]
We show that DDPMs can achieve competitive log-likelihoods while maintaining high sample quality.
We also find that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes.
We show that the sample quality and likelihood of these models scale smoothly with model capacity and training compute, making them easily scalable.
arXiv Detail & Related papers (2021-02-18T23:44:17Z) - Characterizing and Avoiding Problematic Global Optima of Variational
Autoencoders [28.36260646471421]
Variational Auto-encoders (VAEs) are deep generative latent variable models.
Recent work shows that traditional training methods tend to yield solutions that violate desiderata.
We show that both issues stem from the fact that the global optima of the VAE training objective often correspond to undesirable solutions.
arXiv Detail & Related papers (2020-03-17T15:14:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.