Learning to Jump: Thinning and Thickening Latent Counts for Generative Modeling

• URL: http://arxiv.org/abs/2305.18375v1
• Date: Sun, 28 May 2023 05:38:28 GMT
• Title: Learning to Jump: Thinning and Thickening Latent Counts for Generative Modeling
• Authors: Tianqi Chen and Mingyuan Zhou
• Abstract summary: Learning to jump is a general recipe for generative modeling of various types of data. We demonstrate when learning to jump is expected to perform comparably to learning to denoise, and when it is expected to perform better.
• Score: 69.60713300418467
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to denoise has emerged as a prominent paradigm to design state-of-the-art deep generative models for natural images. How to use it to model the distributions of both continuous real-valued data and categorical data has been well studied in recently proposed diffusion models. However, it is found in this paper to have limited ability in modeling some other types of data, such as count and non-negative continuous data, that are often highly sparse, skewed, heavy-tailed, and/or overdispersed. To this end, we propose learning to jump as a general recipe for generative modeling of various types of data. Using a forward count thinning process to construct learning objectives to train a deep neural network, it employs a reverse count thickening process to iteratively refine its generation through that network. We demonstrate when learning to jump is expected to perform comparably to learning to denoise, and when it is expected to perform better. For example, learning to jump is recommended when the training data is non-negative and exhibits strong sparsity, skewness, heavy-tailedness, and/or heterogeneity.

Related papers

• Heat Death of Generative Models in Closed-Loop Learning [63.83608300361159]
  We study the learning dynamics of generative models that are fed back their own produced content in addition to their original training dataset. We show that, unless a sufficient amount of external data is introduced at each iteration, any non-trivial temperature leads the model to degenerate.
  arXiv  Detail & Related papers  (2024-04-02T21:51:39Z)
• Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learning [63.850451635362425]
  Continual learning requires a model to adapt to ongoing changes in the data distribution. We show that the combination of a large language model and an image generation model can similarly provide useful premonitions. We find that the backbone of our pre-trained networks can learn representations useful for the downstream continual learning problem.
  arXiv  Detail & Related papers  (2024-03-12T06:29:54Z)
• Diffusion-based Neural Network Weights Generation [85.6725307453325]
  We propose an efficient and adaptive transfer learning scheme through dataset-conditioned pretrained weights sampling. Specifically, we use a latent diffusion model with a variational autoencoder that can reconstruct the neural network weights.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Training Class-Imbalanced Diffusion Model Via Overlap Optimization [55.96820607533968]
  Diffusion models trained on real-world datasets often yield inferior fidelity for tail classes. Deep generative models, including diffusion models, are biased towards classes with abundant training images. We propose a method based on contrastive learning to minimize the overlap between distributions of synthetic images for different classes.
  arXiv  Detail & Related papers  (2024-02-16T16:47:21Z)
• Continual Learning of Diffusion Models with Generative Distillation [34.52513912701778]
  Diffusion models are powerful generative models that achieve state-of-the-art performance in image synthesis. In this paper, we propose generative distillation, an approach that distils the entire reverse process of a diffusion model.
  arXiv  Detail & Related papers  (2023-11-23T14:33:03Z)
• Generative Forests [26.09279398946235]
  We introduce new tree-based generative models convenient for density modeling and data generation. We also introduce a training algorithm which simplifies the training setting of previous approaches. Experiments are provided on missing data imputation and comparing generated data to real data, displaying the quality of the results.
  arXiv  Detail & Related papers  (2023-08-07T14:58:53Z)
• Learning Sparse Latent Representations for Generator Model [7.467412443287767]
  We present a new unsupervised learning method to enforce sparsity on the latent space for the generator model. Our model consists of only one top-down generator network that maps the latent variable to the observed data.
  arXiv  Detail & Related papers  (2022-09-20T18:58:24Z)
• Distilling Interpretable Models into Human-Readable Code [71.11328360614479]
  Human-readability is an important and desirable standard for machine-learned model interpretability. We propose to train interpretable models using conventional methods, and then distill them into concise, human-readable code. We describe a piecewise-linear curve-fitting algorithm that produces high-quality results efficiently and reliably across a broad range of use cases.
  arXiv  Detail & Related papers  (2021-01-21T01:46:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.