Distilling the Knowledge from Normalizing Flows

• URL: http://arxiv.org/abs/2106.12699v2
• Date: Fri, 25 Jun 2021 09:41:45 GMT
• Title: Distilling the Knowledge from Normalizing Flows
• Authors: Dmitry Baranchuk, Vladimir Aliev, Artem Babenko
• Abstract summary: Normalizing flows are a powerful class of generative models demonstrating strong performance in several speech and vision problems. We propose a simple distillation approach and demonstrate its effectiveness on state-of-the-art conditional flow-based models for image super-resolution and speech synthesis.
• Score: 22.578033953780697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Normalizing flows are a powerful class of generative models demonstrating strong performance in several speech and vision problems. In contrast to other generative models, normalizing flows are latent variable models with tractable likelihoods and allow for stable training. However, they have to be carefully designed to represent invertible functions with efficient Jacobian determinant calculation. In practice, these requirements lead to overparameterized and sophisticated architectures that are inferior to alternative feed-forward models in terms of inference time and memory consumption. In this work, we investigate whether one can distill flow-based models into more efficient alternatives. We provide a positive answer to this question by proposing a simple distillation approach and demonstrating its effectiveness on state-of-the-art conditional flow-based models for image super-resolution and speech synthesis.

Related papers

• Energy-Based Diffusion Language Models for Text Generation [126.23425882687195]
  Energy-based Diffusion Language Model (EDLM) is an energy-based model operating at the full sequence level for each diffusion step. Our framework offers a 1.3$times$ sampling speedup over existing diffusion models.
  arXiv  Detail & Related papers  (2024-10-28T17:25:56Z)
• Guided Flows for Generative Modeling and Decision Making [55.42634941614435]
  We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text synthesis-to-speech. Notably, we are first to apply flow models for plan generation in the offline reinforcement learning setting ax speedup in compared to diffusion models.
  arXiv  Detail & Related papers  (2023-11-22T15:07:59Z)
• Your Autoregressive Generative Model Can be Better If You Treat It as an Energy-Based One [83.5162421521224]
  We propose a unique method termed E-ARM for training autoregressive generative models. E-ARM takes advantage of a well-designed energy-based learning objective. We show that E-ARM can be trained efficiently and is capable of alleviating the exposure bias problem.
  arXiv  Detail & Related papers  (2022-06-26T10:58:41Z)
• Attentive Contractive Flow with Lipschitz-constrained Self-Attention [25.84621883831624]
  We introduce a novel approach called Attentive Contractive Flow (ACF) ACF utilizes a special category of flow-based generative models - contractive flows. We demonstrate that ACF can be introduced into a variety of state of the art flow models in a plug-and-play manner.
  arXiv  Detail & Related papers  (2021-09-24T18:02:49Z)
• Generative Flows with Invertible Attentions [135.23766216657745]
  We introduce two types of invertible attention mechanisms for generative flow models. We exploit split-based attention mechanisms to learn the attention weights and input representations on every two splits of flow feature maps. Our method provides invertible attention modules with tractable Jacobian determinants, enabling seamless integration of it at any positions of the flow-based models.
  arXiv  Detail & Related papers  (2021-06-07T20:43:04Z)
• Distilling Interpretable Models into Human-Readable Code [71.11328360614479]
  Human-readability is an important and desirable standard for machine-learned model interpretability. We propose to train interpretable models using conventional methods, and then distill them into concise, human-readable code. We describe a piecewise-linear curve-fitting algorithm that produces high-quality results efficiently and reliably across a broad range of use cases.
  arXiv  Detail & Related papers  (2021-01-21T01:46:36Z)
• Normalizing Flows with Multi-Scale Autoregressive Priors [131.895570212956]
  We introduce channel-wise dependencies in their latent space through multi-scale autoregressive priors (mAR) Our mAR prior for models with split coupling flow layers (mAR-SCF) can better capture dependencies in complex multimodal data. We show that mAR-SCF allows for improved image generation quality, with gains in FID and Inception scores compared to state-of-the-art flow-based models.
  arXiv  Detail & Related papers  (2020-04-08T09:07:11Z)
• Regularized Autoencoders via Relaxed Injective Probability Flow [35.39933775720789]
  Invertible flow-based generative models are an effective method for learning to generate samples, while allowing for tractable likelihood computation and inference. We propose a generative model based on probability flows that does away with the bijectivity requirement on the model and only assumes injectivity.
  arXiv  Detail & Related papers  (2020-02-20T18:22:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.