Invertible DenseNets with Concatenated LipSwish
- URL: http://arxiv.org/abs/2102.02694v1
- Date: Thu, 4 Feb 2021 15:45:33 GMT
- Title: Invertible DenseNets with Concatenated LipSwish
- Authors: Yura Perugachi-Diaz, Jakub M. Tomczak, Sandjai Bhulai
- Abstract summary: We introduce Invertible Dense Networks (i-DenseNets) as a more efficient alternative to Residual Flows.
We show that the proposed model out-performs Residual Flows when trained as a hybrid model where the model is both a generative and a discriminative model.
- Score: 8.315801422499861
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce Invertible Dense Networks (i-DenseNets), a more parameter
efficient alternative to Residual Flows. The method relies on an analysis of
the Lipschitz continuity of the concatenation in DenseNets, where we enforce
invertibility of the network by satisfying the Lipschitz constant. We extend
this method by proposing a learnable concatenation, which not only improves the
model performance but also indicates the importance of the concatenated
representation. Additionally, we introduce the Concatenated LipSwish as
activation function, for which we show how to enforce the Lipschitz condition
and which boosts performance. The new architecture, i-DenseNet, out-performs
Residual Flow and other flow-based models on density estimation evaluated in
bits per dimension, where we utilize an equal parameter budget. Moreover, we
show that the proposed model out-performs Residual Flows when trained as a
hybrid model where the model is both a generative and a discriminative model.
Related papers
- Fisher Flow Matching for Generative Modeling over Discrete Data [12.69975914345141]
We introduce Fisher-Flow, a novel flow-matching model for discrete data.
Fisher-Flow takes a manifestly geometric perspective by considering categorical distributions over discrete data.
We prove that the gradient flow induced by Fisher-Flow is optimal in reducing the forward KL divergence.
arXiv Detail & Related papers (2024-05-23T15:02:11Z) - Guided Flows for Generative Modeling and Decision Making [55.42634941614435]
We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text synthesis-to-speech.
Notably, we are first to apply flow models for plan generation in the offline reinforcement learning setting ax speedup in compared to diffusion models.
arXiv Detail & Related papers (2023-11-22T15:07:59Z) - Model Merging by Uncertainty-Based Gradient Matching [70.54580972266096]
We propose a new uncertainty-based scheme to improve the performance by reducing the mismatch.
Our new method gives consistent improvements for large language models and vision transformers.
arXiv Detail & Related papers (2023-10-19T15:02:45Z) - Precision-Recall Divergence Optimization for Generative Modeling with
GANs and Normalizing Flows [54.050498411883495]
We develop a novel training method for generative models, such as Generative Adversarial Networks and Normalizing Flows.
We show that achieving a specified precision-recall trade-off corresponds to minimizing a unique $f$-divergence from a family we call the textitPR-divergences.
Our approach improves the performance of existing state-of-the-art models like BigGAN in terms of either precision or recall when tested on datasets such as ImageNet.
arXiv Detail & Related papers (2023-05-30T10:07:17Z) - Invertible Monotone Operators for Normalizing Flows [7.971699294672282]
Normalizing flows model probability by learning invertible transformations that transfer a simple distribution into complex distributions.
We propose the monotone formulation to overcome the issue of the Lipschitz constants using monotone operators.
The resulting model, Monotone Flows, exhibits an excellent performance on multiple density estimation benchmarks.
arXiv Detail & Related papers (2022-10-15T03:40:46Z) - Lipschitz Continuity Retained Binary Neural Network [52.17734681659175]
We introduce the Lipschitz continuity as the rigorous criteria to define the model robustness for BNN.
We then propose to retain the Lipschitz continuity as a regularization term to improve the model robustness.
Our experiments prove that our BNN-specific regularization method can effectively strengthen the robustness of BNN.
arXiv Detail & Related papers (2022-07-13T22:55:04Z) - Attentive Contractive Flow with Lipschitz-constrained Self-Attention [25.84621883831624]
We introduce a novel approach called Attentive Contractive Flow (ACF)
ACF utilizes a special category of flow-based generative models - contractive flows.
We demonstrate that ACF can be introduced into a variety of state of the art flow models in a plug-and-play manner.
arXiv Detail & Related papers (2021-09-24T18:02:49Z) - Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
Implicit neural networks show improved accuracy and significant reduction in memory consumption.
They can suffer from ill-posedness and convergence instability.
This paper provides a new framework to design well-posed and robust implicit neural networks.
arXiv Detail & Related papers (2021-06-06T18:05:02Z) - Invertible DenseNets [7.895232155155041]
We introduce Invertible Dense Networks (i-DenseNets) as a more efficient alternative to Residual Flows.
We demonstrate the performance of i-DenseNets and Residual Flows on toy, MNIST, and CIFAR10 data.
arXiv Detail & Related papers (2020-10-05T16:11:39Z) - Lipschitz Recurrent Neural Networks [100.72827570987992]
We show that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs.
Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks.
arXiv Detail & Related papers (2020-06-22T08:44:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.