Entropy-Informed Weighting Channel Normalizing Flow

• URL: http://arxiv.org/abs/2407.04958v1
• Date: Sat, 6 Jul 2024 04:46:41 GMT
• Title: Entropy-Informed Weighting Channel Normalizing Flow
• Authors: Wei Chen, Shian Du, Shigui Li, Delu Zeng, John Paisley,
• Abstract summary: We propose a regularized and feature-dependent $mathttShuffle$ operation and integrate it into vanilla multi-scale architecture. We observe that such operation guides the variables to evolve in the direction of entropy increase, hence we refer to NFs with the $mathttShuffle$ operation as emphEntropy-Informed Weighting Channel Normalizing Flow (EIW-Flow)
• Score: 7.751853409569806
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Normalizing Flows (NFs) have gained popularity among deep generative models due to their ability to provide exact likelihood estimation and efficient sampling. However, a crucial limitation of NFs is their substantial memory requirements, arising from maintaining the dimension of the latent space equal to that of the input space. Multi-scale architectures bypass this limitation by progressively reducing the dimension of latent variables while ensuring reversibility. Existing multi-scale architectures split the latent variables in a simple, static manner at the channel level, compromising NFs' expressive power. To address this issue, we propose a regularized and feature-dependent $\mathtt{Shuffle}$ operation and integrate it into vanilla multi-scale architecture. This operation heuristically generates channel-wise weights and adaptively shuffles latent variables before splitting them with these weights. We observe that such operation guides the variables to evolve in the direction of entropy increase, hence we refer to NFs with the $\mathtt{Shuffle}$ operation as \emph{Entropy-Informed Weighting Channel Normalizing Flow} (EIW-Flow). Experimental results indicate that the EIW-Flow achieves state-of-the-art density estimation results and comparable sample quality on CIFAR-10, CelebA and ImageNet datasets, with negligible additional computational overhead.

Related papers

• Transformer Neural Autoregressive Flows [48.68932811531102]
  Density estimation can be performed using Normalizing Flows (NFs) We propose a novel solution by exploiting transformers to define a new class of neural flows called Transformer Neural Autoregressive Flows (T-NAFs)
  arXiv  Detail & Related papers  (2024-01-03T17:51:16Z)
• Taming Hyperparameter Tuning in Continuous Normalizing Flows Using the JKO Scheme [60.79981399724534]
  A normalizing flow (NF) is a mapping that transforms a chosen probability distribution to a normal distribution. We present JKO-Flow, an algorithm to solve OT-based CNF without the need of tuning $alpha$.
  arXiv  Detail & Related papers  (2022-11-30T05:53:21Z)
• Deep Equilibrium Optical Flow Estimation [80.80992684796566]
  Recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms. These RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. We propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer.
  arXiv  Detail & Related papers  (2022-04-18T17:53:44Z)
• Discretely Indexed Flows [1.0079626733116611]
  We propose Discretely Indexed flows (DIF) as a new tool for solving variational estimation problems. DIF are built as an extension of Normalizing Flows (NF), in which the deterministic transport becomes discretely indexed. They benefit from both a tractable density as well as a straightforward sampling scheme, and can thus be used for the dual problems of Variational Inference (VI) and of Variational density estimation (VDE)
  arXiv  Detail & Related papers  (2022-04-04T10:13:43Z)
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers [55.90468016961356]
  We propose an efficient token mixer that learns to mix in the Fourier domain. AFNO is based on a principled foundation of operator learning. It can handle a sequence size of 65k and outperforms other efficient self-attention mechanisms.
  arXiv  Detail & Related papers  (2021-11-24T05:44:31Z)
• Self Normalizing Flows [65.73510214694987]
  We propose a flexible framework for training normalizing flows by replacing expensive terms in the gradient by learned approximate inverses at each layer. This reduces the computational complexity of each layer's exact update from $mathcalO(D3)$ to $mathcalO(D2)$. We show experimentally that such models are remarkably stable and optimize to similar data likelihood values as their exact gradient counterparts.
  arXiv  Detail & Related papers  (2020-11-14T09:51:51Z)
• OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport [8.468007443062751]
  A normalizing flow is an invertible mapping between an arbitrary probability distribution and a standard normal distribution. OT-Flow tackles two critical computational challenges that limit a more widespread use of CNFs. On five high-dimensional density estimation and generative modeling tasks, OT-Flow performs competitively to state-of-the-art CNFs.
  arXiv  Detail & Related papers  (2020-05-29T22:31:10Z)
• Learning Likelihoods with Conditional Normalizing Flows [54.60456010771409]
  Conditional normalizing flows (CNFs) are efficient in sampling and inference. We present a study of CNFs where the base density to output space mapping is conditioned on an input x, to model conditional densities p(y|x)
  arXiv  Detail & Related papers  (2019-11-29T19:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.