Free-form Flows: Make Any Architecture a Normalizing Flow

• URL: http://arxiv.org/abs/2310.16624v2
• Date: Wed, 24 Apr 2024 10:05:18 GMT
• Title: Free-form Flows: Make Any Architecture a Normalizing Flow
• Authors: Felix Draxler, Peter Sorrenson, Lea Zimmermann, Armand Rousselot, Ullrich Köthe,
• Abstract summary: We develop a training procedure that uses an efficient estimator for the gradient of the change of variables formula. This enables any dimension-preserving neural network to serve as a generative model through maximum likelihood training. We achieve excellent results in molecule generation benchmarks utilizing $E(n)$-equivariant networks.
• Score: 8.163244519983298
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Normalizing Flows are generative models that directly maximize the likelihood. Previously, the design of normalizing flows was largely constrained by the need for analytical invertibility. We overcome this constraint by a training procedure that uses an efficient estimator for the gradient of the change of variables formula. This enables any dimension-preserving neural network to serve as a generative model through maximum likelihood training. Our approach allows placing the emphasis on tailoring inductive biases precisely to the task at hand. Specifically, we achieve excellent results in molecule generation benchmarks utilizing $E(n)$-equivariant networks. Moreover, our method is competitive in an inverse problem benchmark, while employing off-the-shelf ResNet architectures.

Related papers

• Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch [72.26822499434446]
  Auto-Train-Once (ATO) is an innovative network pruning algorithm designed to automatically reduce the computational and storage costs of DNNs. We provide a comprehensive convergence analysis as well as extensive experiments, and the results show that our approach achieves state-of-the-art performance across various model architectures.
  arXiv  Detail & Related papers  (2024-03-21T02:33:37Z)
• Guided Flows for Generative Modeling and Decision Making [55.42634941614435]
  We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text synthesis-to-speech. Notably, we are first to apply flow models for plan generation in the offline reinforcement learning setting ax speedup in compared to diffusion models.
  arXiv  Detail & Related papers  (2023-11-22T15:07:59Z)
• Kernelised Normalising Flows [10.31916245015817]
  Normalising Flows are non-parametric statistical models characterised by their dual capabilities of density estimation and generation. We present Ferumal flow, a novel kernelised normalising flow paradigm that integrates kernels into the framework.
  arXiv  Detail & Related papers  (2023-07-27T13:18:52Z)
• Training Energy-Based Normalizing Flow with Score-Matching Objectives [36.0810550035231]
  We present a new flow-based modeling approach called energy-based normalizing flow (EBFlow) We demonstrate that by optimizing EBFlow with score-matching objectives, the computation of Jacobian determinants for linear transformations can be entirely bypassed.
  arXiv  Detail & Related papers  (2023-05-24T15:54:29Z)
• Normalizing flow neural networks by JKO scheme [22.320632565424745]
  We develop a neural ODE flow network called JKO-iFlow, inspired by the Jordan-Kinderleherer-Otto scheme. The proposed method stacks residual blocks one after another, allowing efficient block-wise training of the residual blocks. Experiments with synthetic and real data show that the proposed JKO-iFlow network achieves competitive performance.
  arXiv  Detail & Related papers  (2022-12-29T18:55:00Z)
• Invertible Monotone Operators for Normalizing Flows [7.971699294672282]
  Normalizing flows model probability by learning invertible transformations that transfer a simple distribution into complex distributions. We propose the monotone formulation to overcome the issue of the Lipschitz constants using monotone operators. The resulting model, Monotone Flows, exhibits an excellent performance on multiple density estimation benchmarks.
  arXiv  Detail & Related papers  (2022-10-15T03:40:46Z)
• Deep Equilibrium Optical Flow Estimation [80.80992684796566]
  Recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms. These RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. We propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer.
  arXiv  Detail & Related papers  (2022-04-18T17:53:44Z)
• Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation [101.22379613810881]
  We consider data-driven optimization problems where one must maximize a function given only queries at a fixed set of points. This problem setting emerges in many domains where function evaluation is a complex and expensive process. We propose a tractable approximation that allows us to scale our method to high-capacity neural network models.
  arXiv  Detail & Related papers  (2021-02-16T06:04:27Z)
• Self Normalizing Flows [65.73510214694987]
  We propose a flexible framework for training normalizing flows by replacing expensive terms in the gradient by learned approximate inverses at each layer. This reduces the computational complexity of each layer's exact update from $mathcalO(D3)$ to $mathcalO(D2)$. We show experimentally that such models are remarkably stable and optimize to similar data likelihood values as their exact gradient counterparts.
  arXiv  Detail & Related papers  (2020-11-14T09:51:51Z)
• Learning Likelihoods with Conditional Normalizing Flows [54.60456010771409]
  Conditional normalizing flows (CNFs) are efficient in sampling and inference. We present a study of CNFs where the base density to output space mapping is conditioned on an input x, to model conditional densities p(y|x)
  arXiv  Detail & Related papers  (2019-11-29T19:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.