FlowVAT: Normalizing Flow Variational Inference with Affine-Invariant Tempering

• URL: http://arxiv.org/abs/2505.10466v1
• Date: Thu, 15 May 2025 16:20:36 GMT
• Title: FlowVAT: Normalizing Flow Variational Inference with Affine-Invariant Tempering
• Authors: Juehang Qin, Shixiao Liang, Christopher Tunnell,
• Abstract summary: Multi-modal and high-dimensional posteriors present significant challenges for variational inference.<n>We introduce FlowVAT, a conditional tempering approach for normalizing flow variational inference.<n>Our method tempers both the base and target distributions simultaneously, maintaining affine-invariance under tempering.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-modal and high-dimensional posteriors present significant challenges for variational inference, causing mode-seeking behavior and collapse despite the theoretical expressiveness of normalizing flows. Traditional annealing methods require temperature schedules and hyperparameter tuning, falling short of the goal of truly black-box variational inference. We introduce FlowVAT, a conditional tempering approach for normalizing flow variational inference that addresses these limitations. Our method tempers both the base and target distributions simultaneously, maintaining affine-invariance under tempering. By conditioning the normalizing flow on temperature, we leverage overparameterized neural networks' generalization capabilities to train a single flow representing the posterior across a range of temperatures. This preserves modes identified at higher temperatures when sampling from the variational posterior at $T = 1$, mitigating standard variational methods' mode-seeking behavior. In experiments with 2, 10, and 20 dimensional multi-modal distributions, FlowVAT outperforms traditional and adaptive annealing methods, finding more modes and achieving better ELBO values, particularly in higher dimensions where existing approaches fail. Our method requires minimal hyperparameter tuning and does not require an annealing schedule, advancing toward fully-automatic black-box variational inference for complicated posteriors.

Related papers

• Rectified Flows for Fast Multiscale Fluid Flow Modeling [11.597597438962026]
  We introduce a rectified flow framework that learns a time-dependent velocity field.<n>Our method makes each integration step much more effective, using as few as eight steps.<n> Experiments on challenging multiscale flow benchmarks show that rectified flows recover the same posterior distributions as diffusion models.
  arXiv  Detail & Related papers  (2025-06-03T17:40:39Z)
• Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts [64.34482582690927]
  We provide an efficient and principled method for sampling from a sequence of annealed, geometric-averaged, or product distributions derived from pretrained score-based models.<n>We propose Sequential Monte Carlo (SMC) resampling algorithms that leverage inference-time scaling to improve sampling quality.
  arXiv  Detail & Related papers  (2025-03-04T17:46:51Z)
• Policy Gradients for Optimal Parallel Tempering MCMC [0.276240219662896]
  Parallel tempering is a meta-algorithm for Markov Chain Monte Carlo that uses multiple chains to sample from tempered versions of the target distribution.<n>We present an adaptive temperature selection algorithm that dynamically adjusts temperatures during sampling using a policy gradient approach.
  arXiv  Detail & Related papers  (2024-09-03T03:12:45Z)
• Flow matching achieves almost minimax optimal convergence [50.38891696297888]
  Flow matching (FM) has gained significant attention as a simulation-free generative model. This paper discusses the convergence properties of FM for large sample size under the $p$-Wasserstein distance. We establish that FM can achieve an almost minimax optimal convergence rate for $1 leq p leq 2$, presenting the first theoretical evidence that FM can reach convergence rates comparable to those of diffusion models.
  arXiv  Detail & Related papers  (2024-05-31T14:54:51Z)
• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [54.20763128054692]
  We study the dynamics of gradient flow for training a multi-head softmax attention model for in-context learning of multi-task linear regression. We prove that an interesting "task allocation" phenomenon emerges during the gradient flow dynamics.
  arXiv  Detail & Related papers  (2024-02-29T18:43:52Z)
• Stable Training of Normalizing Flows for High-dimensional Variational Inference [2.139348034155473]
  Variational inference with normalizing flows (NFs) is an increasingly popular alternative to MCMC methods. In practice, training deep normalizing flows for approximating high-dimensional distributions is often infeasible due to the high variance of the gradients. We show that previous methods for stabilizing the variance of gradient descent can be insufficient to achieve stable training of Real NVPs.
  arXiv  Detail & Related papers  (2024-02-26T09:04:07Z)
• Time-changed normalizing flows for accurate SDE modeling [5.402030962296633]
  We propose a novel transformation of dynamic normalizing flows, based on time deformation of a Brownian motion. This approach enables us to effectively model some SDEs, that cannot be modeled otherwise.
  arXiv  Detail & Related papers  (2023-12-22T13:57:29Z)
• Detecting and Mitigating Mode-Collapse for Flow-based Sampling of Lattice Field Theories [6.222204646855336]
  We study the consequences of mode-collapse of normalizing flows in the context of lattice field theory. We propose a metric to quantify the degree of mode-collapse and derive a bound on the resulting bias.
  arXiv  Detail & Related papers  (2023-02-27T19:00:22Z)
• Efficient CDF Approximations for Normalizing Flows [64.60846767084877]
  We build upon the diffeomorphic properties of normalizing flows to estimate the cumulative distribution function (CDF) over a closed region. Our experiments on popular flow architectures and UCI datasets show a marked improvement in sample efficiency as compared to traditional estimators.
  arXiv  Detail & Related papers  (2022-02-23T06:11:49Z)
• The Variational Method of Moments [65.91730154730905]
  conditional moment problem is a powerful formulation for describing structural causal parameters in terms of observables. Motivated by a variational minimax reformulation of OWGMM, we define a very general class of estimators for the conditional moment problem. We provide algorithms for valid statistical inference based on the same kind of variational reformulations.
  arXiv  Detail & Related papers  (2020-12-17T07:21:06Z)
• SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows [78.77808270452974]
  SurVAE Flows is a modular framework for composable transformations that encompasses VAEs and normalizing flows. We show that several recently proposed methods, including dequantization and augmented normalizing flows, can be expressed as SurVAE Flows.
  arXiv  Detail & Related papers  (2020-07-06T13:13:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.