LEAPS: A discrete neural sampler via locally equivariant networks

• URL: http://arxiv.org/abs/2502.10843v1
• Date: Sat, 15 Feb 2025 16:16:45 GMT
• Title: LEAPS: A discrete neural sampler via locally equivariant networks
• Authors: Peter Holderrieth, Michael S. Albergo, Tommi Jaakkola,
• Abstract summary: We propose LEAPS, an algorithm to sample from discrete distributions known up to normalization by learning a rate matrix of a continuous-time Markov chain (CTMC)<n> LEAPS can be seen as a continuous-time formulation of annealed importance sampling and sequential Monte Carlo methods, extended so that the variance of the importance weights is offset by the inclusion of the CTMC.<n>We demonstrate the efficacy of LEAPS on problems in statistical physics.
• Score: 3.5032660973169727
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose LEAPS, an algorithm to sample from discrete distributions known up to normalization by learning a rate matrix of a continuous-time Markov chain (CTMC). LEAPS can be seen as a continuous-time formulation of annealed importance sampling and sequential Monte Carlo methods, extended so that the variance of the importance weights is offset by the inclusion of the CTMC. To derive these importance weights, we introduce a set of Radon-Nikodym derivatives of CTMCs over their path measures. Because the computation of these weights is intractable with standard neural network parameterizations of rate matrices, we devise a new compact representation for rate matrices via what we call locally equivariant functions. To parameterize them, we introduce a family of locally equivariant multilayer perceptrons, attention layers, and convolutional networks, and provide an approach to make deep networks that preserve the local equivariance. This property allows us to propose a scalable training algorithm for the rate matrix such that the variance of the importance weights associated to the CTMC are minimal. We demonstrate the efficacy of LEAPS on problems in statistical physics.

Related papers

• Discrete Neural Flow Samplers with Locally Equivariant Transformer [25.911046280803586]
  We propose Discrete Neural Flow Samplers (DNFS), a trainable and efficient framework for discrete sampling.<n>DNFS learns the rate matrix of a continuous-time Markov chain such that the resulting dynamics satisfy the Kolmogorov equation.<n>To further facilitate computational efficiency, we propose locally equivaraint Transformer, a novel parameterisation of the rate matrix.
  arXiv  Detail & Related papers  (2025-05-23T11:06:06Z)
• Diagonal Symmetrization of Neural Network Solvers for the Many-Electron Schrödinger Equation [11.202098800341096]
  We study different ways of incorporating diagonal invariance in neural network ans"atze trained via variational Monte Carlo methods.<n>We show that, contrary to standard ML setups, in-training symmetrization destabilizes training and can lead to worse performance.<n>Our theoretical and numerical results indicate that this unexpected behavior may arise from a unique computational-statistical tradeoff not found in standard ML analyses of symmetrization.
  arXiv  Detail & Related papers  (2025-02-07T20:37:25Z)
• Symmetry-driven embedding of networks in hyperbolic space [0.4779196219827508]
  Hyperbolic models can reproduce the heavy-tailed degree distribution, high clustering, and hierarchical structure of empirical networks. Current algorithms for finding the hyperbolic coordinates of networks, however, do not quantify uncertainty in the inferred coordinates. We present BIGUE, a Markov chain Monte Carlo algorithm that samples the posterior distribution of a Bayesian hyperbolic random graph model.
  arXiv  Detail & Related papers  (2024-06-15T18:44:02Z)
• Weakly supervised covariance matrices alignment through Stiefel matrices estimation for MEG applications [64.20396555814513]
  This paper introduces a novel domain adaptation technique for time series data, called Mixing model Stiefel Adaptation (MSA) We exploit abundant unlabeled data in the target domain to ensure effective prediction by establishing pairwise correspondence with equivalent signal variances between domains. MSA outperforms recent methods in brain-age regression with task variations using magnetoencephalography (MEG) signals from the Cam-CAN dataset.
  arXiv  Detail & Related papers  (2024-01-24T19:04:49Z)
• Online Variational Sequential Monte Carlo [49.97673761305336]
  We build upon the variational sequential Monte Carlo (VSMC) method, which provides computationally efficient and accurate model parameter estimation and Bayesian latent-state inference. Online VSMC is capable of performing efficiently, entirely on-the-fly, both parameter estimation and particle proposal adaptation.
  arXiv  Detail & Related papers  (2023-12-19T21:45:38Z)
• Adaptive Annealed Importance Sampling with Constant Rate Progress [68.8204255655161]
  Annealed Importance Sampling (AIS) synthesizes weighted samples from an intractable distribution. We propose the Constant Rate AIS algorithm and its efficient implementation for $alpha$-divergences.
  arXiv  Detail & Related papers  (2023-06-27T08:15:28Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• How can spherical CNNs benefit ML-based diffusion MRI parameter estimation? [2.4417196796959906]
  Spherical convolutional neural networks (S-CNN) offer distinct advantages over conventional fully-connected networks (FCN) Current clinical practice commonly acquires dMRI data consisting of only 6 diffusion weighted images (DWIs)
  arXiv  Detail & Related papers  (2022-07-01T17:49:26Z)
• Distributed support-vector-machine over dynamic balanced directed networks [10.76210145983805]
  We consider the binary classification problem via distributed Support-Machines. We propose a continuous-time algorithm that incorporates network topology changes in discrete jumps.
  arXiv  Detail & Related papers  (2021-04-01T11:02:10Z)
• Sampling in Combinatorial Spaces with SurVAE Flow Augmented MCMC [83.48593305367523]
  Hybrid Monte Carlo is a powerful Markov Chain Monte Carlo method for sampling from complex continuous distributions. We introduce a new approach based on augmenting Monte Carlo methods with SurVAE Flows to sample from discrete distributions. We demonstrate the efficacy of our algorithm on a range of examples from statistics, computational physics and machine learning, and observe improvements compared to alternative algorithms.
  arXiv  Detail & Related papers  (2021-02-04T02:21:08Z)
• Controllable Orthogonalization in Training DNNs [96.1365404059924]
  Orthogonality is widely used for training deep neural networks (DNNs) due to its ability to maintain all singular values of the Jacobian close to 1. This paper proposes a computationally efficient and numerically stable orthogonalization method using Newton's iteration (ONI) We show that our method improves the performance of image classification networks by effectively controlling the orthogonality to provide an optimal tradeoff between optimization benefits and representational capacity reduction. We also show that ONI stabilizes the training of generative adversarial networks (GANs) by maintaining the Lipschitz continuity of a network, similar to spectral normalization (
  arXiv  Detail & Related papers  (2020-04-02T10:14:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.