Brain-like variational inference

• URL: http://arxiv.org/abs/2410.19315v2
• Date: Fri, 16 May 2025 04:12:05 GMT
• Title: Brain-like variational inference
• Authors: Hadi Vafaii, Dekel Galor, Jacob L. Yates,
• Abstract summary: Inference in both brains and machines can be formalized by maximizing the evidence lower bound (ELBO) in machine learning, or minimizing variational free energy (F) in neuroscience (ELBO = -F)<n>Here, we show that online natural gradient descent on F, under Poisson assumptions, leads to a recurrent spiking neural network that performs variational inference via membrane potential dynamics.<n>The resulting model -- the iterative Poisson variational autoencoder (iP-VAE) -- replaces the encoder network with local updates derived from natural gradient descent on F.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inference in both brains and machines can be formalized by optimizing a shared objective: maximizing the evidence lower bound (ELBO) in machine learning, or minimizing variational free energy (F) in neuroscience (ELBO = -F). While this equivalence suggests a unifying framework, it leaves open how inference is implemented in neural systems. Here, we show that online natural gradient descent on F, under Poisson assumptions, leads to a recurrent spiking neural network that performs variational inference via membrane potential dynamics. The resulting model -- the iterative Poisson variational autoencoder (iP-VAE) -- replaces the encoder network with local updates derived from natural gradient descent on F. Theoretically, iP-VAE yields a number of desirable features such as emergent normalization via lateral competition, and hardware-efficient integer spike count representations. Empirically, iP-VAE outperforms both standard VAEs and Gaussian-based predictive coding models in sparsity, reconstruction, and biological plausibility. iP-VAE also exhibits strong generalization to out-of-distribution inputs, exceeding hybrid iterative-amortized VAEs. These results demonstrate how deriving inference algorithms from first principles can yield concrete architectures that are simultaneously biologically plausible and empirically effective.

Related papers

• A Non-negative VAE:the Generalized Gamma Belief Network [49.970917207211556]
  The gamma belief network (GBN) has demonstrated its potential for uncovering multi-layer interpretable latent representations in text data. We introduce the generalized gamma belief network (Generalized GBN) in this paper, which extends the original linear generative model to a more expressive non-linear generative model. We also propose an upward-downward Weibull inference network to approximate the posterior distribution of the latent variables.
  arXiv  Detail & Related papers  (2024-08-06T18:18:37Z)
• Poisson Variational Autoencoder [0.0]
  Variational autoencoders (VAEs) employ Bayesian inference to interpret sensory inputs.<n>Here, we develop a novel architecture that combines principles of predictive coding with a VAE that encodes inputs into discrete spike counts.<n>Our work provides an interpretable computational framework to study brain-like sensory processing.
  arXiv  Detail & Related papers  (2024-05-23T12:02:54Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• A Neural Network Implementation for Free Energy Principle [3.7937771805690392]
  The free energy principle (FEP) has been widely applied to account for various problems in fields such as cognitive science, neuroscience, social interaction, and hermeneutics. This paper gives a preliminary attempt at bridging FEP and machine learning, via a classical neural network model, the Helmholtz machine. Although the Helmholtz machine is not temporal, it gives an ideal parallel to the vanilla FEP and the hierarchical model of the brain, under which the active inference and predictive coding could be formulated coherently.
  arXiv  Detail & Related papers  (2023-06-11T22:14:21Z)
• Neural Frailty Machine: Beyond proportional hazard assumption in neural survival regressions [30.018173329118184]
  We present neural frailty machine (NFM), a powerful and flexible neural modeling framework for survival regressions. Two concrete models are derived under the framework that extends neural proportional hazard models and non hazard regression models. We conduct experimental evaluations over $6$ benchmark datasets of different scales, showing that the proposed NFM models outperform state-of-the-art survival models in terms of predictive performance.
  arXiv  Detail & Related papers  (2023-03-18T08:15:15Z)
• Bayesian Federated Neural Matching that Completes Full Information [2.6566593102111473]
  Federated learning is a machine learning paradigm where locally trained models are distilled into a global model. We propose a novel approach that overcomes this flaw by introducing a Kullback-Leibler divergence penalty at each iteration.
  arXiv  Detail & Related papers  (2022-11-15T09:47:56Z)
• On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
  Differentiable causal discovery has proposed to factorize the data generating process into a set of modules. We study the generalization and adaption performance of such modular neural causal models. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
  arXiv  Detail & Related papers  (2022-06-09T17:12:32Z)
• EINNs: Epidemiologically-Informed Neural Networks [75.34199997857341]
  We introduce a new class of physics-informed neural networks-EINN-crafted for epidemic forecasting. We investigate how to leverage both the theoretical flexibility provided by mechanistic models as well as the data-driven expressability afforded by AI models.
  arXiv  Detail & Related papers  (2022-02-21T18:59:03Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Efficient Semi-Implicit Variational Inference [65.07058307271329]
  We propose an efficient and scalable semi-implicit extrapolational (SIVI) Our method maps SIVI's evidence to a rigorous inference of lower gradient values.
  arXiv  Detail & Related papers  (2021-01-15T11:39:09Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)
• The Neural Coding Framework for Learning Generative Models [91.0357317238509]
  We propose a novel neural generative model inspired by the theory of predictive processing in the brain. In a similar way, artificial neurons in our generative model predict what neighboring neurons will do, and adjust their parameters based on how well the predictions matched reality.
  arXiv  Detail & Related papers  (2020-12-07T01:20:38Z)
• Iterative VAE as a predictive brain model for out-of-distribution generalization [7.006301658267125]
  We show that iVAEs generalize to distributional shifts significantly better than both PCNs and VAEs. We propose a novel measure of recognizability for individual samples which can be tested against human psychophysical data.
  arXiv  Detail & Related papers  (2020-12-01T15:02:38Z)
• Neural Networks with Recurrent Generative Feedback [61.90658210112138]
  We instantiate this design on convolutional neural networks (CNNs) In the experiments, CNN-F shows considerably improved adversarial robustness over conventional feedforward CNNs on standard benchmarks.
  arXiv  Detail & Related papers  (2020-07-17T19:32:48Z)
• Combining Differentiable PDE Solvers and Graph Neural Networks for Fluid Flow Prediction [79.81193813215872]
  We develop a hybrid (graph) neural network that combines a traditional graph convolutional network with an embedded differentiable fluid dynamics simulator inside the network itself. We show that we can both generalize well to new situations and benefit from the substantial speedup of neural network CFD predictions.
  arXiv  Detail & Related papers  (2020-07-08T21:23:19Z)
• Predictive Coding Approximates Backprop along Arbitrary Computation Graphs [68.8204255655161]
  We develop a strategy to translate core machine learning architectures into their predictive coding equivalents. Our models perform equivalently to backprop on challenging machine learning benchmarks. Our method raises the potential that standard machine learning algorithms could in principle be directly implemented in neural circuitry.
  arXiv  Detail & Related papers  (2020-06-07T15:35:47Z)
• Improved Protein-ligand Binding Affinity Prediction with Structure-Based Deep Fusion Inference [3.761791311908692]
  Predicting accurate protein-ligand binding affinity is important in drug discovery. Recent advances in the deep convolutional and graph neural network based approaches, the model performance depends on the input data representation. We present fusion models to benefit from different feature representations of two neural network models to improve the binding affinity prediction.
  arXiv  Detail & Related papers  (2020-05-17T22:26:27Z)
• Training Deep Energy-Based Models with f-Divergence Minimization [113.97274898282343]
  Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging. We propose a general variational framework termed f-EBM to train EBMs using any desired f-divergence. Experimental results demonstrate the superiority of f-EBM over contrastive divergence, as well as the benefits of training EBMs using f-divergences other than KL.
  arXiv  Detail & Related papers  (2020-03-06T23:11:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.