KL Divergence Between Gaussians: A Step-by-Step Derivation for the Variational Autoencoder Objective

• URL: http://arxiv.org/abs/2604.11744v1
• Date: Mon, 13 Apr 2026 17:18:20 GMT
• Title: KL Divergence Between Gaussians: A Step-by-Step Derivation for the Variational Autoencoder Objective
• Abstract summary: Kullback-Leibler (KL) divergence quantifies discrepancy between two probability distributions.<n>In the context of Variational Autoencoders (VAEs) it serves as a central regularization term.
• Score: 0.5801044612920815
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Kullback-Leibler (KL) divergence is a fundamental concept in information theory that quantifies the discrepancy between two probability distributions. In the context of Variational Autoencoders (VAEs), it serves as a central regularization term, imposing structure on the latent space and thereby enabling the model to exhibit generative capabilities. In this work, we present a detailed derivation of the closed-form expression for the KL divergence between Gaussian distributions, a case of particular importance in practical VAE implementations. Starting from the general definition for continuous random variables, we derive the expression for the univariate case and extend it to the multivariate setting under the assumption of diagonal covariance. Finally, we discuss the interpretation of each term in the resulting expression and its impact on the training dynamics of the model.

Related papers

• Amortized Variational Inference for Joint Posterior and Predictive Distributions in Bayesian Uncertainty Quantification [1.0742675209112622]
  We propose a variational Bayesian framework that directly targets the posterior-predictive distribution and jointly learns variational approximations of both the posterior and the corresponding predictive distribution.<n>The proposed method achieves more accurate predictive distributions than conventional two-stage variational inference, while substantially reducing the cost of online predictive inference.
  arXiv  Detail & Related papers  (2026-05-05T12:56:00Z)
• An Elementary Approach to Scheduling in Generative Diffusion Models [55.171367482496755]
  An elementary approach to characterizing the impact of noise scheduling and time discretization in generative diffusion models is developed.<n> Experiments across different datasets and pretrained models demonstrate that the time discretization strategy selected by our approach consistently outperforms baseline and search-based strategies.
  arXiv  Detail & Related papers  (2026-01-20T05:06:26Z)
• Diffusion Models: A Mathematical Introduction [3.8673630752805437]
  We present a self-contained derivation of diffusion-based generative models.<n>We construct denoising diffusion probabilistic models from first principles.<n>Readers can both follow the theory and implement the corresponding algorithms in practice.
  arXiv  Detail & Related papers  (2025-11-13T16:20:52Z)
• Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis [56.442307356162864]
  We study the theoretical aspects of score-based discrete diffusion models under the Continuous Time Markov Chain (CTMC) framework.<n>We introduce a discrete-time sampling algorithm in the general state space $[S]d$ that utilizes score estimators at predefined time points.<n>Our convergence analysis employs a Girsanov-based method and establishes key properties of the discrete score function.
  arXiv  Detail & Related papers  (2024-10-03T09:07:13Z)
• Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models [59.331993845831946]
  Diffusion models benefit from instillation of task-specific information into the score function to steer the sample generation towards desired properties. This paper provides the first theoretical study towards understanding the influence of guidance on diffusion models in the context of Gaussian mixture models.
  arXiv  Detail & Related papers  (2024-03-03T23:15:48Z)
• Variational excess risk bound for general state space models [0.0]
  We consider variational autoencoders (VAE) for general state space models. We consider a backward factorization of the variational distributions to analyze the excess risk associated with VAE.
  arXiv  Detail & Related papers  (2023-12-15T08:41:07Z)
• Causal Modeling with Stationary Diffusions [89.94899196106223]
  We learn differential equations whose stationary densities model a system's behavior under interventions. We show that they generalize to unseen interventions on their variables, often better than classical approaches. Our inference method is based on a new theoretical result that expresses a stationarity condition on the diffusion's generator in a reproducing kernel Hilbert space.
  arXiv  Detail & Related papers  (2023-10-26T14:01:17Z)
• How to train your VAE [0.0]
  Variational Autoencoders (VAEs) have become a cornerstone in generative modeling and representation learning within machine learning. This paper explores interpreting the Kullback-Leibler (KL) Divergence, a critical component within the Evidence Lower Bound (ELBO) The proposed method redefines the ELBO with a mixture of Gaussians for the posterior probability, introduces a regularization term, and employs a PatchGAN discriminator to enhance texture realism.
  arXiv  Detail & Related papers  (2023-09-22T19:52:28Z)
• The Inverse of Exact Renormalization Group Flows as Statistical Inference [0.0]
  We build on the view of the Exact Renormalization Group (ERG) as an instantiation of Optimal Transport. We provide a new information theoretic perspective for understanding the ERG through the intermediary of Bayesian Statistical Inference.
  arXiv  Detail & Related papers  (2022-12-21T21:38:34Z)
• Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
  We study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
  arXiv  Detail & Related papers  (2022-04-07T17:59:19Z)
• Latent Causal Invariant Model [128.7508609492542]
  Current supervised learning can learn spurious correlation during the data-fitting process. We propose a Latent Causal Invariance Model (LaCIM) which pursues causal prediction.
  arXiv  Detail & Related papers  (2020-11-04T10:00:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.