Optimal Stability of KL Divergence under Gaussian Perturbations

• URL: http://arxiv.org/abs/2604.11026v3
• Date: Thu, 16 Apr 2026 15:03:23 GMT
• Title: Optimal Stability of KL Divergence under Gaussian Perturbations
• Abstract summary: existing relaxed triangle inequalities for KL divergence critically rely on the assumption that all involved distributions are Gaussian.<n>We establish a sharp stability bound between an arbitrary distribution and Gaussian families under mild moment conditions.<n>We prove that this $sqrt$ rate is optimal in general, even within the Gaussian family.
• Score: 30.190488055856207
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We study the problem of characterizing the stability of Kullback-Leibler (KL) divergence under Gaussian perturbations beyond Gaussian families. Existing relaxed triangle inequalities for KL divergence critically rely on the assumption that all involved distributions are Gaussian, which limits their applicability in modern applications such as out-of-distribution (OOD) detection with flow-based generative models. In this paper, we remove this restriction by establishing a sharp stability bound between an arbitrary distribution and Gaussian families under mild moment conditions. Specifically, let $P$ be a distribution with finite second moment, and let $\mathcal{N}_1$ and $\mathcal{N}_2$ be multivariate Gaussian distributions. We show that if $KL(P||\mathcal{N}_1)$ is large and $KL(\mathcal{N}_1||\mathcal{N}_2)$ is at most $ε$, then $KL(P||\mathcal{N}_2) \ge KL(P||\mathcal{N}_1) - O(\sqrtε)$. Moreover, we prove that this $\sqrtε$ rate is optimal in general, even within the Gaussian family. This result reveals an intrinsic stability property of KL divergence under Gaussian perturbations, extending classical Gaussian-only relaxed triangle inequalities to general distributions. The result is non-trivial due to the asymmetry of KL divergence and the absence of a triangle inequality in general probability spaces. As an application, we provide a rigorous foundation for KL-based OOD analysis in flow-based models, removing strong Gaussian assumptions used in prior work. More broadly, our result enables KL-based reasoning in non-Gaussian settings arising in deep learning and reinforcement learning.

Related papers

• Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability [54.40598524756038]
  emphKullback-Leibler (KL) regularization is ubiquitous in reinforcement learning algorithms.<n>Recent studies have demonstrated $-1$-type fast rates for decision making under reverse KL regularization.<n>We take the first step towards addressing this problem via a streamlined analysis of forward-KL-regularized offline CBs.
  arXiv  Detail & Related papers  (2026-05-09T23:17:46Z)
• Exponential families from a single KL identity [7.106986689736826]
  Exponential families encompass the distributions central to modern machine learning -- softmax, Gaussians, and Boltzmann distributions.<n>We isolate a simple identity for exponential families that expresses the KL difference $mathrmKL(q | p_|)$ in terms of the log-partition function $A()$ and the moment $_q$.<n>Remarkably, this identity together with the single fact that $mathrmKL geq 0$ suffices, by direct substitution and rearrangement, to derive a cluster of results that are classically obtained
  arXiv  Detail & Related papers  (2026-04-30T15:48:03Z)
• Dimension-Independent Convergence of Underdamped Langevin Monte Carlo in KL Divergence [50.719298242863744]
  Underdamped Langevin dynamics (ULD) is a widely-used sampler for Gibbs distributions $propto e-V$.<n>We prove the first dimension-free KL divergence bounds for discretized ULD.
  arXiv  Detail & Related papers  (2026-03-02T22:14:38Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Steady-State Behavior of Constant-Stepsize Stochastic Approximation: Gaussian Approximation and Tail Bounds [9.739744677824286]
  Constant-stepsize approximation (SA) is widely used in learning for computational efficiency.<n>Prior work shows that as the stepsize $downarrow 0$, the centered-and-scaled steady state converges weakly to a Gaussian random vector.<n>This paper provides explicit, non-asymptotic error bounds for fixed $$.
  arXiv  Detail & Related papers  (2026-02-15T02:34:50Z)
• Relaxed Triangle Inequality for Kullback-Leibler Divergence Between Multivariate Gaussian Distributions [40.868754235067804]
  The Kullback-Leibler (KL) divergence is not a proper metric and does not satisfy the triangle inequality.<n>We give the supremum of $KL(mathcalN1), mathcalN_3)$ as well as the conditions when the supremum can be attained.
  arXiv  Detail & Related papers  (2026-01-31T09:28:44Z)
• Which exceptional low-dimensional projections of a Gaussian point cloud can be found in polynomial time? [12.775038922010452]
  We study the subset $mathscrF_m,alpha$ of distributions that can be realized by a class of iterative algorithms.<n>Non-rigorous methods from statistical physics yield an indirect characterization of $mathscrF_m,alpha$ in terms of a generalized Parisi formula.
  arXiv  Detail & Related papers  (2024-06-05T05:54:56Z)
• Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications [2.0584253077707477]
  We study the convergence properties of the Gradient Descent (SGD) method for finding a stationary point of an objective function $J(cdot)$. Our results apply to a class of invex'' functions, which have the property that every stationary point is also a global minimizer.
  arXiv  Detail & Related papers  (2023-12-05T15:22:39Z)
• The Schr\"odinger Bridge between Gaussian Measures has a Closed Form [101.79851806388699]
  We focus on the dynamic formulation of OT, also known as the Schr"odinger bridge (SB) problem. In this paper, we provide closed-form expressions for SBs between Gaussian measures.
  arXiv  Detail & Related papers  (2022-02-11T15:59:01Z)
• Non-Gaussian Component Analysis via Lattice Basis Reduction [56.98280399449707]
  Non-Gaussian Component Analysis (NGCA) is a distribution learning problem. We provide an efficient algorithm for NGCA in the regime that $A$ is discrete or nearly discrete.
  arXiv  Detail & Related papers  (2021-12-16T18:38:02Z)
• Convergence of Random Reshuffling Under The Kurdyka-{\L}ojasiewicz Inequality [3.960041987749073]
  We conduct convergence analysis for the non-descent RR with diminishing step sizes based on the Kurdyka-Lojasiewicz (KL) inequality. We derive the corresponding rate of convergence depending on the KL exponent and the suitably selected diminishing step sizes. We also establish similar strong limit-point convergence results for the proximal point method.
  arXiv  Detail & Related papers  (2021-10-10T23:20:04Z)
• On the Properties of Kullback-Leibler Divergence Between Gaussians [22.274623046922628]
  Kullback-Leibler (KL) divergence is one of the most important divergence measures between probability distributions. In this paper, we investigate the properties of KL divergence between Gaussians.
  arXiv  Detail & Related papers  (2021-02-10T15:21:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.