On the Properties of Kullback-Leibler Divergence Between Gaussians

• URL: http://arxiv.org/abs/2102.05485v1
• Date: Wed, 10 Feb 2021 15:21:53 GMT
• Title: On the Properties of Kullback-Leibler Divergence Between Gaussians
• Abstract summary: Kullback-Leibler (KL) divergence is one of the most important divergence measures between probability distributions. In this paper, we investigate the properties of KL divergence between Gaussians.
• Score: 22.274623046922628
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Kullback-Leibler (KL) divergence is one of the most important divergence measures between probability distributions. In this paper, we investigate the properties of KL divergence between Gaussians. Firstly, for any two $n$-dimensional Gaussians $\mathcal{N}_1$ and $\mathcal{N}_2$, we find the supremum of $KL(\mathcal{N}_1||\mathcal{N}_2)$ when $KL(\mathcal{N}_2||\mathcal{N}_1)\leq \epsilon$ for $\epsilon>0$. This reveals the approximate symmetry of small KL divergence between Gaussians. We also find the infimum of $KL(\mathcal{N}_1||\mathcal{N}_2)$ when $KL(\mathcal{N}_2||\mathcal{N}_1)\geq M$ for $M>0$. Secondly, for any three $n$-dimensional Gaussians $\mathcal{N}_1, \mathcal{N}_2$ and $\mathcal{N}_3$, we find a tight bound of $KL(\mathcal{N}_1||\mathcal{N}_3)$ if $KL(\mathcal{N}_1||\mathcal{N}_2)$ and $KL(\mathcal{N}_2||\mathcal{N}_3)$ are bounded. This reveals that the KL divergence between Gaussians follows a relaxed triangle inequality. Importantly, all the bounds in the theorems presented in this paper are independent of the dimension $n$.

Related papers

• On the convergence of graph Laplacians with a symmetric divergence [0.0]
  A key estimate for the geodesic distance $d_g$ is that there exists $K > 0$ such that $0 leq d_g(p, q)2 - |p-q|2 leq K d_g(p, q)4$ for all $p, q in mathcalM$.<n>We demonstrate that this is sufficient for the pointwise convergence of graph Laplacians constructed with $D$ and discuss examples where $D$ is given by the Sinkhorn divergence on a family of probability measures para
  arXiv  Detail & Related papers  (2026-07-07T06:41:21Z)
• Optimal Stability of KL Divergence under Gaussian Perturbations [30.190488055856207]
  existing relaxed triangle inequalities for KL divergence critically rely on the assumption that all involved distributions are Gaussian.<n>We establish a sharp stability bound between an arbitrary distribution and Gaussian families under mild moment conditions.<n>We prove that this $sqrt$ rate is optimal in general, even within the Gaussian family.
  arXiv  Detail & Related papers  (2026-04-13T05:49:59Z)
• Relaxed Triangle Inequality for Kullback-Leibler Divergence Between Multivariate Gaussian Distributions [40.868754235067804]
  The Kullback-Leibler (KL) divergence is not a proper metric and does not satisfy the triangle inequality.<n>We give the supremum of $KL(mathcalN1), mathcalN_3)$ as well as the conditions when the supremum can be attained.
  arXiv  Detail & Related papers  (2026-01-31T09:28:44Z)
• On the Capacity Region of Individual Key Rates in Vector Linear Secure Aggregation [55.126702858312456]
  We show that it is not necessary for every user to hold a key, thereby strictly enlarging the best-known achievable region in the literature.<n>Our results uncover the novel fact that it is not necessary for every user to hold a key, thereby strictly enlarging the best-known achievable region in the literature.
  arXiv  Detail & Related papers  (2026-01-06T18:34:07Z)
• Algorithmic contiguity from low-degree conjecture and applications in correlated random graphs [0.0]
  We provide evidence of computational hardness for two problems. One of the main ingredient in our proof is to derive certain forms of emphalgorithm contiguity between two probability measures. This framework provides a useful tool for performing reductions between different tasks.
  arXiv  Detail & Related papers  (2025-02-14T00:24:51Z)
• The Communication Complexity of Approximating Matrix Rank [50.6867896228563]
  We show that this problem has randomized communication complexity $Omega(frac1kcdot n2log|mathbbF|)$. As an application, we obtain an $Omega(frac1kcdot n2log|mathbbF|)$ space lower bound for any streaming algorithm with $k$ passes.
  arXiv  Detail & Related papers  (2024-10-26T06:21:42Z)
• Exploring topological entanglement through Dehn surgery [1.3328842853079743]
  We compute the partition function of a closed 3-manifold obtained from Dehn fillings of the link complement. We have given explicit results for all hyperbolic knots $mathcalK$ up to six crossings.
  arXiv  Detail & Related papers  (2024-02-12T07:38:14Z)
• Provably learning a multi-head attention layer [55.2904547651831]
  Multi-head attention layer is one of the key components of the transformer architecture that sets it apart from traditional feed-forward models. In this work, we initiate the study of provably learning a multi-head attention layer from random examples. We prove computational lower bounds showing that in the worst case, exponential dependence on $m$ is unavoidable.
  arXiv  Detail & Related papers  (2024-02-06T15:39:09Z)
• SQ Lower Bounds for Learning Mixtures of Linear Classifiers [43.63696593768504]
  We show that known algorithms for this problem are essentially best possible, even for the special case of uniform mixtures. The key technical ingredient is a new construction of spherical designs that may be of independent interest.
  arXiv  Detail & Related papers  (2023-10-18T10:56:57Z)
• Rigorous derivation of the Efimov effect in a simple model [68.8204255655161]
  We consider a system of three identical bosons in $mathbbR3$ with two-body zero-range interactions and a three-body hard-core repulsion of a given radius $a>0$.
  arXiv  Detail & Related papers  (2023-06-21T10:11:28Z)
• A random matrix model for random approximate $t$-designs [1.534667887016089]
  A random matrix model is proposed to describe the probability distribution of $delta(nu_mathcalS,t)$ for any $t$. We show that our model satisfies the so-called spectral gap conjecture, i.e. we prove that with the probability $1$ there is $tinmathbbZ_+$ such that $sup satisfieskinmathbbZ_+delta(k)=delta(t)$.
  arXiv  Detail & Related papers  (2022-10-14T14:50:06Z)
• Monogamy of entanglement between cones [68.8204255655161]
  We show that monogamy is not only a feature of quantum theory, but that it characterizes the minimal tensor product of general pairs of convex cones. Our proof makes use of a new characterization of products of simplices up to affine equivalence.
  arXiv  Detail & Related papers  (2022-06-23T16:23:59Z)
• On the speed of uniform convergence in Mercer's theorem [6.028247638616059]
  A continuous positive definite kernel $K(mathbf x, mathbf y)$ on a compact set can be represented as $sum_i=1infty lambda_iphi_i(mathbf x)phi_i(mathbf y)$ where $(lambda_i,phi_i)$ are eigenvalue-eigenvector pairs of the corresponding integral operator. We estimate the speed of this convergence in terms of the decay rate of eigenvalues and demonstrate that for $3m
  arXiv  Detail & Related papers  (2022-05-01T15:07:57Z)
• Uncertainties in Quantum Measurements: A Quantum Tomography [52.77024349608834]
  The observables associated with a quantum system $S$ form a non-commutative algebra $mathcal A_S$. It is assumed that a density matrix $rho$ can be determined from the expectation values of observables. Abelian algebras do not have inner automorphisms, so the measurement apparatus can determine mean values of observables.
  arXiv  Detail & Related papers  (2021-12-14T16:29:53Z)
• Topological entanglement and hyperbolic volume [1.1909611351044664]
  Chern-Simons theory provides setting to visualise the $m$-moment of reduced density matrix as a three-manifold invariant $Z(M_mathcalK_m)$. For SU(2) group, we show that $Z(M_mathcalK_m)$ can grow at mostly in $k$. We conjecture that $ln Z(M_mathcalK_m)$ is the hyperbolic volume of the knot complement $S3backslash mathcalK_m
  arXiv  Detail & Related papers  (2021-06-07T07:51:03Z)
• Linear Bandits on Uniformly Convex Sets [88.3673525964507]
  Linear bandit algorithms yield $tildemathcalO(nsqrtT)$ pseudo-regret bounds on compact convex action sets. Two types of structural assumptions lead to better pseudo-regret bounds.
  arXiv  Detail & Related papers  (2021-03-10T07:33:03Z)
• Near-Optimal SQ Lower Bounds for Agnostically Learning Halfspaces and ReLUs under Gaussian Marginals [49.60752558064027]
  We study the fundamental problems of agnostically learning halfspaces and ReLUs under Gaussian marginals. Our lower bounds provide strong evidence that current upper bounds for these tasks are essentially best possible.
  arXiv  Detail & Related papers  (2020-06-29T17:10:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.