Optimal Bounds between $f$-Divergences and Integral Probability Metrics

• URL: http://arxiv.org/abs/2006.05973v3
• Date: Sat, 5 Jun 2021 19:47:28 GMT
• Title: Optimal Bounds between $f$-Divergences and Integral Probability Metrics
• Authors: Rohit Agrawal, Thibaut Horel
• Abstract summary: Families of $f$-divergences and Integral Probability Metrics are widely used to quantify similarity between probability distributions. We systematically study the relationship between these two families from the perspective of convex duality. We obtain new bounds while also recovering in a unified manner well-known results, such as Hoeffding's lemma.
• Score: 8.401473551081748
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The families of $f$-divergences (e.g. the Kullback-Leibler divergence) and Integral Probability Metrics (e.g. total variation distance or maximum mean discrepancies) are widely used to quantify the similarity between probability distributions. In this work, we systematically study the relationship between these two families from the perspective of convex duality. Starting from a tight variational representation of the $f$-divergence, we derive a generalization of the moment-generating function, which we show exactly characterizes the best lower bound of the $f$-divergence as a function of a given IPM. Using this characterization, we obtain new bounds while also recovering in a unified manner well-known results, such as Hoeffding's lemma, Pinsker's inequality and its extension to subgaussian functions, and the Hammersley-Chapman-Robbins bound. This characterization also allows us to prove new results on topological properties of the divergence which may be of independent interest.

Related papers

• Statistical Inference of Optimal Allocations I: Regularities and their Implications [3.904240476752459]
  We first derive Hadamard differentiability of the value function through a detailed analysis of the general properties of the sorting operator. Building on our Hadamard differentiability results, we demonstrate how the functional delta method can be used to directly derive the properties of the value function process.
  arXiv  Detail & Related papers  (2024-03-27T04:39:13Z)
• Divergences induced by dual subtractive and divisive normalizations of exponential families and their convex deformations [7.070726553564701]
  We show that skewed Bhattacharryya distances between probability densities of an exponential family amounts to skewed Jensen divergences induced by the cumulant function. We then show how comparative convexity with respect to a pair of quasi-arithmetic means allows to deform both convex functions and their arguments.
  arXiv  Detail & Related papers  (2023-12-20T08:59:05Z)
• High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance [59.211456992422136]
  We propose algorithms with high-probability convergence results under less restrictive assumptions. These results justify the usage of the considered methods for solving problems that do not fit standard functional classes in optimization.
  arXiv  Detail & Related papers  (2023-02-02T10:37:23Z)
• Kernel-based off-policy estimation without overlap: Instance optimality beyond semiparametric efficiency [53.90687548731265]
  We study optimal procedures for estimating a linear functional based on observational data. For any convex and symmetric function class $mathcalF$, we derive a non-asymptotic local minimax bound on the mean-squared error.
  arXiv  Detail & Related papers  (2023-01-16T02:57:37Z)
• Function-space regularized R\'enyi divergences [6.221019624345409]
  We propose a new family of regularized R'enyi divergences parametrized by a variational function space. We prove several properties of these new divergences, showing that they interpolate between the classical R'enyi divergences and IPMs. We show that the proposed regularized R'enyi divergences inherit features from IPMs such as the ability to compare distributions that are not absolutely continuous.
  arXiv  Detail & Related papers  (2022-10-10T19:18:04Z)
• Data-Driven Influence Functions for Optimization-Based Causal Inference [105.5385525290466]
  We study a constructive algorithm that approximates Gateaux derivatives for statistical functionals by finite differencing. We study the case where probability distributions are not known a priori but need to be estimated from data.
  arXiv  Detail & Related papers  (2022-08-29T16:16:22Z)
• Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiency [111.83670279016599]
  We study reinforcement learning for partially observed decision processes (POMDPs) with infinite observation and state spaces. We make the first attempt at partial observability and function approximation for a class of POMDPs with a linear structure.
  arXiv  Detail & Related papers  (2022-04-20T21:15:38Z)
• Generalization Bounds via Convex Analysis [12.411844611718958]
  We show that it is possible to replace the mutual information by any strongly convex function of the joint input-output distribution. Examples include bounds stated in terms of $p$-norm divergences and the Wasserstein-2 distance.
  arXiv  Detail & Related papers  (2022-02-10T12:30:45Z)
• R\'enyi divergence inequalities via interpolation, with applications to generalised entropic uncertainty relations [91.3755431537592]
  We investigate quantum R'enyi entropic quantities, specifically those derived from'sandwiched' divergence. We present R'enyi mutual information decomposition rules, a new approach to the R'enyi conditional entropy tripartite chain rules and a more general bipartite comparison.
  arXiv  Detail & Related papers  (2021-06-19T04:06:23Z)
• Federated Functional Gradient Boosting [75.06942944563572]
  We study functional minimization in Federated Learning. For both FFGB.C and FFGB.L, the radii of convergence shrink to zero as the feature distributions become more homogeneous.
  arXiv  Detail & Related papers  (2021-03-11T21:49:19Z)
• $(f,\Gamma)$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics [6.221019624345409]
  We develop a framework for constructing information-theoretic divergences that subsume both $f$-divergences and integral probability metrics (IPMs) We show that they can be expressed as a two-stage mass-redistribution/mass-transport process. Using statistical learning as an example, we demonstrate their advantage in training generative adversarial networks (GANs) for heavy-tailed, not-absolutely continuous sample distributions.
  arXiv  Detail & Related papers  (2020-11-11T18:17:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.