Unbiased Estimating Equation on Inverse Divergence and Its Conditions

• URL: http://arxiv.org/abs/2404.16519v1
• Date: Thu, 25 Apr 2024 11:22:48 GMT
• Title: Unbiased Estimating Equation on Inverse Divergence and Its Conditions
• Authors: Masahiro Kobayashi, Kazuho Watanabe,
• Abstract summary: This paper focuses on the Bregman divergence defined by the reciprocal function, called the inverse divergence. For the loss function defined by the monotonically increasing function $f$ and inverse divergence, the conditions for the statistical model and function $f$ under which the estimating equation is unbiased are clarified.
• Score: 0.10742675209112622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper focuses on the Bregman divergence defined by the reciprocal function, called the inverse divergence. For the loss function defined by the monotonically increasing function $f$ and inverse divergence, the conditions for the statistical model and function $f$ under which the estimating equation is unbiased are clarified. Specifically, we characterize two types of statistical models, an inverse Gaussian type and a mixture of generalized inverse Gaussian type distributions, to show that the conditions for the function $f$ are different for each model. We also define Bregman divergence as a linear sum over the dimensions of the inverse divergence and extend the results to the multi-dimensional case.

Related papers

• Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation [59.86921150579892]
  We deal with the problem of gradient estimation for differentiable relaxations of algorithms, operators, simulators, and other non-differentiable functions. We develop variance reduction strategies for differentiable sorting and ranking, differentiable shortest-paths on graphs, differentiable rendering for pose estimation, as well as differentiable cryo-ET simulations.
  arXiv  Detail & Related papers  (2024-10-10T17:10:00Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  This paper presents a succinct derivation of the training and generalization performance of a variety of high-dimensional ridge regression models. We provide an introduction and review of recent results on these topics, aimed at readers with backgrounds in physics and deep learning.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• Divergences induced by dual subtractive and divisive normalizations of exponential families and their convex deformations [7.070726553564701]
  We show that skewed Bhattacharryya distances between probability densities of an exponential family amounts to skewed Jensen divergences induced by the cumulant function. We then show how comparative convexity with respect to a pair of quasi-arithmetic means allows to deform both convex functions and their arguments.
  arXiv  Detail & Related papers  (2023-12-20T08:59:05Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• Understanding the bias-variance tradeoff of Bregman divergences [13.006468721874372]
  This paper builds upon the work of Pfau (2013), which generalized the bias variance tradeoff to any Bregman divergence loss function. We show that, similarly to the label, the central prediction can be interpreted as the mean of a random variable, where the mean operates in a dual space defined by the loss function itself.
  arXiv  Detail & Related papers  (2022-02-08T22:06:16Z)
• An Indirect Rate-Distortion Characterization for Semantic Sources: General Model and the Case of Gaussian Observation [83.93224401261068]
  Source model is motivated by the recent surge of interest in the semantic aspect of information. intrinsic state corresponds to the semantic feature of the source, which in general is not observable. Rate-distortion function is the semantic rate-distortion function of the source.
  arXiv  Detail & Related papers  (2022-01-29T02:14:24Z)
• On the Double Descent of Random Features Models Trained with SGD [78.0918823643911]
  We study properties of random features (RF) regression in high dimensions optimized by gradient descent (SGD) We derive precise non-asymptotic error bounds of RF regression under both constant and adaptive step-size SGD setting. We observe the double descent phenomenon both theoretically and empirically.
  arXiv  Detail & Related papers  (2021-10-13T17:47:39Z)
• Loss function based second-order Jensen inequality and its application to particle variational inference [112.58907653042317]
  Particle variational inference (PVI) uses an ensemble of models as an empirical approximation for the posterior distribution. PVI iteratively updates each model with a repulsion force to ensure the diversity of the optimized models. We derive a novel generalization error bound and show that it can be reduced by enhancing the diversity of models.
  arXiv  Detail & Related papers  (2021-06-09T12:13:51Z)
• Covariance Prediction via Convex Optimization [0.0]
  We describe a covariance predictor that has the form of a generalized linear model, i.e., an affine function of the features followed by an inverse link function. The log-likelihood is a concave function of the predictor parameters, so fitting the predictor involves convex optimization.
  arXiv  Detail & Related papers  (2021-01-29T06:06:58Z)
• Equivalence of Convergence Rates of Posterior Distributions and Bayes Estimators for Functions and Nonparametric Functionals [4.375582647111708]
  We study the posterior contraction rates of a Bayesian method with Gaussian process priors in nonparametric regression. For a general class of kernels, we establish convergence rates of the posterior measure of the regression function and its derivatives. Our proof shows that, under certain conditions, to any convergence rate of Bayes estimators there corresponds the same convergence rate of the posterior distributions.
  arXiv  Detail & Related papers  (2020-11-27T19:11:56Z)
• Unbiased Estimation Equation under $f$-Separable Bregman Distortion Measures [0.3553493344868413]
  We discuss unbiased estimation equations in a class of objective function using a monotonically increasing function $f$ and Bregman divergence. The choice of the function $f$ gives desirable properties such as robustness against outliers. In this study, we clarify the combination of Bregman divergence, statistical model, and function $f$ in which the bias correction term vanishes.
  arXiv  Detail & Related papers  (2020-10-23T10:33:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.