Overcoming Saturation in Density Ratio Estimation by Iterated Regularization

• URL: http://arxiv.org/abs/2402.13891v2
• Date: Mon, 3 Jun 2024 11:40:32 GMT
• Title: Overcoming Saturation in Density Ratio Estimation by Iterated Regularization
• Authors: Lukas Gruber, Markus Holzleitner, Johannes Lehner, Sepp Hochreiter, Werner Zellinger,
• Abstract summary: We show that a class of kernel methods for density ratio estimation suffers from error saturation. We introduce iterated regularization in density ratio estimation to achieve fast error rates.
• Score: 11.244546184962996
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimating the ratio of two probability densities from finitely many samples, is a central task in machine learning and statistics. In this work, we show that a large class of kernel methods for density ratio estimation suffers from error saturation, which prevents algorithms from achieving fast error convergence rates on highly regular learning problems. To resolve saturation, we introduce iterated regularization in density ratio estimation to achieve fast error rates. Our methods outperform its non-iteratively regularized versions on benchmarks for density ratio estimation as well as on large-scale evaluations for importance-weighted ensembling of deep unsupervised domain adaptation models.

Related papers

• Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
  We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data. Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
  arXiv  Detail & Related papers  (2024-04-24T09:04:36Z)
• Double Debiased Covariate Shift Adaptation Robust to Density-Ratio Estimation [7.8856737627153874]
  We propose a doubly robust estimator for covariate shift adaptation via importance weighting. Our estimator reduces the bias arising from the density ratio estimation errors. Notably, our estimator remains consistent if either the density ratio estimator or the regression function is consistent.
  arXiv  Detail & Related papers  (2023-10-25T13:38:29Z)
• Adaptive learning of density ratios in RKHS [3.047411947074805]
  Estimating the ratio of two probability densities from finitely many observations is a central problem in machine learning and statistics. We analyze a large class of density ratio estimation methods that minimize a regularized Bregman divergence between the true density ratio and a model in a reproducing kernel Hilbert space.
  arXiv  Detail & Related papers  (2023-07-30T08:18:39Z)
• Sobolev Space Regularised Pre Density Models [51.558848491038916]
  We propose a new approach to non-parametric density estimation that is based on regularizing a Sobolev norm of the density. This method is statistically consistent, and makes the inductive validation model clear and consistent.
  arXiv  Detail & Related papers  (2023-07-25T18:47:53Z)
• Anomaly Detection with Variance Stabilized Density Estimation [49.46356430493534]
  We present a variance-stabilized density estimation problem for maximizing the likelihood of the observed samples. To obtain a reliable anomaly detector, we introduce a spectral ensemble of autoregressive models for learning the variance-stabilized distribution. We have conducted an extensive benchmark with 52 datasets, demonstrating that our method leads to state-of-the-art results.
  arXiv  Detail & Related papers  (2023-06-01T11:52:58Z)
• On double-descent in uncertainty quantification in overparametrized models [24.073221004661427]
  Uncertainty quantification is a central challenge in reliable and trustworthy machine learning. We show a trade-off between classification accuracy and calibration, unveiling a double descent like behavior in the calibration curve of optimally regularized estimators. This is in contrast with the empirical Bayes method, which we show to be well calibrated in our setting despite the higher generalization error and overparametrization.
  arXiv  Detail & Related papers  (2022-10-23T16:01:08Z)
• A Unified Framework for Multi-distribution Density Ratio Estimation [101.67420298343512]
  Binary density ratio estimation (DRE) provides the foundation for many state-of-the-art machine learning algorithms. We develop a general framework from the perspective of Bregman minimization divergence. We show that our framework leads to methods that strictly generalize their counterparts in binary DRE.
  arXiv  Detail & Related papers  (2021-12-07T01:23:20Z)
• Density Ratio Estimation via Infinitesimal Classification [85.08255198145304]
  We propose DRE-infty, a divide-and-conquer approach to reduce Density ratio estimation (DRE) to a series of easier subproblems. Inspired by Monte Carlo methods, we smoothly interpolate between the two distributions via an infinite continuum of intermediate bridge distributions. We show that our approach performs well on downstream tasks such as mutual information estimation and energy-based modeling on complex, high-dimensional datasets.
  arXiv  Detail & Related papers  (2021-11-22T06:26:29Z)
• Featurized Density Ratio Estimation [82.40706152910292]
  In our work, we propose to leverage an invertible generative model to map the two distributions into a common feature space prior to estimation. This featurization brings the densities closer together in latent space, sidestepping pathological scenarios where the learned density ratios in input space can be arbitrarily inaccurate. At the same time, the invertibility of our feature map guarantees that the ratios computed in feature space are equivalent to those in input space.
  arXiv  Detail & Related papers  (2021-07-05T18:30:26Z)
• TREND: Truncated Generalized Normal Density Estimation of Inception Embeddings for Accurate GAN Evaluation [27.80517509528215]
  Frech'et Inception distance is one of the most widely used metrics for evaluation of GANs. We argue that this is an over-simplified assumption, which may lead to unreliable evaluation results. We propose a novel metric for accurate evaluation of GANs, named TREND (TRuncated gEneralized Normal Density estimation of inception embeddings)
  arXiv  Detail & Related papers  (2021-04-30T05:51:07Z)
• Telescoping Density-Ratio Estimation [21.514983459970903]
  We introduce a new framework, telescoping density-ratio estimation (TRE) TRE enables the estimation of ratios between highly dissimilar densities in high-dimensional spaces. Our experiments demonstrate that TRE can yield substantial improvements over existing single-ratio methods.
  arXiv  Detail & Related papers  (2020-06-22T12:55:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.