Adaptive learning of density ratios in RKHS

• URL: http://arxiv.org/abs/2307.16164v3
• Date: Tue, 2 Jan 2024 09:32:23 GMT
• Title: Adaptive learning of density ratios in RKHS
• Authors: Werner Zellinger, Stefan Kindermann, Sergei V. Pereverzyev
• Abstract summary: Estimating the ratio of two probability densities from finitely many observations is a central problem in machine learning and statistics. We analyze a large class of density ratio estimation methods that minimize a regularized Bregman divergence between the true density ratio and a model in a reproducing kernel Hilbert space.
• Score: 3.047411947074805
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating the ratio of two probability densities from finitely many observations of the densities is a central problem in machine learning and statistics with applications in two-sample testing, divergence estimation, generative modeling, covariate shift adaptation, conditional density estimation, and novelty detection. In this work, we analyze a large class of density ratio estimation methods that minimize a regularized Bregman divergence between the true density ratio and a model in a reproducing kernel Hilbert space (RKHS). We derive new finite-sample error bounds, and we propose a Lepskii type parameter choice principle that minimizes the bounds without knowledge of the regularity of the density ratio. In the special case of quadratic loss, our method adaptively achieves a minimax optimal error rate. A numerical illustration is provided.

Related papers

• Overcoming Saturation in Density Ratio Estimation by Iterated Regularization [11.244546184962996]
  We show that a class of kernel methods for density ratio estimation suffers from error saturation. We introduce iterated regularization in density ratio estimation to achieve fast error rates.
  arXiv  Detail & Related papers  (2024-02-21T16:02:14Z)
• Sobolev Space Regularised Pre Density Models [51.558848491038916]
  We propose a new approach to non-parametric density estimation that is based on regularizing a Sobolev norm of the density. This method is statistically consistent, and makes the inductive validation model clear and consistent.
  arXiv  Detail & Related papers  (2023-07-25T18:47:53Z)
• Anomaly Detection with Variance Stabilized Density Estimation [49.46356430493534]
  We present a variance-stabilized density estimation problem for maximizing the likelihood of the observed samples. To obtain a reliable anomaly detector, we introduce a spectral ensemble of autoregressive models for learning the variance-stabilized distribution. We have conducted an extensive benchmark with 52 datasets, demonstrating that our method leads to state-of-the-art results.
  arXiv  Detail & Related papers  (2023-06-01T11:52:58Z)
• MCD: Marginal Contrastive Discrimination for conditional density estimation [0.0]
  Marginal Contrastive Discrimination, MCD, reformulates the conditional density function into two factors, the marginal density function of the target variable and a ratio of density functions. Our benchmark reveals that our method significantly outperforms in practice existing methods on most density models and regression datasets.
  arXiv  Detail & Related papers  (2022-06-03T14:22:29Z)
• A Unified Framework for Multi-distribution Density Ratio Estimation [101.67420298343512]
  Binary density ratio estimation (DRE) provides the foundation for many state-of-the-art machine learning algorithms. We develop a general framework from the perspective of Bregman minimization divergence. We show that our framework leads to methods that strictly generalize their counterparts in binary DRE.
  arXiv  Detail & Related papers  (2021-12-07T01:23:20Z)
• Density Ratio Estimation via Infinitesimal Classification [85.08255198145304]
  We propose DRE-infty, a divide-and-conquer approach to reduce Density ratio estimation (DRE) to a series of easier subproblems. Inspired by Monte Carlo methods, we smoothly interpolate between the two distributions via an infinite continuum of intermediate bridge distributions. We show that our approach performs well on downstream tasks such as mutual information estimation and energy-based modeling on complex, high-dimensional datasets.
  arXiv  Detail & Related papers  (2021-11-22T06:26:29Z)
• Information Theoretic Structured Generative Modeling [13.117829542251188]
  A novel generative model framework called the structured generative model (SGM) is proposed that makes straightforward optimization possible. The implementation employs a single neural network driven by an orthonormal input to a single white noise source adapted to learn an infinite Gaussian mixture model. Preliminary results show that SGM significantly improves MINE estimation in terms of data efficiency and variance, conventional and variational Gaussian mixture models, as well as for training adversarial networks.
  arXiv  Detail & Related papers  (2021-10-12T07:44:18Z)
• Meta-Learning for Relative Density-Ratio Estimation [59.75321498170363]
  Existing methods for (relative) density-ratio estimation (DRE) require many instances from both densities. We propose a meta-learning method for relative DRE, which estimates the relative density-ratio from a few instances by using knowledge in related datasets. We empirically demonstrate the effectiveness of the proposed method by using three problems: relative DRE, dataset comparison, and outlier detection.
  arXiv  Detail & Related papers  (2021-07-02T02:13:45Z)
• Continuous Wasserstein-2 Barycenter Estimation without Minimax Optimization [94.18714844247766]
  Wasserstein barycenters provide a geometric notion of the weighted average of probability measures based on optimal transport. We present a scalable algorithm to compute Wasserstein-2 barycenters given sample access to the input measures.
  arXiv  Detail & Related papers  (2021-02-02T21:01:13Z)
• Reducing the Variance of Variational Estimates of Mutual Information by Limiting the Critic's Hypothesis Space to RKHS [0.0]
  Mutual information (MI) is an information-theoretic measure of dependency between two random variables. Recent methods realize parametric probability distributions or critic as a neural network to approximate unknown density ratios. We argue that the high variance characteristic is due to the uncontrolled complexity of the critic's hypothesis space.
  arXiv  Detail & Related papers  (2020-11-17T14:32:48Z)
• Telescoping Density-Ratio Estimation [21.514983459970903]
  We introduce a new framework, telescoping density-ratio estimation (TRE) TRE enables the estimation of ratios between highly dissimilar densities in high-dimensional spaces. Our experiments demonstrate that TRE can yield substantial improvements over existing single-ratio methods.
  arXiv  Detail & Related papers  (2020-06-22T12:55:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.