Estimates on Learning Rates for Multi-Penalty Distribution Regression
- URL: http://arxiv.org/abs/2006.09017v2
- Date: Wed, 29 Nov 2023 04:20:33 GMT
- Title: Estimates on Learning Rates for Multi-Penalty Distribution Regression
- Authors: Zhan Yu, Daniel W. C. Ho
- Abstract summary: We study a multi-penalty regularization algorithm for distribution regression under the framework of learning theory.
We embed the distributions to reproducing a kernel Hilbert space $mathcalH_K$ associated with Mercer kernel $K$ via mean embedding technique.
The work also derives learning rates for distribution regression in the nonstandard setting $f_rhonotinmathcalH_K$, which is not explored in existing literature.
- Score: 5.999239529678357
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper is concerned with functional learning by utilizing two-stage
sampled distribution regression. We study a multi-penalty regularization
algorithm for distribution regression under the framework of learning theory.
The algorithm aims at regressing to real valued outputs from probability
measures. The theoretical analysis on distribution regression is far from
maturity and quite challenging, since only second stage samples are observable
in practical setting. In the algorithm, to transform information from samples,
we embed the distributions to a reproducing kernel Hilbert space
$\mathcal{H}_K$ associated with Mercer kernel $K$ via mean embedding technique.
The main contribution of the paper is to present a novel multi-penalty
regularization algorithm to capture more features of distribution regression
and derive optimal learning rates for the algorithm. The work also derives
learning rates for distribution regression in the nonstandard setting
$f_{\rho}\notin\mathcal{H}_K$, which is not explored in existing literature.
Moreover, we propose a distribution regression-based distributed learning
algorithm to face large-scale data or information challenge. The optimal
learning rates are derived for the distributed learning algorithm. By providing
new algorithms and showing their learning rates, we improve the existing work
in different aspects in the literature.
Related papers
- A Mirror Descent-Based Algorithm for Corruption-Tolerant Distributed Gradient Descent [57.64826450787237]
We show how to analyze the behavior of distributed gradient descent algorithms in the presence of adversarial corruptions.
We show how to use ideas from (lazy) mirror descent to design a corruption-tolerant distributed optimization algorithm.
Experiments based on linear regression, support vector classification, and softmax classification on the MNIST dataset corroborate our theoretical findings.
arXiv Detail & Related papers (2024-07-19T08:29:12Z) - Distributional Reinforcement Learning with Dual Expectile-Quantile Regression [51.87411935256015]
quantile regression approach to distributional RL provides flexible and effective way of learning arbitrary return distributions.
We show that distributional guarantees vanish, and we empirically observe that the estimated distribution rapidly collapses to its mean estimation.
Motivated by the efficiency of $L$-based learning, we propose to jointly learn expectiles and quantiles of the return distribution in a way that allows efficient learning while keeping an estimate of the full distribution of returns.
arXiv Detail & Related papers (2023-05-26T12:30:05Z) - Domain Generalization by Functional Regression [3.209698860006188]
We study domain generalization as a problem of functional regression.
Our concept leads to a new algorithm for learning a linear operator from marginal distributions of inputs to the corresponding conditional distributions of outputs given inputs.
arXiv Detail & Related papers (2023-02-09T16:07:21Z) - A Polynomial Time Algorithm for Learning Halfspaces with Tsybakov Noise [55.45544638410858]
We study the problem of PAC learning homogeneous halfspaces in the presence of Tsybakov noise.
Our algorithm learns the true halfspace within any accuracy $epsilon$.
arXiv Detail & Related papers (2020-10-04T22:19:06Z) - Distributional Reinforcement Learning via Moment Matching [54.16108052278444]
We formulate a method that learns a finite set of statistics from each return distribution via neural networks.
Our method can be interpreted as implicitly matching all orders of moments between a return distribution and its Bellman target.
Experiments on the suite of Atari games show that our method outperforms the standard distributional RL baselines.
arXiv Detail & Related papers (2020-07-24T05:18:17Z) - Covariate Distribution Aware Meta-learning [3.494950334697974]
We propose a computationally feasible meta-learning algorithm by introducing meaningful relaxations.
We demonstrate the gains of our algorithm over bootstrapped based meta-learning baselines on popular classification benchmarks.
arXiv Detail & Related papers (2020-07-06T05:00:13Z) - Meta-learning with Stochastic Linear Bandits [120.43000970418939]
We consider a class of bandit algorithms that implement a regularized version of the well-known OFUL algorithm, where the regularization is a square euclidean distance to a bias vector.
We show both theoretically and experimentally, that when the number of tasks grows and the variance of the task-distribution is small, our strategies have a significant advantage over learning the tasks in isolation.
arXiv Detail & Related papers (2020-05-18T08:41:39Z) - A Distributional Analysis of Sampling-Based Reinforcement Learning
Algorithms [67.67377846416106]
We present a distributional approach to theoretical analyses of reinforcement learning algorithms for constant step-sizes.
We show that value-based methods such as TD($lambda$) and $Q$-Learning have update rules which are contractive in the space of distributions of functions.
arXiv Detail & Related papers (2020-03-27T05:13:29Z) - Scalable Approximate Inference and Some Applications [2.6541211006790983]
In this thesis, we propose a new framework for approximate inference.
Our proposed four algorithms are motivated by the recent computational progress of Stein's method.
Results on simulated and real datasets indicate the statistical efficiency and wide applicability of our algorithm.
arXiv Detail & Related papers (2020-03-07T04:33:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.