A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning

• URL: http://arxiv.org/abs/2401.02325v2
• Date: Sun, 7 Jan 2024 22:22:33 GMT
• Title: A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning
• Authors: Parvin Malekzadeh, Konstantinos N. Plataniotis, Zissis Poulos, Zeyu Wang
• Abstract summary: This paper introduces a generalized quantile Huber loss function derived from Wasserstein distance (WD) calculation. Compared to the classical quantile Huber loss, this innovative loss function enhances robustness against outliers. Empirical tests on Atari games, a common application in distributional RL, and a recent hedging strategy using distributional RL, validate our proposed loss function.
• Score: 19.89141873890568
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distributional Reinforcement Learning (RL) estimates return distribution mainly by learning quantile values via minimizing the quantile Huber loss function, entailing a threshold parameter often selected heuristically or via hyperparameter search, which may not generalize well and can be suboptimal. This paper introduces a generalized quantile Huber loss function derived from Wasserstein distance (WD) calculation between Gaussian distributions, capturing noise in predicted (current) and target (Bellman-updated) quantile values. Compared to the classical quantile Huber loss, this innovative loss function enhances robustness against outliers. Notably, the classical Huber loss function can be seen as an approximation of our proposed loss, enabling parameter adjustment by approximating the amount of noise in the data during the learning process. Empirical tests on Atari games, a common application in distributional RL, and a recent hedging strategy using distributional RL, validate the effectiveness of our proposed loss function and its potential for parameter adjustments in distributional RL. The implementation of the proposed loss function is available here.

Related papers

• Semiparametric conformal prediction [79.6147286161434]
  Risk-sensitive applications require well-calibrated prediction sets over multiple, potentially correlated target variables. We treat the scores as random vectors and aim to construct the prediction set accounting for their joint correlation structure. We report desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification [1.3778851745408134]
  We propose a novel ensemble method, namely EnsLoss, to combine loss functions within the Empirical risk minimization framework. We first transform the CC conditions of losses into loss-derivatives, thereby bypassing the need for explicit loss functions. We theoretically establish the statistical consistency of our approach and provide insights into its benefits.
  arXiv  Detail & Related papers  (2024-09-02T02:40:42Z)
• Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise [51.87307904567702]
  Quantile regression is a leading approach for obtaining such intervals via the empirical estimation of quantiles in the distribution of outputs. We propose Relaxed Quantile Regression (RQR), a direct alternative to quantile regression based interval construction that removes this arbitrary constraint. We demonstrate that this added flexibility results in intervals with an improvement in desirable qualities.
  arXiv  Detail & Related papers  (2024-06-05T13:36:38Z)
• Variance-Reducing Couplings for Random Features [57.73648780299374]
  Random features (RFs) are a popular technique to scale up kernel methods in machine learning. We find couplings to improve RFs defined on both Euclidean and discrete input spaces. We reach surprising conclusions about the benefits and limitations of variance reduction as a paradigm.
  arXiv  Detail & Related papers  (2024-05-26T12:25:09Z)
• Robust Non-parametric Knowledge-based Diffusion Least Mean Squares over Adaptive Networks [12.266804067030455]
  The proposed algorithm leads to a robust estimation of an unknown parameter vector in a group of cooperative estimators. Results show the robustness of the proposed algorithm in the presence of different noise types.
  arXiv  Detail & Related papers  (2023-12-03T06:18:59Z)
• Distributional Reinforcement Learning with Dual Expectile-Quantile Regression [51.87411935256015]
  quantile regression approach to distributional RL provides flexible and effective way of learning arbitrary return distributions. We show that distributional guarantees vanish, and we empirically observe that the estimated distribution rapidly collapses to its mean estimation. Motivated by the efficiency of $L$-based learning, we propose to jointly learn expectiles and quantiles of the return distribution in a way that allows efficient learning while keeping an estimate of the full distribution of returns.
  arXiv  Detail & Related papers  (2023-05-26T12:30:05Z)
• A Jensen-Shannon Divergence Based Loss Function for Bayesian Neural Networks [0.0]
  We formulate a novel loss function for BNNs based on the geometric JS divergence and show that the conventional KL divergence-based loss function is its special case. We demonstrate performance improvements over the state-of-the-art KL divergence-based BNN on the classification of a noisy CIFAR data set.
  arXiv  Detail & Related papers  (2022-09-23T01:47:09Z)
• Statistical Properties of the log-cosh Loss Function Used in Machine Learning [0.0]
  We present the distribution function from which the log-cosh loss arises. We also examine the use of the log-cosh function for quantile regression.
  arXiv  Detail & Related papers  (2022-08-09T07:03:58Z)
• How do noise tails impact on deep ReLU networks? [2.5889847253961418]
  We show how the optimal rate of convergence depends on p, the degree of smoothness and the intrinsic dimension in a class of nonparametric regression functions. We also contribute some new results on the approximation theory of deep ReLU neural networks.
  arXiv  Detail & Related papers  (2022-03-20T00:27:32Z)
• Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise [51.080620762639434]
  We propose a more efficient parameterization of the posterior approximation for sampling-free variational inference. Our approach yields competitive results for standard regression problems and scales well to large-scale image classification tasks.
  arXiv  Detail & Related papers  (2021-03-15T16:16:18Z)
• Approximation Schemes for ReLU Regression [80.33702497406632]
  We consider the fundamental problem of ReLU regression. The goal is to output the best fitting ReLU with respect to square loss given to draws from some unknown distribution.
  arXiv  Detail & Related papers  (2020-05-26T16:26:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.