On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression

• URL: http://arxiv.org/abs/2008.07281v1
• Date: Wed, 12 Aug 2020 22:41:26 GMT
• Title: On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression
• Authors: Jun Qi, Jun Du, Sabato Marco Siniscalchi, Xiaoli Ma, Chin-Hui Lee
• Abstract summary: We exploit the properties of mean absolute error (MAE) as a loss function for the deep neural network (DNN) based vector-to-vector regression. We show that MAE can be interpreted as an error modeled by Laplacian distribution.
• Score: 79.86233860519621
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we exploit the properties of mean absolute error (MAE) as a loss function for the deep neural network (DNN) based vector-to-vector regression. The goal of this work is two-fold: (i) presenting performance bounds of MAE, and (ii) demonstrating new properties of MAE that make it more appropriate than mean squared error (MSE) as a loss function for DNN based vector-to-vector regression. First, we show that a generalized upper-bound for DNN-based vector- to-vector regression can be ensured by leveraging the known Lipschitz continuity property of MAE. Next, we derive a new generalized upper bound in the presence of additive noise. Finally, in contrast to conventional MSE commonly adopted to approximate Gaussian errors for regression, we show that MAE can be interpreted as an error modeled by Laplacian distribution. Speech enhancement experiments are conducted to corroborate our proposed theorems and validate the performance advantages of MAE over MSE for DNN based regression.

Related papers

• The Prevalence of Neural Collapse in Neural Multivariate Regression [3.691119072844077]
  We show that neural networks exhibit Neural Collapse (NC) during the final stage of training for the classification problem. To our knowledge, this is the first empirical and theoretical study of neural collapse in the context of regression.
  arXiv  Detail & Related papers  (2024-09-06T10:45:58Z)
• The Generalization Error of Stochastic Mirror Descent on Over-Parametrized Linear Models [37.6314945221565]
  Deep networks are known to generalize well to unseen data. Regularization properties ensure interpolating solutions with "good" properties are found. We present simulation results that validate the theory and introduce two data models.
  arXiv  Detail & Related papers  (2023-02-18T22:23:42Z)
• Variational Laplace Autoencoders [53.08170674326728]
  Variational autoencoders employ an amortized inference model to approximate the posterior of latent variables. We present a novel approach that addresses the limited posterior expressiveness of fully-factorized Gaussian assumption. We also present a general framework named Variational Laplace Autoencoders (VLAEs) for training deep generative models.
  arXiv  Detail & Related papers  (2022-11-30T18:59:27Z)
• The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Models [7.00422423634143]
  We study the Gradient Descent (SGD) algorithm in nonparametric statistics: kernel regression. The directional bias property of SGD, which is known in the linear regression setting, is generalized to the kernel regression.
  arXiv  Detail & Related papers  (2022-04-29T19:44:01Z)
• Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization [52.7137956951533]
  We argue that devising simpler methods for learning predictors on existing features is a promising direction for future research. We introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift. Under a natural model, we prove that the DARE solution is the minimax-optimal predictor for a constrained set of test distributions.
  arXiv  Detail & Related papers  (2022-02-14T16:42:16Z)
• Improving predictions of Bayesian neural nets via local linearization [79.21517734364093]
  We argue that the Gauss-Newton approximation should be understood as a local linearization of the underlying Bayesian neural network (BNN) Because we use this linearized model for posterior inference, we should also predict using this modified model instead of the original one. We refer to this modified predictive as "GLM predictive" and show that it effectively resolves common underfitting problems of the Laplace approximation.
  arXiv  Detail & Related papers  (2020-08-19T12:35:55Z)
• Analyzing Upper Bounds on Mean Absolute Errors for Deep Neural Network Based Vector-to-Vector Regression [79.86233860519621]
  We show that, in vector-to-vector regression utilizing deep neural networks (DNNs), a generalized loss of error (MAE) is between a mean absolute error and an expected feature error. We propose a proposed upper bounds of MAE for DNN based vector-to-vector regression.
  arXiv  Detail & Related papers  (2020-08-04T19:39:41Z)
• Robust Compressed Sensing using Generative Models [98.64228459705859]
  In this paper we propose an algorithm inspired by the Median-of-Means (MOM) Our algorithm guarantees recovery for heavy-tailed data, even in the presence of outliers.
  arXiv  Detail & Related papers  (2020-06-16T19:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.