Controlled abstention neural networks for identifying skillful predictions for regression problems

• URL: http://arxiv.org/abs/2104.08236v1
• Date: Fri, 16 Apr 2021 17:16:32 GMT
• Title: Controlled abstention neural networks for identifying skillful predictions for regression problems
• Authors: Elizabeth A. Barnes and Randal J. Barnes
• Abstract summary: We introduce a novel loss function, termed "abstention loss", that allows neural networks to identify forecasts of opportunity for regression problems. The abstention loss is applied during training to preferentially learn from the more confident samples.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we look for specific states of the system that lead to more predictable behavior than others, often termed "forecasts of opportunity". When these opportunities are not present, scientists need prediction systems that are capable of saying "I don't know." We introduce a novel loss function, termed "abstention loss", that allows neural networks to identify forecasts of opportunity for regression problems. The abstention loss works by incorporating uncertainty in the network's prediction to identify the more confident samples and abstain (say "I don't know") on the less confident samples. The abstention loss is designed to determine the optimal abstention fraction, or abstain on a user-defined fraction via a PID controller. Unlike many methods for attaching uncertainty to neural network predictions post-training, the abstention loss is applied during training to preferentially learn from the more confident samples. The abstention loss is built upon a standard computer science method. While the standard approach is itself a simple yet powerful tool for incorporating uncertainty in regression problems, we demonstrate that the abstention loss outperforms this more standard method for the synthetic climate use cases explored here. The implementation of proposed loss function is straightforward in most network architectures designed for regression, as it only requires modification of the output layer and loss function.

Related papers

• Enforcing Equity in Neural Climate Emulators [0.0]
  We propose a custom loss function which punishes neural network emulators with unequal quality of predictions. The loss function does not specify a particular definition of equity to bias the neural network towards. Our results show that neural climate emulators trained with our loss function provide more equitable predictions.
  arXiv  Detail & Related papers  (2024-06-28T03:47:54Z)
• ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference [54.17205151960878]
  We introduce a sampling-free approach that is generic and easy to deploy. We produce reliable uncertainty estimates on par with state-of-the-art methods at a significantly lower computational cost.
  arXiv  Detail & Related papers  (2022-11-21T13:23:09Z)
• Interpretable Self-Aware Neural Networks for Robust Trajectory Prediction [50.79827516897913]
  We introduce an interpretable paradigm for trajectory prediction that distributes the uncertainty among semantic concepts. We validate our approach on real-world autonomous driving data, demonstrating superior performance over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-11-16T06:28:20Z)
• SIoU Loss: More Powerful Learning for Bounding Box Regression [0.0]
  Loss function SIoU was suggested, where penalty metrics were redefined considering the angle of the vector between the desired regression. Applied to conventional Neural Networks and datasets it is shown that SIoU improves both the speed of training and the accuracy of the inference.
  arXiv  Detail & Related papers  (2022-05-25T12:46:21Z)
• Training Uncertainty-Aware Classifiers with Conformalized Deep Learning [7.837881800517111]
  Deep neural networks are powerful tools to detect hidden patterns in data and leverage them to make predictions, but they are not designed to understand uncertainty. We develop a novel training algorithm that can lead to more dependable uncertainty estimates, without sacrificing predictive power.
  arXiv  Detail & Related papers  (2022-05-12T05:08:10Z)
• Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization [52.7137956951533]
  We argue that devising simpler methods for learning predictors on existing features is a promising direction for future research. We introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift. Under a natural model, we prove that the DARE solution is the minimax-optimal predictor for a constrained set of test distributions.
  arXiv  Detail & Related papers  (2022-02-14T16:42:16Z)
• Robust Neural Regression via Uncertainty Learning [5.654198773446211]
  Deep neural networks tend to underestimate uncertainty and produce overly confident predictions. We propose a simple solution by extending the time-tested iterative reweighted least square (IRLS) in generalised linear regression. We use two sub-networks to parametrise the prediction and uncertainty estimation, enabling easy handling of complex inputs and nonlinear response.
  arXiv  Detail & Related papers  (2021-10-12T23:19:13Z)
• Learning to Predict Trustworthiness with Steep Slope Loss [69.40817968905495]
  We study the problem of predicting trustworthiness on real-world large-scale datasets. We observe that the trustworthiness predictors trained with prior-art loss functions are prone to view both correct predictions and incorrect predictions to be trustworthy. We propose a novel steep slope loss to separate the features w.r.t. correct predictions from the ones w.r.t. incorrect predictions by two slide-like curves that oppose each other.
  arXiv  Detail & Related papers  (2021-09-30T19:19:09Z)
• Controlled abstention neural networks for identifying skillful predictions for classification problems [0.0]
  We introduce a novel loss function, termed the "NotWrong loss", that allows neural networks to identify forecasts of opportunity for classification problems. The NotWrong loss is applied during training to preferentially learn from the more confident samples. We show that the NotWrong loss outperforms other existing loss functions for multiple climate use cases.
  arXiv  Detail & Related papers  (2021-04-16T17:18:32Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)
• An Uncertainty-based Human-in-the-loop System for Industrial Tool Wear Analysis [68.8204255655161]
  We show that uncertainty measures based on Monte-Carlo dropout in the context of a human-in-the-loop system increase the system's transparency and performance. A simulation study demonstrates that the uncertainty-based human-in-the-loop system increases performance for different levels of human involvement.
  arXiv  Detail & Related papers  (2020-07-14T15:47:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.