Uncertainty Estimation in Autoregressive Structured Prediction

• URL: http://arxiv.org/abs/2002.07650v5
• Date: Thu, 11 Feb 2021 09:42:35 GMT
• Title: Uncertainty Estimation in Autoregressive Structured Prediction
• Authors: Andrey Malinin, Mark Gales
• Abstract summary: This work aims to investigate uncertainty estimation for autoregressive structured prediction tasks. We consider: uncertainty estimation for sequence data at the token-level and complete sequence-level; interpretations for, and applications of, various measures of uncertainty. This work also provides baselines for token-level and sequence-level error detection, and sequence-level out-of-domain input detection on the WMT'14 English-French and WMT'17 English-German translation datasets.
• Score: 16.441252243846534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Uncertainty estimation is important for ensuring safety and robustness of AI systems. While most research in the area has focused on un-structured prediction tasks, limited work has investigated general uncertainty estimation approaches for structured prediction. Thus, this work aims to investigate uncertainty estimation for autoregressive structured prediction tasks within a single unified and interpretable probabilistic ensemble-based framework. We consider: uncertainty estimation for sequence data at the token-level and complete sequence-level; interpretations for, and applications of, various measures of uncertainty; and discuss both the theoretical and practical challenges associated with obtaining them. This work also provides baselines for token-level and sequence-level error detection, and sequence-level out-of-domain input detection on the WMT'14 English-French and WMT'17 English-German translation and LibriSpeech speech recognition datasets.

Related papers

• SConU: Selective Conformal Uncertainty in Large Language Models [59.25881667640868]
  We propose a novel approach termed Selective Conformal Uncertainty (SConU) We develop two conformal p-values that are instrumental in determining whether a given sample deviates from the uncertainty distribution of the calibration set at a specific manageable risk level. Our approach not only facilitates rigorous management of miscoverage rates across both single-domain and interdisciplinary contexts, but also enhances the efficiency of predictions.
  arXiv  Detail & Related papers  (2025-04-19T03:01:45Z)
• Post-hoc Probabilistic Vision-Language Models [51.12284891724463]
  Vision-language models (VLMs) have found remarkable success in classification, retrieval, and generative tasks. We propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our results show promise for safety-critical applications of large-scale models.
  arXiv  Detail & Related papers  (2024-12-08T18:16:13Z)
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [69.83976050879318]
  In large language models (LLMs), identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions.
  arXiv  Detail & Related papers  (2023-11-15T05:58:35Z)
• Discretization-Induced Dirichlet Posterior for Robust Uncertainty Quantification on Regression [17.49026509916207]
  Uncertainty quantification is critical for deploying deep neural networks (DNNs) in real-world applications. For vision regression tasks, current AuxUE designs are mainly adopted for aleatoric uncertainty estimates. We propose a generalized AuxUE scheme for more robust uncertainty quantification on regression tasks.
  arXiv  Detail & Related papers  (2023-08-17T15:54:11Z)
• Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
  We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
  arXiv  Detail & Related papers  (2023-08-03T12:43:21Z)
• Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation [5.55197751179213]
  Existing supervised methods treat WSD as a classification task and have achieved remarkable performance. This paper extensively studies uncertainty estimation (UE) on the benchmark designed for WSD. We examine the capability of capturing data and model uncertainties by the model with the selected UE score on well-designed test scenarios and discover that the model reflects data uncertainty satisfactorily but underestimates model uncertainty.
  arXiv  Detail & Related papers  (2023-05-22T15:18:15Z)
• Integrating Uncertainty into Neural Network-based Speech Enhancement [27.868722093985006]
  Supervised masking approaches in the time-frequency domain aim to employ deep neural networks to estimate a multiplicative mask to extract clean speech. This leads to a single estimate for each input without any guarantees or measures of reliability. We study the benefits of modeling uncertainty in clean speech estimation.
  arXiv  Detail & Related papers  (2023-05-15T15:55:12Z)
• On Attacking Out-Domain Uncertainty Estimation in Deep Neural Networks [11.929914721626849]
  We show that state-of-the-art uncertainty estimation algorithms could fail catastrophically under our proposed adversarial attack. In particular, we aim at attacking the out-domain uncertainty estimation.
  arXiv  Detail & Related papers  (2022-10-03T23:33:38Z)
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [77.39239190539871]
  We show how uncertainty estimation can be leveraged to enable safety critical image segmentation in autonomous driving. We introduce a new uncertainty measure based on disagreeing predictions as measured by a dissimilarity function. We show experimentally that our proposed approach is much less computationally intensive at inference time than competing methods.
  arXiv  Detail & Related papers  (2021-05-28T09:23:05Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)
• Probabilistic Deep Learning for Instance Segmentation [9.62543698736491]
  We propose a generic method to obtain model-inherent uncertainty estimates within proposal-free instance segmentation models. We evaluate our method on the BBBC010 C. elegans dataset, where it yields competitive performance.
  arXiv  Detail & Related papers  (2020-08-24T19:51:48Z)
• On the uncertainty of self-supervised monocular depth estimation [52.13311094743952]
  Self-supervised paradigms for monocular depth estimation are very appealing since they do not require ground truth annotations at all. We explore for the first time how to estimate the uncertainty for this task and how this affects depth accuracy. We propose a novel peculiar technique specifically designed for self-supervised approaches.
  arXiv  Detail & Related papers  (2020-05-13T09:00:55Z)
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning [70.72363097550483]
  In this study, we focus on in-domain uncertainty for image classification. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE)
  arXiv  Detail & Related papers  (2020-02-15T23:28:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.