Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Tasks

• URL: http://arxiv.org/abs/2407.02138v2
• Date: Thu, 06 Feb 2025 17:32:04 GMT
• Title: Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Tasks
• Authors: Wataru Hashimoto, Hidetaka Kamigaito, Taro Watanabe,
• Abstract summary: Trustworthiness in model predictions is crucial for safety-critical applications in the real world. Deep neural networks often suffer from the issues of uncertainty estimation, such as miscalibration. We propose $k$-Nearest Neighbor Uncertainty Estimation ($k$NN-UE), which uses not only the distances from the neighbors, but also the ratio of labels in the neighbors.
• Score: 26.336947440529713
• License:
• Abstract: Trustworthiness in model predictions is crucial for safety-critical applications in the real world. However, deep neural networks often suffer from the issues of uncertainty estimation, such as miscalibration. In this study, we propose $k$-Nearest Neighbor Uncertainty Estimation ($k$NN-UE), which is a new uncertainty estimation method that uses not only the distances from the neighbors, but also the ratio of labels in the neighbors. Experiments on sentiment analysis, natural language inference, and named entity recognition show that our proposed method outperforms the baselines and recent density-based methods in several calibration and uncertainty metrics. Moreover, our analyses indicate that approximate nearest neighbor search techniques reduce the inference overhead without significantly degrading the uncertainty estimation performance when they are appropriately combined.

Related papers

• Confidence Intervals and Simultaneous Confidence Bands Based on Deep Learning [0.36832029288386137]
  We provide a valid non-parametric bootstrap method that correctly disentangles data uncertainty from the noise inherent in the adopted optimization algorithm. The proposed ad-hoc method can be easily integrated into any deep neural network without interfering with the training process.
  arXiv  Detail & Related papers  (2024-06-20T05:51:37Z)
• One step closer to unbiased aleatoric uncertainty estimation [71.55174353766289]
  We propose a new estimation method by actively de-noising the observed data. By conducting a broad range of experiments, we demonstrate that our proposed approach provides a much closer approximation to the actual data uncertainty than the standard method.
  arXiv  Detail & Related papers  (2023-12-16T14:59:11Z)
• Uncertainty-Aware Lidar Place Recognition in Novel Environments [11.30020653282995]
  We investigate the task of uncertainty-aware lidar place recognition. Each predicted place must have an associated uncertainty that can be used to identify and reject incorrect predictions. We introduce a novel evaluation protocol and present the first comprehensive benchmark for this task.
  arXiv  Detail & Related papers  (2022-10-04T04:06:44Z)
• On Attacking Out-Domain Uncertainty Estimation in Deep Neural Networks [11.929914721626849]
  We show that state-of-the-art uncertainty estimation algorithms could fail catastrophically under our proposed adversarial attack. In particular, we aim at attacking the out-domain uncertainty estimation.
  arXiv  Detail & Related papers  (2022-10-03T23:33:38Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [68.34559610536614]
  We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
  arXiv  Detail & Related papers  (2021-11-22T08:54:10Z)
• Dense Uncertainty Estimation [62.23555922631451]
  In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation. We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
  arXiv  Detail & Related papers  (2021-10-13T01:23:48Z)
• CertainNet: Sampling-free Uncertainty Estimation for Object Detection [65.28989536741658]
  Estimating the uncertainty of a neural network plays a fundamental role in safety-critical settings. In this work, we propose a novel sampling-free uncertainty estimation method for object detection. We call it CertainNet, and it is the first to provide separate uncertainties for each output signal: objectness, class, location and size.
  arXiv  Detail & Related papers  (2021-10-04T17:59:31Z)
• Adversarial Attack for Uncertainty Estimation: Identifying Critical Regions in Neural Networks [0.0]
  We propose a novel method to capture data points near decision boundary in neural network that are often referred to a specific type of uncertainty. Uncertainty estimates are derived from the input perturbations, unlike previous studies that provide perturbations on the model's parameters. We show that the proposed method has revealed a significant outperformance over other methods and provided less risk to capture model uncertainty in machine learning.
  arXiv  Detail & Related papers  (2021-07-15T21:30:26Z)
• The Aleatoric Uncertainty Estimation Using a Separate Formulation with Virtual Residuals [51.71066839337174]
  Existing methods can quantify the error in the target estimation, but they tend to underestimate it. We propose a new separable formulation for the estimation of a signal and of its uncertainty, avoiding the effect of overfitting. We demonstrate that the proposed method outperforms a state-of-the-art technique for signal and uncertainty estimation.
  arXiv  Detail & Related papers  (2020-11-03T12:11:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.