Uncertainty in Extreme Multi-label Classification
- URL: http://arxiv.org/abs/2210.10160v1
- Date: Tue, 18 Oct 2022 20:54:33 GMT
- Title: Uncertainty in Extreme Multi-label Classification
- Authors: Jyun-Yu Jiang, Wei-Cheng Chang, Jiong Zhong, Cho-Jui Hsieh, and
Hsiang-Fu Yu
- Abstract summary: eXtreme Multi-label Classification (XMC) is an essential task in the era of big data for web-scale machine learning applications.
In this paper, we aim to investigate general uncertainty quantification approaches for tree-based XMC models with a probabilistic ensemble-based framework.
In particular, we analyze label-level and instance-level uncertainty in XMC, and propose a general approximation framework based on beam search to efficiently estimate the uncertainty with a theoretical guarantee under long-tail XMC predictions.
- Score: 81.14232824864787
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Uncertainty quantification is one of the most crucial tasks to obtain
trustworthy and reliable machine learning models for decision making. However,
most research in this domain has only focused on problems with small label
spaces and ignored eXtreme Multi-label Classification (XMC), which is an
essential task in the era of big data for web-scale machine learning
applications. Moreover, enormous label spaces could also lead to noisy
retrieval results and intractable computational challenges for uncertainty
quantification. In this paper, we aim to investigate general uncertainty
quantification approaches for tree-based XMC models with a probabilistic
ensemble-based framework. In particular, we analyze label-level and
instance-level uncertainty in XMC, and propose a general approximation
framework based on beam search to efficiently estimate the uncertainty with a
theoretical guarantee under long-tail XMC predictions. Empirical studies on six
large-scale real-world datasets show that our framework not only outperforms
single models in predictive performance, but also can serve as strong
uncertainty-based baselines for label misclassification and out-of-distribution
detection, with significant speedup. Besides, our framework can further yield
better state-of-the-art results based on deep XMC models with uncertainty
quantification.
Related papers
- Quantifying Prediction Consistency Under Model Multiplicity in Tabular LLMs [10.494477811252034]
Fine-tuning large language models can lead to textitfine-tuning multiplicity, where equally well-performing models make conflicting predictions on the same inputs.
This raises critical concerns about the robustness and reliability of Tabular LLMs.
This work proposes a novel metric to quantify the robustness of individual predictions without expensive model retraining.
arXiv Detail & Related papers (2024-07-04T22:22:09Z) - Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode.
We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
arXiv Detail & Related papers (2024-06-05T16:35:30Z) - Uncertainty-aware Evidential Fusion-based Learning for Semi-supervised Medical Image Segmentation [0.0]
This paper integrates the evidential predictive results in the cross-region of mixed and original samples to reallocate the confidence degree and uncertainty measure of each voxel.
We design a voxel-level learning strategy by introducing information entropy to combine with the fused uncertainty measure to estimate voxel prediction more precisely.
The experimental results on LA, Pancreas-CT, ACDC and TBAD datasets demonstrate the superior performance of our proposed method in comparison with the existing state of the arts.
arXiv Detail & Related papers (2024-04-09T09:58:10Z) - On Least Square Estimation in Softmax Gating Mixture of Experts [78.3687645289918]
We investigate the performance of the least squares estimators (LSE) under a deterministic MoE model.
We establish a condition called strong identifiability to characterize the convergence behavior of various types of expert functions.
Our findings have important practical implications for expert selection.
arXiv Detail & Related papers (2024-02-05T12:31:18Z) - Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation [50.920911532133154]
The intrinsic ill-posedness and ordinal-sensitive nature of monocular depth estimation (MDE) models pose major challenges to the estimation of uncertainty degree.
We propose to model the uncertainty of MDE models from the perspective of the inherent probability distributions.
By simply introducing additional training regularization terms, our model, with surprisingly simple formations and without requiring extra modules or multiple inferences, can provide uncertainty estimations with state-of-the-art reliability.
arXiv Detail & Related papers (2023-07-19T12:11:15Z) - Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture
of Stochastic Expert [24.216869988183092]
We focus on capturing the data-inherent uncertainty (aka aleatoric uncertainty) in segmentation, typically when ambiguities exist in input images.
We propose a novel mixture of experts (MoSE) model, where each expert network estimates a distinct mode of aleatoric uncertainty.
We develop a Wasserstein-like loss that directly minimizes the distribution distance between the MoSE and ground truth annotations.
arXiv Detail & Related papers (2022-12-14T16:48:21Z) - Quantifying Model Uncertainty for Semantic Segmentation using Operators
in the RKHS [20.348825818435767]
We present a framework for high-resolution predictive uncertainty quantification of semantic segmentation models.
We use a multi-moment functional definition of uncertainty associated with the model's feature space in the kernel reproducing the Hilbert space (RKHS)
This leads to a significantly more accurate view of model uncertainty than conventional Bayesian methods.
arXiv Detail & Related papers (2022-11-03T17:10:49Z) - Non-Linear Spectral Dimensionality Reduction Under Uncertainty [107.01839211235583]
We propose a new dimensionality reduction framework, called NGEU, which leverages uncertainty information and directly extends several traditional approaches.
We show that the proposed NGEU formulation exhibits a global closed-form solution, and we analyze, based on the Rademacher complexity, how the underlying uncertainties theoretically affect the generalization ability of the framework.
arXiv Detail & Related papers (2022-02-09T19:01:33Z) - Uncertainty-Aware Multiple Instance Learning fromLarge-Scale Long Time
Series Data [20.2087807816461]
This paper proposes an uncertainty-aware multiple instance (MIL) framework to identify the most relevant periodautomatically.
We further incorporate another modality toaccommodate unreliable predictions by training a separate model and conduct uncertainty aware fusion.
Empirical resultsdemonstrate that the proposed method can effectively detect thetypes of vessels based on the trajectory.
arXiv Detail & Related papers (2021-11-16T17:09:02Z) - Approaching Neural Network Uncertainty Realism [53.308409014122816]
Quantifying or at least upper-bounding uncertainties is vital for safety-critical systems such as autonomous vehicles.
We evaluate uncertainty realism -- a strict quality criterion -- with a Mahalanobis distance-based statistical test.
We adopt it to the automotive domain and show that it significantly improves uncertainty realism compared to a plain encoder-decoder model.
arXiv Detail & Related papers (2021-01-08T11:56:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.