Uncertainty in Extreme Multi-label Classification
- URL: http://arxiv.org/abs/2210.10160v1
- Date: Tue, 18 Oct 2022 20:54:33 GMT
- Title: Uncertainty in Extreme Multi-label Classification
- Authors: Jyun-Yu Jiang, Wei-Cheng Chang, Jiong Zhong, Cho-Jui Hsieh, and
Hsiang-Fu Yu
- Abstract summary: eXtreme Multi-label Classification (XMC) is an essential task in the era of big data for web-scale machine learning applications.
In this paper, we aim to investigate general uncertainty quantification approaches for tree-based XMC models with a probabilistic ensemble-based framework.
In particular, we analyze label-level and instance-level uncertainty in XMC, and propose a general approximation framework based on beam search to efficiently estimate the uncertainty with a theoretical guarantee under long-tail XMC predictions.
- Score: 81.14232824864787
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Uncertainty quantification is one of the most crucial tasks to obtain
trustworthy and reliable machine learning models for decision making. However,
most research in this domain has only focused on problems with small label
spaces and ignored eXtreme Multi-label Classification (XMC), which is an
essential task in the era of big data for web-scale machine learning
applications. Moreover, enormous label spaces could also lead to noisy
retrieval results and intractable computational challenges for uncertainty
quantification. In this paper, we aim to investigate general uncertainty
quantification approaches for tree-based XMC models with a probabilistic
ensemble-based framework. In particular, we analyze label-level and
instance-level uncertainty in XMC, and propose a general approximation
framework based on beam search to efficiently estimate the uncertainty with a
theoretical guarantee under long-tail XMC predictions. Empirical studies on six
large-scale real-world datasets show that our framework not only outperforms
single models in predictive performance, but also can serve as strong
uncertainty-based baselines for label misclassification and out-of-distribution
detection, with significant speedup. Besides, our framework can further yield
better state-of-the-art results based on deep XMC models with uncertainty
quantification.
Related papers
- Noise-Adaptive Conformal Classification with Marginal Coverage [53.74125453366155]
We introduce an adaptive conformal inference method capable of efficiently handling deviations from exchangeability caused by random label noise.
We validate our method through extensive numerical experiments demonstrating its effectiveness on synthetic and real data sets.
arXiv Detail & Related papers (2025-01-29T23:55:23Z) - Uncertainty separation via ensemble quantile regression [23.667247644930708]
This paper introduces a novel and scalable framework for uncertainty estimation and separation.
Our framework is scalable to large datasets and demonstrates superior performance on synthetic benchmarks.
arXiv Detail & Related papers (2024-12-18T11:15:32Z) - Ranked from Within: Ranking Large Multimodal Models for Visual Question Answering Without Labels [64.94853276821992]
Large multimodal models (LMMs) are increasingly deployed across diverse applications.
Traditional evaluation methods are largely dataset-centric, relying on fixed, labeled datasets and supervised metrics.
We explore unsupervised model ranking for LMMs by leveraging their uncertainty signals, such as softmax probabilities.
arXiv Detail & Related papers (2024-12-09T13:05:43Z) - Quantifying Prediction Consistency Under Model Multiplicity in Tabular LLMs [10.494477811252034]
Fine-tuning large language models can lead to textitfine-tuning multiplicity, where equally well-performing models make conflicting predictions on the same inputs.
This raises critical concerns about the robustness and reliability of Tabular LLMs.
This work proposes a novel metric to quantify the robustness of individual predictions without expensive model retraining.
arXiv Detail & Related papers (2024-07-04T22:22:09Z) - Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode.
We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
arXiv Detail & Related papers (2024-06-05T16:35:30Z) - On Least Square Estimation in Softmax Gating Mixture of Experts [78.3687645289918]
We investigate the performance of the least squares estimators (LSE) under a deterministic MoE model.
We establish a condition called strong identifiability to characterize the convergence behavior of various types of expert functions.
Our findings have important practical implications for expert selection.
arXiv Detail & Related papers (2024-02-05T12:31:18Z) - Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture
of Stochastic Expert [24.216869988183092]
We focus on capturing the data-inherent uncertainty (aka aleatoric uncertainty) in segmentation, typically when ambiguities exist in input images.
We propose a novel mixture of experts (MoSE) model, where each expert network estimates a distinct mode of aleatoric uncertainty.
We develop a Wasserstein-like loss that directly minimizes the distribution distance between the MoSE and ground truth annotations.
arXiv Detail & Related papers (2022-12-14T16:48:21Z) - Quantifying Model Uncertainty for Semantic Segmentation using Operators
in the RKHS [20.348825818435767]
We present a framework for high-resolution predictive uncertainty quantification of semantic segmentation models.
We use a multi-moment functional definition of uncertainty associated with the model's feature space in the kernel reproducing the Hilbert space (RKHS)
This leads to a significantly more accurate view of model uncertainty than conventional Bayesian methods.
arXiv Detail & Related papers (2022-11-03T17:10:49Z) - Uncertainty-Aware Multiple Instance Learning fromLarge-Scale Long Time
Series Data [20.2087807816461]
This paper proposes an uncertainty-aware multiple instance (MIL) framework to identify the most relevant periodautomatically.
We further incorporate another modality toaccommodate unreliable predictions by training a separate model and conduct uncertainty aware fusion.
Empirical resultsdemonstrate that the proposed method can effectively detect thetypes of vessels based on the trajectory.
arXiv Detail & Related papers (2021-11-16T17:09:02Z) - Approaching Neural Network Uncertainty Realism [53.308409014122816]
Quantifying or at least upper-bounding uncertainties is vital for safety-critical systems such as autonomous vehicles.
We evaluate uncertainty realism -- a strict quality criterion -- with a Mahalanobis distance-based statistical test.
We adopt it to the automotive domain and show that it significantly improves uncertainty realism compared to a plain encoder-decoder model.
arXiv Detail & Related papers (2021-01-08T11:56:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.