A Critical Synthesis of Uncertainty Quantification and Foundation Models for Semantic Segmentation

• URL: http://arxiv.org/abs/2608.18709v1
• Date: Wed, 19 Aug 2026 09:09:42 GMT
• Title: A Critical Synthesis of Uncertainty Quantification and Foundation Models for Semantic Segmentation
• Abstract summary: Foundation models are breaking what seemed to be impossible not long ago by enabling unprecedented accuracy and cross-domain generalization.<n>Their lack of interpretability, tendency to be overconfident, and sensitivity to real-world domain shifts pose critical challenges for safety- and mission-critical applications.<n>Uncertainty quantification (UQ) offers a principled way to address these issues, but its integration into segmentation foundation models has yet to be explored.
• Score: 8.144213556496569
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models are increasingly breaking what seemed to be impossible not long ago by enabling unprecedented accuracy and cross-domain generalization. Yet their lack of interpretability, tendency to be overconfident, and sensitivity to real-world domain shifts pose critical challenges for safety- and mission-critical applications. Uncertainty quantification (UQ) offers a principled way to address these issues, but its integration into segmentation foundation models has yet to be explored. In this paper we present the first systematic evaluation of UQ methods applied to a foundation model for semantic segmentation. We fine-tune a lightweight DPT decoder on top of the pretrained SAM2 encoder to establish a simple yet competitive baseline and benchmark four representative UQ approaches - Monte Carlo Dropout, Deep Sub-Ensemble, Test-Time Augmentation, and Evidential Deep Learning - across Cityscapes, NYUv2, and two challenging out-of-domain settings. Our analysis compares segmentation accuracy, calibration, uncertainty quality, and inference time, revealing clear trade-offs between predictive performance, reliability, and computational cost. These results highlight both the promise and the current limitations of uncertainty-aware foundation models, pointing to the need for future work that jointly optimizes accuracy, robustness, and efficiency for real-world deployment.

Related papers

• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness [4.129847064263056]
  We systematically evaluate the performance of Large Language Models for rubric-based short-answer grading.<n>We find that alignment is strong for binary tasks but degrades with increased rubric granularity.<n>Experiments reveal that while the model is resilient to prompt injection, it is sensitive to synonym substitutions.
  arXiv  Detail & Related papers  (2025-12-21T05:22:04Z)
• Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting [18.018179328110048]
  We introduce a predictability-aligned diagnostic framework grounded in spectral coherence.<n>We provide the first systematic evidence of "predictability drift", demonstrating that a task's forecasting difficulty varies sharply over time.<n>Our evaluation reveals a key architectural trade-off: complex models are superior for low-predictability data, whereas linear models are highly effective on more predictable tasks.
  arXiv  Detail & Related papers  (2025-09-27T02:56:06Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• A Critical Synthesis of Uncertainty Quantification and Foundation Models in Monocular Depth Estimation [13.062551984263031]
  Metric depth estimation, which involves predicting absolute distances, poses particular challenges.<n>We fuse five different uncertainty quantification methods with the current state-of-the-art DepthAnythingV2 foundation model.<n>Our findings identify fine-tuning with the Gaussian Negative Log-Likelihood Loss (GNLL) as a particularly promising approach.
  arXiv  Detail & Related papers  (2025-01-14T15:13:00Z)
• SURE: SUrvey REcipes for building reliable and robust deep networks [12.268921703825258]
  In this paper, we revisit techniques for uncertainty estimation within deep neural networks and consolidate a suite of techniques to enhance their reliability. We rigorously evaluate SURE against the benchmark of failure prediction, a critical testbed for uncertainty estimation efficacy. When applied to real-world challenges, such as data corruption, label noise, and long-tailed class distribution, SURE exhibits remarkable robustness, delivering results that are superior or on par with current state-of-the-art specialized methods.
  arXiv  Detail & Related papers  (2024-03-01T13:58:19Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Uncertainty-Aware Deep Calibrated Salient Object Detection [74.58153220370527]
  Existing deep neural network based salient object detection (SOD) methods mainly focus on pursuing high network accuracy. These methods overlook the gap between network accuracy and prediction confidence, known as the confidence uncalibration problem. We introduce an uncertaintyaware deep SOD network, and propose two strategies to prevent deep SOD networks from being overconfident.
  arXiv  Detail & Related papers  (2020-12-10T23:28:36Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.