Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

• URL: http://arxiv.org/abs/2606.25760v1
• Date: Wed, 24 Jun 2026 12:34:28 GMT
• Title: Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
• Abstract summary: We present Argus, a cross-regime benchmark for post-hoc UQ in single-step executable GUI grounding.<n>We evaluate methods for logit-based scores, sampling and consistency measures, hidden-state and density estimators, P(True) and verbalised-confidence prompting.<n>The main finding is selective transfer: UQ rankings are stable across datasets for a fixed model, but degrade across model classes and observable interfaces.
• Score: 6.896160570537455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer-use agents turn vision-language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions. Yet evidence on post-hoc uncertainty quantification (UQ) for these agents is fragmented across isolated model and dataset pairs, leaving it unclear whether UQ rankings stay stable when the agent, benchmark, or observable interface changes. We present Argus, a cross-regime benchmark for post-hoc UQ in single-step executable GUI grounding: a 27-method open-weight matrix over 4 VLM agents and 4 datasets, plus an 8-method closed-source matrix across 3 frontier vendors where logits, hidden states, and attention maps are unavailable. Evaluated methods span logit-based scores, sampling and consistency measures, hidden-state and density estimators (Mahalanobis, SAPLMA), attention-based scores, P(True) and verbalised-confidence prompting, and split-conformal prediction. The main finding is selective transfer: UQ rankings are stable across datasets for a fixed model, but degrade across model classes and observable interfaces. Hidden-state and density methods are the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes. Within-model ranking transfer is strong (Spearman rho up to 0.969), but cross-tier transfer to closed-source vendors averages only +0.08, so closed-source UQ should be reranked on the target rather than extrapolated. Conformal click regions show score-level discrimination is not enough for deployment: locally weighted disks shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch. We release per-item records, calibration/test splits, UQ scores, and analysis scripts for regime-aware UQ selection in GUI agents.

Related papers

• Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents [0.0]
  We study whether three common families of single-turn UQ methods transfer to an interactive trajectory setting.<n>We find that transfer is often useful but uneven. Token-probability scores are highly sensitive to the choice of aggregator used across turns.<n>Black-box self-consistency is often the strongest UQ family, with trajectory-equivalence and action-set consistency typically ranking highest among its variants.
  arXiv  Detail & Related papers  (2026-08-12T01:39:28Z)
• Understanding Software Defect Prediction: A Large-scale Empirical Study Across Uncertainty Quantification and Performance Evaluation [12.540031438888716]
  Software defect prediction (SDP) classifiers produce probabilities used for inspection prioritization, threshold tuning, and risk communication.<n>We conducted a large-scale empirical study of probability-based uncertainty quantification (UQ) for SDP.<n>We evaluated five UQ metrics, six performance metrics, and three calibration metrics for 16 representative classifiers.
  arXiv  Detail & Related papers  (2026-07-02T08:05:27Z)
• MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events [0.7685926099671206]
  Machine learning in high-stakes domains such as healthcare requires strong predictive performance and reliable uncertainty quantification.<n>We introduce MADE, a living MLTC benchmark derived from medical device adverse event reports.<n>We systematically assess entropy-/consistency-based and self-verbalized UQ methods.
  arXiv  Detail & Related papers  (2026-04-16T16:28:16Z)
• VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering [53.662676566188175]
  A key bottleneck lies in the lack of public, large-scale, high-quality Scientific Visual Question Answering (SVQA) datasets.<n>We propose a verification-centric Generate-then-Verify framework that first generates QA pairs with figure-associated textual context.<n>We instantiate this framework to curate VeriSciQA, a dataset of 20,351 QA pairs spanning 20 scientific domains and 12 figure types.
  arXiv  Detail & Related papers  (2025-11-25T04:14:52Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework [6.160524342384932]
  We propose a trustworthy uncertainty quantification (UQ) method based on the predictability, computability, and stability framework for veridical data science.<n> PCS-UQ addresses model selection by using a prediction check to screen out unsuitable models.<n> PCS-UQ then fits these screened algorithms across multiple bootstraps to assess inter-sample variability and algorithmic instability.
  arXiv  Detail & Related papers  (2025-05-13T17:58:16Z)
• Sample, Don't Search: Rethinking Test-Time Alignment for Language Models [55.2480439325792]
  We introduce QAlign, a new test-time alignment approach.<n>As we scale test-time compute, QAlign converges to sampling from the optimal aligned distribution for each individual prompt.<n>By adopting recent advances in Markov chain Monte Carlo for text generation, our method enables better-aligned outputs without modifying the underlying model or even requiring logit access.
  arXiv  Detail & Related papers  (2025-04-04T00:41:40Z)
• Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution [27.411948623919226]
  We leverage Multi-modal Large Language Models (MLLMs) to regress accurate quality scores.<n>A key challenge is that the quality score is inherently continuous, typically modeled as a Gaussian distribution, whereas MLLMs generate discrete token outputs.<n>We propose a distribution-based approach that discretizes the score distribution into a soft label.<n>This method preserves the characteristics of the score distribution, achieving high accuracy and maintaining inter-image relationships.
  arXiv  Detail & Related papers  (2025-01-20T16:04:57Z)
• RICA2: Rubric-Informed, Calibrated Assessment of Actions [8.641411594566714]
  We present RICA2 - a deep probabilistic model that score rubric and accounts for prediction uncertainty for action quality assessment (AQA) We demonstrate that our method establishes new state of the art on public benchmarks, including FineDiving, MTL-AQA, and JIGSAWS, with superior performance in score prediction and uncertainty calibration.
  arXiv  Detail & Related papers  (2024-08-04T20:35:33Z)
• Towards Clear Expectations for Uncertainty Estimation [64.20262246029286]
  Uncertainty Quantification (UQ) is crucial to achieve trustworthy Machine Learning (ML) Most UQ methods suffer from disparate and inconsistent evaluation protocols. This opinion paper offers a new perspective by specifying those requirements through five downstream tasks.
  arXiv  Detail & Related papers  (2022-07-27T07:50:57Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.