Perturbation-Based Uncertainty for Failure Detection in Vision-Language-Action Models

• URL: http://arxiv.org/abs/2606.20754v1
• Date: Thu, 18 Jun 2026 05:41:45 GMT
• Title: Perturbation-Based Uncertainty for Failure Detection in Vision-Language-Action Models
• Abstract summary: We propose a model-agnostic framework for inference-time uncertainty estimation through hidden activation perturbations.<n>We show that perturbation-based uncertainty consistently improves failure detection under distribution shift compared to sampling-based uncertainty.
• Score: 5.736588561666141
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models have shown strong performance in robotic manipulation, but reliable uncertainty quantification remains challenging, particularly under distribution shift. Unlike autoregressive policies, many modern VLA models generate continuous actions through regression or flow-based generation, where explicit predictive probabilities are unavailable. Moreover, existing approaches often rely on stochastic action sampling or supervised failure labels, limiting their applicability across diverse pretrained VLA models. In this work, we propose a label-free and model-agnostic framework for inference-time uncertainty estimation through hidden activation perturbations, motivated by Bayesian perspectives on local model variations. Specifically, we inject Gaussian perturbations into transformer hidden activations and estimate epistemic signals from disagreement across perturbed action predictions. Experiments on LIBERO and LIBERO-PRO show that perturbation-based uncertainty consistently improves failure detection under distribution shift compared to sampling-based uncertainty, providing a practical uncertainty signal for VLA models.

Related papers

• Dynamic Distribution-Aware Uncertainty Tracking in Vision-Language Representation Learning [13.524679557181997]
  Uncertainty Quantification aims to measure the reliability of model predictions.<n>Post-hoc approaches are widely adopted due to their lightweight nature.<n>We propose a Dynamic Distribution-Aware Uncertainty Quantification framework (DDA-UQ) that shifts the paradigm from static mapping to a dynamic distribution-aware process.
  arXiv  Detail & Related papers  (2026-08-10T01:53:28Z)
• From Uncertainty to Failure Attribution: Self-Diagnosing Models for Failure Attribution under Distribution Shift [0.3948325938742681]
  We introduce a problem setting for failure attribution under distribution shift.<n>This enables the models not only to detect OOD samples, but also to find out the reason for their failure.<n>The solution we propose is called self-diagnosing models, which are capable of jointly learning predictive output, predictive uncertainty, and a failure attribution signal.
  arXiv  Detail & Related papers  (2026-08-08T06:32:20Z)
• Uncertainty Quantification for Flow-Based Vision-Language-Action Models [33.28454469934064]
  Vision-language-action models (VLAs) combine vision-language backbones with expressive generative action heads trained via flow matching on large-scale robotic datasets.<n>Despite their strong empirical performance in robotic manipulation, VLAs lack mechanisms to quantify confidence in their predictions and to detect when their actions may be unreliable.<n>We propose SAVE, a framework for uncertainty-guided active fine-tuning that reduces the number of costly expert demonstrations required to adapt VLAs to new tasks.
  arXiv  Detail & Related papers  (2026-06-16T15:19:09Z)
• FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty [20.637224080557534]
  In many applications, such as robotics, it is crucial to quantify the uncertainty in the output of vision-language models.<n>We develop FUSE, a probabilistic framework for capturing two complementary sources of uncertainty in vision-language modeling.
  arXiv  Detail & Related papers  (2026-06-01T22:11:00Z)
• ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control [7.018221049093963]
  Vision-language-action (VLA) models have emerged as generalist robotic controllers capable of mapping visual observations and natural language instructions to continuous action sequences.<n>We introduce ReconVLA, a reliable conformal model that produces uncertainty-guided and failure-aware control signals.<n>Our results show that conformalized action predictions consistently improve failure anticipation, reduce catastrophic errors, and provide a calibrated measure of confidence without retraining or modifying the underlying VLA.
  arXiv  Detail & Related papers  (2026-04-17T20:20:43Z)
• Shifting Uncertainty to Critical Moments: Towards Reliable Uncertainty Quantification for VLA Model [10.781664119473966]
  We propose a unified uncertainty quantification approach for predicting rollout success versus failure.<n>Our method substantially improves failure prediction accuracy and yields more reliable signals for failure detection.
  arXiv  Detail & Related papers  (2026-03-18T22:52:03Z)
• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach [78.4812458793128]
  We propose textbfTACO, a test-time-scaling framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks.<n>Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits.
  arXiv  Detail & Related papers  (2025-12-02T14:42:54Z)
• Post-hoc Probabilistic Vision-Language Models [54.05237186168399]
  Vision-language models (VLMs) have found remarkable success in classification, retrieval, and generative tasks.<n>We propose post-hoc uncertainty estimation in VLMs that does not require additional training.<n>Our results show promise for safety-critical applications of large-scale models.
  arXiv  Detail & Related papers  (2024-12-08T18:16:13Z)
• Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models [104.55763564037831]
  We train a regression model that leverages attention maps, probabilities on the current generation step, and recurrently computed uncertainty scores from previously generated tokens.<n>Our evaluation shows that the proposed method is highly effective for selective generation, achieving substantial improvements over rivaling unsupervised and supervised approaches.
  arXiv  Detail & Related papers  (2024-08-20T09:42:26Z)
• Predictive Uncertainty Quantification for Bird's Eye View Segmentation: A Benchmark and Novel Loss Function [10.193504550494486]
  This paper introduces a benchmark for predictive uncertainty quantification in Bird's Eye View (BEV) segmentation.<n>Our study focuses on the effectiveness of quantified uncertainty in detecting misclassified and out-of-distribution pixels.<n>We propose a novel loss function, Uncertainty-Focal-Cross-Entropy (UFCE), specifically designed for highly imbalanced data.
  arXiv  Detail & Related papers  (2024-05-31T16:32:46Z)
• Toward Reliable Human Pose Forecasting with Uncertainty [51.628234388046195]
  We develop an open-source library for human pose forecasting, including multiple models, supporting several datasets. We devise two types of uncertainty in the problem to increase performance and convey better trust.
  arXiv  Detail & Related papers  (2023-04-13T17:56:08Z)
• Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model [68.34559610536614]
  We argue that the aleatoric uncertainty is an inherent attribute of the data and can only be correctly estimated with an unbiased oracle model. We propose a new sampling and selection strategy at train time to approximate the oracle model for aleatoric uncertainty estimation. Our results show that our solution achieves both accurate deterministic results and reliable uncertainty estimation.
  arXiv  Detail & Related papers  (2021-11-22T08:54:10Z)
• Tracking the risk of a deployed model and detecting harmful distribution shifts [105.27463615756733]
  In practice, it may make sense to ignore benign shifts, under which the performance of a deployed model does not degrade substantially. We argue that a sensible method for firing off a warning has to both (a) detect harmful shifts while ignoring benign ones, and (b) allow continuous monitoring of model performance without increasing the false alarm rate.
  arXiv  Detail & Related papers  (2021-10-12T17:21:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.